Handling sensitive topics: security issues
Bug, beg, or just a bother?
This week and next I’m going to be talking about how to handle sensitive issues that come in through the support inbox. I don’t mean ‘complicated’ or ‘long-running’ here. Rather, I’m talking about inquiries or complaints that could cause bad outcomes to your business if they’re not handled correctly. A report of a serious vulnerability in your product, for instance, needs special handling by a domain expert, ideally someone on your security team, if you have one. A frustrated customer that’s boiling over into threats to evaluate competitors is a flashing red light for customer health, and should immediately be reported to Customer Success (CS). But these issues are rare enough that you may not have established policies and procedures around handling them. If not, it’s time to change that! This week we’ll talk about that first category: security issues.
An example
One Friday afternoon, as the support engineer on duty is wrapping up for the day, an email comes into your ticketing system:
Hi,
I noticed that your OIDC configuration has a problem: I was able to successfully authenticate as another user. Details are in the attached document. Please let me know if you have any questions about this. Does your organization have a bug bounty program?
What will you, the support engineer on duty, do upon receiving this report?
Send a noncommittal answer and make a note to ask Engineering and/or Security on Monday: The technical details are above your head, and you’re not sure how real the report is. But everyone’s done for the day, so you can follow up next week.
Immediately open a security incident and bring in the engineer on duty: This sounds bad, especially since the reporter claims to have actively exploited a weakness. Better safe than sorry, so ring the alarm bells now.
Escalate to the Security team: They have the expertise to deal with it. But are they still working at this hour? Well, not your problem. They’ll get to it when they’re back.
Ignore it: This is probably bogus, especially because they’re immediately asking about getting a bounty for it. Just close the issue.
A few themes are immediately apparent here: first, in the absence of a defined policy, a lot is left up to the discretion (and knowledge base) of the engineer on duty. Is this a real issue? Is it possible to determine in the five minutes before quittin’ time? Second, why is this decision being left to a support engineer? Exploited security issues can be of enormous consequence to the business as a whole, as we’ll talk about in a moment, and it’s not fair to put that on the shoulders of the support team. And though it's less of an issue in this specific case, it can often be difficult for an untrained eye to even determine what is and is not a security issue. (More on this below.) For these and many other reasons, it’s better to take the determination out of Support’s hands entirely and provide a specific procedure for addressing issues like this.
Consequences of mishandling
As I briefly mentioned above, security issues are of critical importance to the health of your business. It’s not an exaggeration to state that they can be literal company-killers. An unpatched vulnerability could lead to a successful ransomware attack which could destroy (or disclose) all of a company’s proprietary data. An improperly secured S3 bucket or cloud Mongo database could find your customers’ authentication details uploaded to Have I Been Pwned. A weakness in your authentication process could allow an attacker to take over your largest customer’s administrative account. For a large company, any of these scenarios would be at minimum a tremendous setback. For a startup, any of them would be an extinction event.
So why are so many companies content to let Support field these issues with little to no guidance? I don’t have any strong insights here, but here’s my speculation, for what it’s worth: in early startups, company killers are a dime a dozen and can often strike with little or no warning. Competitors can kill you. Lack of new business can kill you. Unexpected government regulation can kill you. And at bottom, all that matters for a startup is growth and revenue. But ‘kills the company’ and ‘has no effect’ aren’t the only outcomes of a security incident. There’s a whole range of possible effects of a poorly managed security issue that are worth consideration and mitigation, because they can have a direct effect on both revenue and company growth. Just a few possibilities:
Customer loss. If a customer reported a security issue to you and got blown off, or didn’t feel that you addressed it correctly, how likely are they going to be to want to keep doing business with you? If they discover that you were aware of a security vulnerability that was later exploited, to their detriment, you may very well be hearing from their attorneys.
Reputational damage. Take the above point and expand it beyond your customer base. Data breaches, product downtime, and other outcomes of vulnerability exploitation can do a number on your company’s reputation in your industry. This makes it harder to keep and retain customers, and will weigh heavily on any future attempts to raise capital.
Fines (and other penalties). Even though there still isn’t a federal data breach notification standard, state-level laws and their associated penalties should be enough to make your company pay attention. And if you’re doing business in Europe, GDPR (General Data Protection Regulation) fines for data exposure can be up €20 million (or even more in some cases).
Even if a poorly managed security incident doesn’t kill your company outright, there are plenty of ways it can haunt your business for years to come.
Side note: beg bounties
So named in the early 2020s, beg bounties (as distinguished from legitimate bug bounties) aren’t new but in my own anecdotal experience are getting more and more prevalent. In brief, these are low-quality bug reports, often either trivial (‘your website headers include time zone information, permitting attackers to determine the time zone where your server is located’) or completely incorrect (‘your Wordpress site exposes /wp-admin/‘ where the site in question isn’t Wordpress-based at all…), and asking for a bug bounty fee in exchange for this ‘valuable’ information. These can safely be ignored, but to a support engineer, not all of these are obviously worthless. This is yet another argument for having established processes, and a trained eye looking over each and every security report.
Making sure it’s done right
So what do I suggest? Ideally, you should just have your security team address all security-related inquiries directly. They’re in the best position to know what is and is not a vulnerability, and what is and is not urgent. Moreover, these teams are already making security-related tradeoffs on behalf of the company, and are comfortable with the parameters and the company’s appetite for risk. Exploit this knowledge, and let them take the lead on investigating and responding to security-related support tickets.
That said, you won’t always be fortunate enough to have a security team that has the capacity to field all security issues that come into your support inbox. For smaller and less well-staffed companies, a two-pronged approach has worked well for teams I’ve led:
Educate your team: make sure everyone on your team has a basic understanding of software vulnerabilities, the possible consequences of exploitation, and knowledge of where to go to learn more about specific security-related topics. If your budget permits, a basic security certification would be ideal for anyone on your team who doesn’t already have that level of knowledge.
Have a designated security contact: hopefully this is an actual security person, but if your organization is too small for that, it needs to be an experienced engineer with an understanding of and appreciation for software security. Worst case scenario, in a tiny organization, this can be your CTO or founding engineer. Whoever it is, your team needs someone they can ask very specific security-related questions to and get an authoritative answer. Crucially, the security contact needs to understand that this is an on-call role and they’re always on call: if a report comes in at midnight, someone needs to be available to investigate it.
With those two things in place, you’ll be able to build out a straightforward-enough response policy for security reports. Your team will be capable of initial triage, rooting out the obvious beg bounties and irrelevant reports. For anything that looks legit, has serious enough possible consequences, or is complex enough that your team is unclear about either, the designated security contact can step in to investigate further. Between your team and your security escalation point, you will be able to ensure valid reports are investigated promptly and thoroughly.
Next time: what about sensitive issues that aren’t about security, but just angry customers?
Thanks for reading Andy's Support Notes 💻💥📝!