Staffing levels and support coverage, part 2
I didn't have time for a short post, so I made a long one
Continuing from last week, here’s some deeper discussion of determining the right staffing levels for the third support coverage model: limited off-hours support.
Off-hours coverage
Adding in some sort of coverage for support issues that come in outside your normal business hours can be as simple or as complicated as you like. Let’s look at the simplest end of it first: no additional staffing, but existing resources are assigned responsibility for off-hours support issues. Typically this comes in the form of an on-duty rotation, where a given person is responsible for issues coming in overnight, or on the weekend, and that duty rotates among team members. The benefits of doing it this way are clear: no additional staffing needs, and it is always clear who is responsible for those off-hours issues.
The drawbacks, unfortunately, are also very clear: folks will be working, either actively (responding to issues) or passively (available to respond to issues, though not currently doing so) for far more than 40 hours a week, which is not usually anyone’s idea of a good time. There needs to be some sort of compensation for those extra hours worked, whether in extra pay or in comp time where engineers can take days off in trade for off-hours shifts worked. The former is a straight trade of additional pay for additional work, but is liable to vastly increase burnout risk. The latter, though making it appear on paper that there’s no additional budgetary impact, still leads to fewer engineer-hours available during normal business hours, increasing support load for the entire team. That is not to say that this method should never be used, but it has tradeoffs that you and the rest of company leadership need to be aware of.
The other method for handling off-hours support is to bring in additional assistance, whether by increasing your actual team size or by outsourcing off-hours support handling to a third party. Practically speaking, adding resources to the team is straightforward from a budgetary impact perspective, but can get very complicated when you consider that those people will need to be working those off-hours times, which means either working the graveyard shift or living in a time zone, probably overseas, where those hours are normal or close to normal working hours. In either case, managing those folks can be a challenge.
You can sidestep some of these issues by outsourcing off-hours to a contract team. There’s no need to directly manage the people handling the support issues, and in this modern world it is straightforward to find resources that can handle any hours of the day or night. The downsides here, of course, are not hard to find either. Finding and hiring these resources is one thing, training them is another entirely. If it takes six months to completely bring a support engineer up to speed, do you plan to train your off-hours contract team to the same level? How many issues are they expected to handle on their own, or will they just be a glorified answering service? The more independent you need them to be, the more you can expect to pay for the privilege.
Case study
In my last role where we had to provide 24-7 response with a small team, our method was a hybrid of the on-call and contractor approaches. For quite a while, the model looked like this:
Monday-Friday: US-based support was on call between 0800ET and 2300PT.
Weekends: One engineer was on duty every weekend, responsible for every ticket that came in between 0800ET and 2300PT.
Everything else: a Europe-based contractor handled initial ticket response and escalated as necessary to the engineer on duty.
We managed this with a team size of 3-10, as overall ticket load grew and we expanded the team to ensure support load never got too high. Eventually we also expanded to hire a couple of full-time Europe-based engineers, which let us scale back contractor coverage to weekend overnights, escalating to the engineer who was on weekend duty. But that’s a tale for another time.
What worked about this
We achieved sub-15-minute initial ticket response on a team that, by all rights, should never have been able to manage that.
We had enough overlap that for most of the day there were at least 3 or 4 engineers available in case of ticket surges or time off.
Because we didn’t expect any actual issue handling from the outsourced team, we were able to onboard them quickly and with minimal training.
Budget-wise, the off-hours contractor cost was a small fraction of what it would cost to hire and train full-time staff to handle those off hours.
What didn’t work about this
There was no defined escalation path for overnight ticket escalations. Until we had European staff, all overnight escalations went directly to me, which eventually got very tedious.
Because the outsourced team had no authority—or training—to solve customer issues, escalations were the norm for tickets coming in during off hours.
The only reason we had coverage for late evenings PT is because our CTO kindly offered to remain on duty alone for those hours. Though it tended to be a very quiet time for incoming support issues, we were very reliant on one already-busy person and would have been in a lot of trouble if he had to bow out on short notice.
What I’d do differently next time
Don’t rely on non-support staff for handling support tickets. While it worked for us, I was acutely aware that eventually I’d have to find the budget to hire actual support engineers to cover the hours that our CTO was currently doing for ‘free’.
Pay more, and train more, when dealing with contractors. If our outsourced off-hours support response had been capable of basic product troubleshooting, there would have been far fewer overnight escalations and far more good nights of sleep (for me, mostly).
Go international sooner. While it’s more difficult to train, and manage, a support engineer who is multiple time zones offset from the rest of the team, it provides a lot of room to expand for off-hours support tickets. And if you hire carefully, that solo support engineer can build and lead their own support team as the overall support organization grows.
If you design your team well, you can sustain a surprisingly high level of customer issue handling, even during off hours, with a relatively small team. By carefully balancing team size, engineer location, and contractor quality, your team will be able to punch above its weight as your organization and customer base grow. But always keep in mind that eventually, if all goes well, your team will need to start aiming for the final support model of providing high-quality customer support at all hours of the day and night.
Next time: we’ll finally get to the 800-pound gorilla of full 24/7 support coverage. It ain’t cheap, but sometimes there’s no alternative.
Thanks for reading Andy's Support Notes 💻💥📝!