Trying to build a new on-call schedule or fix an existing one that doesn't work? Here is how to do it the right way.
Many engineers have experience with on-call that simply doesn’t work for their team. It’s either based on a years-old setup that hasn’t changed with the progress of the organization. Or it was done in a rush not getting the necessary care it deserves.
Here is how the @techroastshow characterized a poor on-call in one of their short videos.
@techroastshow on call again #techbro #techie #softwareengineer #siliconvalley #seattle #sanfrancisco #bayarea #engineeringlife #comedy @Rich Rosales ♬ Mad World (feat. Michael Andrews) - Gary Jules
Any engineer with experience with it would probably agree, that on-call is inherently stressful and that it should be crafted properly. Once set up it will quite literally control hours of valuable engineering time, so let’s make one that works.
On-Call is a practice of always having a team member on standby, ready to respond in case of an urgent incident, even if it occurs outside of the regular working hours. It’s one of the core processes of Incident Management and a key to minimizing downtime and ensuring a reliable service.
On-Call schedule is a dedicated calendar allowing teams to assign and monitor the on-call shifts.
Google SRE (site reliability engineering) teams usually have a schedule where each engineer is on-call for one week every month. During this week they are ready to respond to incidents at any time of the day and night.
The rest of the month they spend on engineering (ideally 50% of their time) and on other operational, non-project tasks (around 25% of their time).
Recommended reading: Google SRE Book: Being On-call
With 24/7 service availability being a goal of many modern organizations having a good on-call process is vital to:
Creating a quality on-call is a challenging task because there is no one-fits-all model. But on-call doesn’t have to be difficult, sleep-depriving, or inevitably leading to burnout.
Here are 7 strategies to help you in designing an effective on-call schedule:
Every team is different and every team member has different priorities and preferences. Understand what your team needs and consider different individual situations to help everyone work more efficiently.
There might be a consensus on what the on-call should look like within your team and a complicated setup might not be necessary. For example, if your team agrees on a week and weekend schedule there is no need to go for more complex follow-the-sun schedules with other teams across the world.
Nobody likes being woken up by incident alerts at 4 in the morning. Not just that poor sleep leads to worse performance during day hours, but any reaction time and incident-solving capability are decreased whenever an engineer needs to wake up. Not to mention the mental and physical health consequences.
Implementing the follow-the-sun model where possible is a commonly used practice that prevents those issues. If the location of the on-call team members allows it, it should be considered first.
You can nicely see different working hours across time zones with worldtimebuddy.
Being on-call outside business hours can be complicated and there are unexpected situations when an engineer can’t respond as fast as it would be necessary. To prevent any incident from going unsolved and causing damages it’s best to always have a backup on-call engineer that is able to step up and fix the issue in case of an emergency.
Backup on-call carries the same responsibilities as the primary one and it needs to be treated that way. First, the team needs to understand that being a backup is no different from primary - you must be ready to react within minutes. Secondly, managers need to treat it that way and consider being a backup on-call engineer equivalent to regular on-call duties, especially when it comes to compensation.
The responsibilities of on-call engineers need to be clearly defined - ideally in a written form so there is a single source of truth of what is the incident response process.
Responsibilities are specific to a given organization, but good questions to answer and write down are for example:
Other process-related questions:
Creating a supportive culture within a team can significantly improve both employee happiness and incident response effectiveness. Every once in a while personal emergencies or important life events come up.
Maybe an old friend is in town or maybe you just hit that runners high. Encouraging team members to help each other and switch duties to step in for others makes all the difference. When teams care for each other the on-call challenge feels much more manageable.
Staying on the same page and having a unified source of truth is key to alerting, collaboration and overall effective incident response. Using a tool like Better Uptime allows for easy on-call scheduling, alerting and incident collaboration - helping your team to maximize its potential and create the best on-call both on side of metrics as well as on the side of teams well-being.
Recommended reading: Better Uptime vs. Pagerduty vs. Opsgenie
As products, organizations, and teams develop, there is always a need to iterate and fine-tune to accommodate for changes. Don’t be afraid to revisit old processes and ask your team for feedback frequently. On-call is not a static process.
Focusing on improving incident metrics is a great place to start. But also look into what directly influences the well-being of on-call engineers:
In general, the fewer alerts an on-call person receives the lower the chance of something like alert fatigue developing.
Alert fatigue means that people responsible for responding to alerts become numbed to new incoming alerts. This leads to missed or ignored alerts as well as delayed response times.
There are two main reasons for alert fatigue:
Sometimes on-call gets a bad reputation among engineers. The anxiety that it brings up is caused by poor choices companies make, but it doesn't have to be that way.
Here are 4 most common mistakes that companies make when designing on-call:
On-call teams are the most valuable resource a company has in tackling incidents. Not allowing them to have a healthy work-life balance will eventually lead to lower performance (or worse, people leaving) over time.
Flexible and changing schedules are harder for team leads to manage of course, but the extra effort is worth it. When people can reschedule on-calls to accommodate their lives, it creates a great team spirit and empowers everyone to support each other.
Alerting rules need to be designed properly to avoid on-call teams being overwhelmed with alerts. Being woken up in the middle of the night because of a minor issue that can easily wait till the morning is something that can be avoided with proper alerting policies.
On-call responsibilities can vary between the day and night shifts. A minor incident alert might be useful for the on-call engineer during the middle of the day, but is definitely not vital during middle of the night. Using severity levels is a best practice when it comes to defining the priority of incidents.
Having only operational engineers responsible for the reliability of service is risky at best. It gives the developers who write the software no incentive to ship a reliable code. This often leads to more bugs and incidents.
To prevent this, enter the: you built it, you maintain it setup.
It’s the idea that engineers that write code, should also take care of their systems and fix issues if they arise. Since they are the most familiar with the inner system workings, they are in the best position to troubleshoot and should be more efficient than purely operational teams.
An added benefit of this approach is that it gives engineers the incentive to test more before deployment. Giving a better sense of ownership, it aligns the motivations to ship, but also to ship reliable code, creating a more resilient system.
Every team has team members with a specific preference, location, and experience. When building on-call those need to be considered and built upon. For example, in a team of two people in the same city, it’s impossible to go for a follow-the-sun model, but it allows for having one primary and one backup on-call person.
Those organizational factors need to be considered during scheduling to create a good result for the whole team. Most importantly keeping them in mind prevents you from implementing “best practices” (or “those are done at Google, practices”) that are outside of your organizational capabilities.
Having developer-first on-call also means paying people what they are worth. To do so it’s necessary to pick the right way of compensation.
The common types of on-call compensation models are:
This is often the default setup for many smaller companies. It’s not inherently a bad setup. When people don’t spend significant time solving incidents, or when they only solve incidents during business hours it can be fine.
In cases when solving incidents becomes an inconvenience this form of no-compensation can become problematic and needs to be well communicated with the team.
This model means that on-call responsibilities and expectations are clearly outlined in the contract and reflected by the salary. For example a year base salary contract should include that it comes with a responsibility to be on-call once a week every month.
No monthly calculations are necessary with this setup making it simple to implement and manage.
On the other hand it limits scheduling flexibility since there is not much motivation to switch duties because there is no extra reward for it. When switches occur they tend to be transactional: “I will take a day for you if you take a day for me”.
An alternative model to the base salary pay is to pay only for the time spent working on an incident. This creates a direct relationship between incidents and monetary rewards making the compensation easy to understand.
However, this model can be tricky because it creates a financial disincentive to reduce incidents and to treat incident with the necessary urgency. Even though this might be an extreme example it’s a possibility to consider.
Another issue is that when incidents don’t occur, on-call people might feel they are carrying computers around and giving their peace of mind for free.
This setup is simply paying employees for the time they spend on-call, even if no issues arise.
Paying for the whole time spent is a great way to justify the need to be ready and available at moment’s notice.
The main disadvantage of paying for on-call time is that incidents occur at random making some duties easy and some extremelly challenging. Naturally this can cause some team-members to feel this compensation is unfair.
This unfairness can be assesed by paying more for on-call with more severe incidents. Esentially creating different rates for “high” and “low” workloads.
When a regular compensation model won’t be enough. You can explore other options that can be added to the regular model that you decide to go for. This can include for example:
Creating a good on-call schedule is an ongoing process, so don’t worry if you don’t get it 100% right the first time.
Hopefully you now have a good foundation to start designing and improving your on-call. If you want to dive a bit deeper feel free to read more in: Being on-call: Google SRE book
If you want to read more about how you can improve your incident management process, check out out status page articles:
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Are you a developer and love writing and sharing your knowledge with the world? Join our guest writing program and get paid for writing amazing technical guides. We'll get them to the right readers that will appreciate them.
Write for us
Write a script, app or project on top of Better Stack and share it with the world. Make a public repository and share it with us at our email.
[email protected]or submit a pull request and help us build better products for everyone.
See the full list of amazing projects on github
