On-Call Best Practices for Sustainable Engineering

July 24, 2026

Implementing on-call best practices is crucial for sustainable engineering, aiming to reduce engineer burnout and improve incident response by designing fair rotations, optimizing alert management, and fostering a supportive culture. Effective strategies involve distributing on-call load equitably, compensating engineers fairly, and providing clear escalation policies and runbooks. These measures collectively enhance work-life balance and operational efficiency within SRE and DevOps teams.

Understanding On-Call Challenges and Goals

On-call burnout frequently stems from issues such as alert fatigue, where a high volume of non-actionable alerts desensitizes engineers. For instance, a PagerDuty setup with a single engineer on call and no escalation policies can lead to this, as seen in early SRE team configurations. Another challenge is an uneven On-Call Load Distribution, where one engineer absorbs most of the incident response, leading to an "always-on" readiness that disrupts work-life balance. Lack of formal compensation or recognition for out-of-hours work also contributes to burnout, as it signals that an engineer's time outside business hours is undervalued.

The primary goals of effective on-call management are to mitigate these issues and improve overall operational efficiency. Key objectives include reducing Mean Time To Resolution (MTTR) from alert to resolution, which is a primary health metric for incident response programs. This involves optimizing processes and runbooks to address Incident Recurrence Rates. Another goal is to reduce Alert Volume per Shift by improving the signal-to-noise ratio through deduplication, suppression, and prioritization filters, ensuring that only actionable alerts page engineers. Ultimately, effective on-call management aims to foster engineer well-being by creating fair rotations, providing compensatory time off, and conducting blameless postmortems to learn from incidents without assigning blame.

Designing Fair and Effective On-Call Rotations

Effective on-call rotations are built on models that distribute the load equitably and provide clear escalation paths. The Primary/Secondary model is common for mid-sized teams and ideal for onboarding junior engineers. In this setup, a primary responder handles initial pages, with a secondary engineer automatically paged if the primary doesn't acknowledge within a defined window, often 5 minutes. An escalation policy might then page an Engineering Manager after 15 minutes if neither responds, and a director after 30 minutes. Some teams rotate last week's primary into this week's secondary role to maintain context and speed up escalations. For larger teams (9+ engineers) across multiple time zones, a "Follow-the-Sun" model ensures continuous coverage without requiring individuals to work outside their local business hours. Smaller teams often benefit from weekly rotations for predictability.

Fair compensation for on-call duty is critical to prevent burnout and acknowledge the disruption to personal life. Compensation models vary; some organizations offer direct pay for out-of-hours paging, while others provide compensatory time off after particularly heavy weeks. The key is transparency and consistency, ensuring engineers feel valued for their commitment. On-call tools like PagerDuty assist in managing these rotations and escalation policies, helping to reduce alert fatigue by ensuring only actionable alerts reach engineers. Regular review of metrics such as On-Call Load Distribution helps identify if the burden is unevenly spread, allowing for adjustments before burnout occurs.

Optimizing Alert Management and Incident Response

Effective alert management is crucial for reducing alert fatigue and improving incident response. The objective is not to eliminate alerts, but to ensure every alert demands immediate action. This is achieved by refining the signal-to-noise ratio through deduplication, suppression, and prioritization filters. When an alert triggers, it should always be actionable. On-call tools like PagerDuty are instrumental in this process, allowing for sophisticated routing and escalation policies. For example, a PagerDuty setup can be configured to page a primary responder, then automatically escalate to a secondary engineer if the alert isn't acknowledged within 5 minutes, and further to an Engineering Manager after 15 minutes, and a director after 30 minutes.

Well-defined runbooks are essential for efficient incident resolution, providing clear steps for addressing common issues and reducing Mean Time To Resolution (MTTR). These runbooks should be regularly updated and accessible to all on-call engineers. Integrating DevOps practices can further streamline incident response by fostering better communication between teams and ensuring developers support the services they build. This approach helps in delegating issues to the most appropriate team members with ease, rather than creating communication silos. Monitoring metrics such as Alert Volume per Shift and Incident Recurrence Rate helps identify areas for improvement, such as recurring issues that indicate a need for automation or enhanced runbooks.

Continuous Improvement Through Postmortems and Metrics

Blameless postmortems are crucial for continuous improvement in on-call systems. Following an incident, the goal is to understand the sequence of events and identify systemic issues rather than assigning blame to individuals. This approach, central to Google's SRE culture, focuses on analyzing processes to prevent recurrence. A well-conducted postmortem answers "What happened?" to facilitate learning and improve future incident response.

Evaluating the effectiveness of an on-call program relies on key metrics, which should be reviewed monthly. Mean Time To Resolution (MTTR) is a primary health metric for the incident response program, measuring the time from alert to resolution. Alert Volume per Shift tracks the total alerts fired versus those that are actionable, helping to refine the signal-to-noise ratio and combat alert fatigue. Incident Recurrence Rate highlights if the same issues are reappearing, indicating potential remediation gaps that could be addressed through automation or improved runbooks. Tools like PagerDuty can assist in tracking these metrics, providing data to justify adjustments to rotation models or additional headcount before engineers experience burnout.

Fostering a Supportive On-Call Culture and Training

A healthy on-call system requires leadership to prioritize engineer well-being, moving beyond informal obligations to establish clear support systems. This includes transparent compensation models, such as direct pay for out-of-hours shifts or compensatory time off after heavy on-call periods. Recognition of the burden and fair treatment increases engineer willingness to participate. A culture of shared responsibility also means empowering engineers rather than punishing them for incidents.

Effective training and onboarding are crucial for new and junior engineers. Best practices include:

Comprehensive Training Programs: Address both incident response processes and common technical issues.
Up-to-Date Runbooks: Provide clear, actionable steps for resolving frequent problems, reducing MTTR.
Shadowing and Mentorship: New employees should shadow experienced on-call engineers to gain practical knowledge.
Access to Past Incident Reports: Reviewing how similar incidents were resolved in the past provides valuable context and learning opportunities.
Graduated Responsibility: Employ a Primary/Secondary model where junior engineers take the primary on-call role, with senior engineers as automatic backups. For instance, a PagerDuty escalation policy can page a junior engineer first, then automatically escalate to a senior engineer if unacknowledged within 5 minutes. This builds skills while providing a safety net.

Leadership's behavior shapes the on-call culture; valuing rest, providing compensation, and celebrating lessons learned from blameless postmortems build resilience and prevent burnout.

Frequently Asked Questions

How do you create a healthy on-call rotation?

A healthy on-call rotation involves transparent compensation, fair treatment, clear support systems, and leadership prioritizing engineer well-being, moving beyond informal obligations. It also includes comprehensive training, up-to-date runbooks, and a culture of shared responsibility.

What are the benefits of a good on-call system?

A good on-call system reduces engineer burnout, improves incident response times (MTTR), fosters continuous learning through blameless postmortems, and ensures the reliability and stability of services. It also empowers engineers through training and graduated responsibility.

How can companies fight pager fatigue?

Companies can fight pager fatigue by refining the signal-to-noise ratio of alerts, implementing automation to address recurring issues, and ensuring transparent compensation models or compensatory time off for on-call duties. Monitoring metrics like Alert Volume per Shift helps identify areas for improvement.

Should workers be compensated for being on-call?

Yes, workers should be compensated for being on-call through transparent compensation models, such as direct pay for out-of-hours shifts or compensatory time off after heavy on-call periods. This recognizes the burden and increases willingness to participate.

What are the signs of a poor on-call process?

Signs of a poor on-call process include high incident recurrence rates, frequent alert fatigue, long Mean Time To Resolution (MTTR), and engineers experiencing burnout due to informal obligations or lack of support. A lack of blameless postmortems and inadequate training also indicate a poor process.

How do you train on-call teams?

Training on-call teams involves comprehensive programs addressing incident response and technical issues, providing up-to-date runbooks, offering shadowing and mentorship opportunities, granting access to past incident reports, and implementing graduated responsibility models like Primary/Secondary on-call.

Conclusion

Creating an on-call system that supports engineers rather than burning them out is not just a moral imperative, but a strategic advantage. By prioritizing well-being, implementing smart tools, and fostering a culture of continuous improvement, organizations can transform on-call from a dreaded duty into a valuable growth opportunity. This holistic approach ensures both system reliability and team resilience.

Sources & References

Want to actually learn Engineering?

Curo turns topics like this into a personalized, guided learning board - built around what you already know. Free to start.

Try Curo

Or jump straight in:

on-call best practices On-call that doesn't burn people out incident response SRE DevOps