How to Prepare a Disaster Recovery Runbook: A Technical Template

How to write a DR runbook: measurable triggers, boot order by dependency, a step template with commands and verification, an offline copy and the drill cycle.
Technical summary
Technology
Active Directory, SQL Server, Veeam, PowerShell, NIST SP 800-34
Topic
Disaster recovery runbook structure for the technical team
Level
Intermediate
Prerequisites
  • System inventory and dependency information
  • Approved RPO/RTO targets
In this article9 sections

When the storage array goes down at 02:00, a strategy document saves no one. The system administrator, working under stress with a reduced team, needs to know which server to bring up in what order, with which command, and how to tell the job is done. The runbook is the technical implementation document that fills this gap. Below we give the sections of a DR runbook, the recovery order, the step template and the cycle that keeps the document alive, along with an example section.

The difference between a runbook and a business continuity plan

A business continuity plan (BCP) answers the questions “in how much time, which business functions, on whose decision”; it covers crisis management, communication and alternative working arrangements. This management-level structure is described in the contents of a business continuity plan article. The runbook is the field counterpart of the BCP’s technical recovery section: it is written at the level of commands, screens, parameters, expected output and the person responsible. The runbook’s owner, its approval and its link to the organization-wide plan are defined within disaster recovery plan consulting.

A good runbook has a simple test: can a team member who knows the system infrastructure but never built that system carry out the document alone? If a step says “make the necessary configuration,” it isn’t a runbook, it’s a reminder note. Many runbooks are exactly like this: they rely on the knowledge in the author’s head and are useless when the author is on leave.

Sections of the runbook

The runbook doesn’t have to be a single long document. Organizing it as a main document plus system-specific appendices makes updating easier. The main document contains these sections:

  1. Scope and version: Which systems it covers, last update date, last drill date, version history.
  2. Trigger criteria: In what situation the runbook is activated and who makes that decision. Example: “The storage array in the primary data center has been unreachable for 60 minutes and the vendor support case estimates repair time above 4 hours.” A vague criterion causes decision delay; the most expensive minutes of an outage are often spent on the “should we wait or switch over” debate.
  3. Scenario selection: Loss of a single server, loss of storage, loss of a site and ransomware follow different paths. In the ransomware scenario, restoring doesn’t start before a clean recovery point is chosen.
  4. Roles and communication: The recovery coordinator; those responsible for systems, network, databases and applications; a backup person and communication channel for each. How to communicate if the email server isn’t working should also be in writing.
  5. Order of return: The dependency map and the wave plan.
  6. System appendices: A step-by-step recovery procedure for each system.
  7. Closing: The decision to open to users, the failback plan, the post-incident review.

Order of return and dependencies

Systems are brought up by technical dependency, not business priority. ERP may be the most important system for the business; but bringing up ERP while authentication and the database are down gains nothing. A typical wave plan:

Wave Systems Gate criterion
0 – Foundation Network, firewall, hypervisor, storage Management network is reachable
1 – Identity AD DS, DNS, DHCP, PKI Sign-in and name resolution work
2 – Data SQL Server, file server Databases ONLINE, shares accessible
3 – Application ERP, CRM, integration services Application services running, test transaction successful
4 – Access RDS, VPN, web front ends, email relays User test successful

When mapping dependencies, look for hidden ones: the license server, time synchronization (NTP), access to the certificate revocation list, the SMTP relay, the service account an integration uses, IP addresses embedded in application configuration. These mostly surface in the first drill. They don’t show up on paper. For which RTO tier systems belong to and the matching technology choice, see the choosing technology by RPO/RTO target article.

Step template: command, verification, owner

Every step in the system appendices should carry the same fields:

  • Step number and title: Short, in the imperative (“Restore DC01”).
  • Prerequisite: Which step must be finished before this one starts.
  • Action: The command to run or the screen path to follow; server name, IP and parameters are written out explicitly.
  • Expected result: What successful output looks like.
  • Verification: An independent check that proves the step was really completed.
  • Estimated time: The time measured in the last drill.
  • Owner and backup.
  • On failure: The alternative path or whom to escalate to.

Verification commands are the most valuable part of the runbook. “The service started” and “the system works” aren’t the same thing; replication being broken while the NTDS service runs, or databases sitting in RECOVERY_PENDING while the SQL service is up, are common. Example verification commands for the identity and data waves:

# Wave 1 – AD DS and DNS verification
dcdiag /s:DC01 /q
repadmin /replsummary
nslookup erp.contoso.local DC01
Get-Service NTDS, DNS, Netlogon, DFSR -ComputerName DC01

# Wave 2 – SQL Server database state
sqlcmd -S SQL01 -E -Q "SELECT name, state_desc FROM sys.databases WHERE state_desc <> 'ONLINE'"

# Wave 3 – application service port
Test-NetConnection -ComputerName APP01 -Port 443

If the SQL query returns an empty result, all databases are ONLINE. dcdiag /q prints only errors; clean output is expected. The T-SQL steps for database restore should be consistent with the restore order in the SQL Server backup strategy article.

Access information and the offline copy

The runbook must be kept in a place and format that won’t be affected by the disaster. A single copy in a shared folder or SharePoint may be unreachable exactly when it’s needed. When ransomware encrypts the file server, the runbook is in it too. In the field these layers are used together:

  • Printed copy: In a locked cabinet at the second location; renewed with every version change.
  • Offline digital copy: An encrypted USB drive or storage independent of the corporate network.
  • Credentials: No passwords are written in the runbook; it records the name of the password vault and where to find the emergency (break-glass) accounts to use if the vault is unreachable. If the password vault is on premises and tied to the domain, it should have an offline export or a copy kept in a sealed envelope.
  • Keys and access: Backup encryption passwords, BitLocker recovery keys, DSRM passwords, hypervisor and storage management accounts, firewall management access, license keys and vendor support contract numbers.
  • Installation media: The backup software’s installer and configuration backup; if the backup server is lost, it has to be brought up first.

If the backup server is joined to the domain, access to the backup console becomes difficult in the scenario where AD is lost. The runbook should spell out these “chicken and egg” situations: if there is no AD, which local account and which IP to use to connect to the backup server. Without this line, the first hour is lost.

Example runbook section

The table below shows, in runbook format, restoring the primary SQL Server server to a host at the DR site in an environment backed up with Veeam. Prerequisite: wave 1 (identity and DNS) must be complete. Names are examples; add timing from your own drill measurements.

No Action Verification Owner
2.1 In the backup console, select the restore point to use for SQL01; in the ransomware scenario, get clean point approval The date and time of the chosen point are recorded Systems specialist, coordinator approval
2.2 Restore SQL01 to the DR host as a full VM restore, with the network card disconnected The VM boots, sign-in screen on the console Systems specialist
2.3 Connect the VM network to the DR server VLAN, check the IP and DNS settings ipconfig /all shows the expected values; nslookup sql01 returns the new IP Network specialist
2.4 Check the state of the SQL Server and SQL Agent services Services running; recovery completed in the error log Database specialist
2.5 Run the database state query, apply the latest log backups if needed The query returns an empty result; the last transaction time is recorded Database specialist
2.6 Send the go-ahead notice for wave 3 to the application team The coordinator approved the go-ahead Coordinator

The “last transaction time” in step 2.5 is the evidence of the actual RPO and is written into the drill report. Domain controllers don’t fit this pattern: restoring an old DC backup like an ordinary VM and connecting it to the network while working DCs exist can cause directory consistency problems. DC steps are written according to an AD-specific procedure; the methods by scenario are in the Active Directory backup and restore article.

The drill and update cycle

A runbook that has never been carried out is a list of assumptions. The cycle that keeps the document alive has four steps:

  1. Drill: A tabletop review (steps are read aloud, gaps noted), a partial technical test (a single system is restored on an isolated network) and a planned full test. A common practice for critical systems is a partial test every few months and a full test at least once a year.
  2. Measurement: The real duration of each step, deviations from the expected result, work that had to be done but isn’t in the document.
  3. Update: Within a week after the drill, the runbook is corrected, the version number is increased and the printed and offline copies are renewed.
  4. Change triggers: A new server, an IP change, an application version upgrade, a change of backup software or a change of roles in the team makes a runbook update mandatory. Adding the question “Is the DR runbook affected?” to the change management form is the simplest way to build this link.

The times measured in the drill are compared with the target RTO; the gap is closed either with technology (replication, faster storage) or with process (automation, parallel steps). The second is usually cheaper. If you are looking for a public framework, the NIST SP 800-34 Rev. 1 contingency planning guide for information systems is a good start.

Checklist

  • Trigger criteria are measurable and the decision-maker is defined.
  • The order of return is divided into waves by technical dependency.
  • Every step has the action, verification, time and owner fields filled in.
  • Printed and offline copies of the runbook are at the second location.
  • The path to the backup console without AD is written down and tested.
  • Encryption passwords and recovery keys are in an accessible place independent of the runbook.
  • The last drill date and the measured RTO are at the top of the document.

For runbook preparation, dependency analysis and drill planning, you can find the scope of our support on our backup and disaster recovery and infrastructure consulting pages.

Frequently asked questions

Can the DR runbook and the business continuity plan be the same document?

In small environments they can be kept in the same document, but in separate sections. The business continuity plan is written at the level of decisions and communication, the runbook at the level of commands and verification; mixing them makes the technical steps unreadable.

Should passwords be written in the runbook?

No. The runbook records not the password itself but the vault where the password is kept, and where to find the emergency account to use if the vault is unreachable. The vault should also have an offline copy that won’t be affected by the disaster.

How often should the runbook be updated?

After every drill and with every infrastructure change that affects the runbook. Adding a runbook impact question to the change management process keeps updates from being forgotten.

What determines the order of return?

Technical dependency, not business priority. Network, hypervisor and storage first; then identity and DNS; then databases, applications and the user access layer.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek System Support

Server installation, configuration, incident response and scheduled maintenance. Active Directory, virtualization and backup are managed with the same discipline.

Knowledge Center

Other articles