Chapter 18 · Part IV · Detect & respond

Incident response & recovery

Sooner or later something gets through. What happens next decides whether it is a bad afternoon or a lost year: who gets called, what evidence survives, how fast the damage stops spreading, and whether there is a copy of your data the attacker could not reach. This chapter is about doing those things well, and about the arithmetic of backups that survive ransomware.

6functions in NIST's Cybersecurity Framework 2.0 (February 2024): Govern, Identify, Protect, Detect, Respond, Recover
3-2-1the backup rule CISA gives: 3 copies of important data, on 2 different media types, with 1 copy offsite
2012 → 2025NIST's incident handling guide SP 800-61 Rev. 2 was superseded by Rev. 3 in April 2025, rebuilt around CSF 2.0

Common mix-up: an alert is not an incident. NIST calls any observable occurrence an event; an adverse event is one with a negative consequence; an incident is one that actually or imminently jeopardizes the confidentiality, integrity or availability of information, or violates security policy. Most of the work at the start of a response is deciding which of the three you are looking at, and writing down why.

Before you start: this chapter assumes the diagnostic habits from The troubleshooting toolkit on networks.jelia.nyc (work from evidence, change one thing at a time, write it down), and it starts where Chapter 17, Logs, detection & ATT&CK stops: an alert has fired and someone has to act on it.

The shape of a response

Four phases, then six functions

For more than a decade, most incident response plans in the English-speaking world were organized around one picture from NIST Special Publication 800-61 Revision 2 (2012): a loop of four phases. Preparation comes first. Then Detection & Analysis, where you notice something and work out what it is. Then Containment, Eradication & Recovery, where you stop the spread, remove the attacker and put things back; Rev. 2 drew an arrow back from this phase to analysis, because containing a problem often reveals more of it. Finally Post-Incident Activity, the lessons learned, which feeds back into preparation.

Revision 3, published in April 2025, keeps those activities but changes the frame. NIST's argument is that the old picture assumed incidents were rare, narrow and over in a day or two, handled by a separate team. Today, it says, incidents are frequent, recovery often takes weeks or months, and response has to be woven into ordinary risk management. So Rev. 3 is written as a Community Profile of the Cybersecurity Framework (CSF) 2.0, whose six functions are Govern, Identify, Protect, Detect, Respond and Recover.

In the Rev. 3 model, Govern, Identify and Protect are the broad preparation layer: they are not incident response themselves, but they make it possible. Detect, Respond and Recover are the response. Between them sits continuous improvement, the Improvement category inside Identify (ID.IM), and lessons can arrive at any moment, not only at the end. NIST is explicit that organizations may keep using whatever lifecycle model suits them. This chapter uses the four classic phases as a mental model because they match the order in which decisions arrive during a bad day, and it names the CSF function where it helps.

Two incident response lifecycle models Left: the SP 800-61 Rev. 2 loop of Preparation, Detection and Analysis, Containment Eradication and Recovery, and Post-Incident Activity. Right: the Rev. 3 model with Detect, Respond and Recover on top, Improvement in the middle, and Govern, Identify and Protect as preparation underneath. SP 800-61 REV. 2 (2012) Preparation Detection &Analysis Containment,Eradication &Recovery Post-IncidentActivity loops back as scope grows SP 800-61 REV. 3 (2025) · CSF 2.0 INCIDENT RESPONSE Detect Respond Recover Improvement (ID.IM): lessons, any time PREPARATION · BROADER RISK MANAGEMENT Govern Identify Protect dashed: lessons flow in, and back out to every function
Same work, different frame. Rev. 3's Table 1 maps the old phases onto the functions: Preparation to Govern, Identify and Protect; Detection & Analysis to Detect; Containment, Eradication & Recovery to Respond and Recover; Post-Incident Activity to Identify's Improvement category, which also touches every other phase.
Before the smoke

Preparation is most of the job

A fire drill does not put out fires. It makes sure that when there is one, nobody is standing in a stairwell wondering which way to go. Incident preparation is the same kind of work, done on a quiet Tuesday, and it is where most responses are actually won or lost. The questions to answer ahead of time are dull and specific.

  • Who decides, and who do they call? A named incident lead with authority to take systems offline, a deputy, and a contact sheet that includes leadership, legal counsel, the people who run each critical system, your insurer if you have cyber cover, and any outside responders you would hire. Keep it on paper or on a phone, not only on the file server that might be encrypted.
  • How will you talk? If the attacker owns your email or chat, they can read your response plan as you write it. CISA's ransomware checklist tells responders to use out-of-band communication, such as phone calls, to avoid tipping the attacker off. Decide now what that channel is.
  • What do you have, and what matters most? An asset inventory and a ranked list of critical systems. When everything is down, you restore in that order. CISA's guide explicitly recommends prioritizing restoration from a predefined critical asset list.
  • Can you see what happened? Logging with enough retention to look back past the attacker's arrival. That is Chapter 17's subject; here it matters because an investigation can only answer questions the logs can answer.
  • Runbooks for the likely cases. Short, numbered pages: a lost laptop, a phished account, a ransomware outbreak, a leaked key. Each says what to check, what to switch off, who to tell.
  • Practice. A tabletop exercise walks the team through a made-up incident around a table, asking "what do we do now?" at each step. It costs an afternoon and finds the missing phone number, the backup nobody knows how to restore, the admin who is the only person with a password. CISA publishes free tabletop exercise packages, and the drill further down this page is a small one.
Detection & analysis

Triage: is it real, how big, how bad

Triage is the first hour's question: of all the alerts and reports in front of you, which deserve people right now? It has three parts. Validity: is this an incident, or a false alarm, an admin doing something unusual, a misconfigured scanner? Scope: one machine, one account, one site, or everything? Impact: what is it costing the organization, and what could it cost if it keeps going?

Scope is the one people get wrong. The machine that raised the alert is rarely the first machine the attacker touched; it is merely the first one where your detection happened to fire. So the first useful act of analysis is to take what you know about this machine (the file, the remote address, the account, the time) and search for the same traces everywhere else. Those traces are indicators of compromise. Every new host or account they turn up widens the scope and usually moves the start of the timeline earlier.

Severity needs to be decided on a scale everyone agreed in advance, so the night shift and the CEO mean the same thing by "high." Rev. 2 offered three dimensions that still work well as a starting point:

DimensionLevels (SP 800-61 Rev. 2, Tables 3-2 to 3-4)The question it answers
Functional impactNone · Low · Medium · High (no longer able to provide some critical services to any users)What can't we do right now?
Information impactNone · Privacy breach · Proprietary breach · Integrity loss (can be more than one)What data was seen, taken, changed or deleted?
RecoverabilityRegular · Supplemented · Extended · Not recoverableCan we fix this with the people we have, and is the time to recover predictable?

Two habits make triage better. Start a written timeline in the first minute: every observation and every action, with the time and who did it. Memory is unreliable under stress, and the timeline becomes the spine of the review afterwards. And separate the roles: one person leads and decides, others investigate, one person handles communication. A lead with their hands on a keyboard stops leading.

Don't destroy what you need

Evidence and the order of volatility

The first instinct when a computer is misbehaving is to turn it off. Sometimes that is right, but it has a cost: everything in memory is gone. Running processes, open network connections, decrypted keys, malware that never wrote itself to disk, the command history of an attacker who is still logged in. All of it lives in RAM, and RAM forgets in milliseconds once the power goes.

RFC 3227, Guidelines for Evidence Collection and Archiving (2002), puts the rule simply: collect from the most volatile to the least volatile. Its example order is below. The most fragile evidence goes first, because collecting anything else changes it.

#Evidence (RFC 3227 §2.1, example order)How long it lasts
1Registers, cacheNanoseconds; in practice captured only as part of memory
2Routing table, ARP cache, process table, kernel statistics, memoryUntil the next change or power-off
3Temporary file systemsUntil reboot or cleanup
4DiskUntil overwritten; normal use slowly overwrites deleted data
5Remote logging and monitoring data relevant to the systemUntil log retention rolls it off
6Physical configuration, network topologyUntil someone re-cables
7Archival mediaYears

The modern practical version for a suspicious laptop is: if your endpoint tool can do it, capture memory and a triage collection of key files and logs, then use the tool's network containment, which cuts the machine off from everything except the security console while leaving it powered on. You have stopped the spread and kept the evidence. If you have no such tool, unplug the network cable or turn off Wi-Fi rather than the power. CISA's checklist says the same: power devices down only if you cannot disconnect them from the network, because powering down loses volatile evidence. In the cloud, take a snapshot of the affected volumes before changing anything.

Two more rules apply whether or not anyone ever goes to court. Record chain of custody: for every piece of evidence, who collected it, when, from where, and every hand it passed through. And compute a cryptographic hash of each image or file when you collect it, so you can show later that it has not changed (Chapter 2 explains why a hash can do that). Work on copies, never on the original.

Go deeper: what RFC 3227 says not to do

Section 2.2 of RFC 3227 lists things to avoid, and they are still the classic mistakes. Don't shut down until you have finished collecting. Don't trust the programs on the affected system, because an attacker may have replaced them; run your collection tools from trusted media. Don't run programs that modify the access times of every file, such as an archive tool run over the whole disk. And be careful about disconnecting from the network if the attacker's code is likely to notice and wipe evidence; Rev. 2 of SP 800-61 gives a similar warning that some malware does more damage when it loses contact.

For the detail of collection and examination, NIST SP 800-86, Guide to Integrating Forensic Techniques into Incident Response, is the companion document. Whether your evidence must meet a legal standard depends on where you are and what happened, which is a question for counsel, not this page.

Containment

Stop the spread without tipping them off

Containment buys time. It does not remove the attacker, it limits how much more they can do while you work out how to remove them. Rev. 2 lists the criteria for choosing a strategy, and they are worth reading as a list of trade-offs: potential damage and theft, the need to preserve evidence, service availability, the time and resources the strategy takes, how complete it is, and how long it will be in place, from an emergency workaround for four hours to a permanent fix.

ActionStopsCostsWatch out for
Isolate a host (endpoint containment, unplug)That host spreading or being used to reach othersThe user's work; any service on that hostIsolating one machine while three others are infected
Disable or reset accountsStolen credentials being usedThe real user is locked out; services using the account may breakService accounts that nobody remembers depend on; tokens and sessions that outlive a password change
Block indicators (addresses, domains, hashes)Known command-and-control channels and known filesVery littleThe attacker switching to a channel you haven't seen yet
Take a whole segment or site offlineEverything in it, at onceLarge: whole departments stopDoing it too late, or not knowing which switch does what
Watch and waitNothing; it gathers intelligenceThe attacker keeps working while you watchRev. 2 calls delayed containment dangerous; do it only deliberately, with legal advice

The subtle risk is tipping off the attacker. An intruder with domain administrator rights who sees accounts being disabled one at a time, or reads an all-staff email announcing the breach, may decide it is time to cash out: deploy ransomware to every machine they can reach, or wipe their tracks. CISA's guidance is to isolate systems in a coordinated manner using out-of-band communication, for exactly this reason. Containment works best as a planned cut-over: find the scope first, then disable the compromised accounts, reset the credentials, block the indicators and isolate the hosts in one short window.

The second subtlety is short-term versus long-term containment. Short-term actions are blunt and fast: pull the cable, block the address. Long-term containment keeps the business running safely while eradication is prepared, for example by moving the affected systems to an isolated network with tight rules, patching other systems against the same entry point, and raising monitoring around the accounts the attacker used. Some temporary measures will need to come back out; write them down in the timeline so they do.

Eradication & recovery

Rebuild, reset, patch, watch

Eradication means removing everything the attacker left: malware, new accounts, scheduled tasks, changed configurations, stolen credentials. The hard question is whether to clean or rebuild. Cleaning means finding and removing each artifact; it is faster, but you are betting you found all of them. Rebuilding means wiping the system and reinstalling from a known-good image, then restoring data; it is slower, but it does not depend on the completeness of your hunt. For anything the attacker had administrative control of, rebuilding is the safer default. CISA recommends maintaining "golden images" of critical systems for exactly this.

Three steps get skipped surprisingly often:

  • Reset credentials, including the ones no human types. Service accounts, API keys, the backup system's login, cloud access keys, and in a Windows domain the keys that sign authentication tickets. CISA's checklist calls for password resets for all affected systems and accounts once the environment has been cleaned and rebuilt; resetting before the attacker is out just hands them the new passwords.
  • Close the way in. If the attacker entered through an unpatched remote-access appliance or a phished password without MFA, restoring everything without fixing that is an invitation to come back.
  • Watch for return. After recovery, keep heightened monitoring on the indicators and the accounts involved for weeks. Attackers who lose access often try the same door again, and persistence mechanisms are easy to miss.

Recovery itself is restoring service in priority order, on clean systems, from clean data, and validating before reconnecting: are the systems patched, are the accounts reset, do the applications work, and do the logs look normal? A system put back early that is still compromised is the start of incident number two.

Instrument 1 · Tabletop drill

Run the first day yourself

A fictional company, Example Logistics (example.com, 120 staff), gets an alert on a Monday morning. You lead the response through eight decisions. Every choice changes a scoreboard by fixed rules: uncontained infections grow by 35% an hour (faster if the attacker is tipped off), each action takes clock time, and later questions depend on what you did earlier. At the end you get an after-action timeline to review. There are no trick answers, but some choices only show their cost later.

Clock08:40
Machines compromised1 of 12
Evidence preserved100%
Communication50%
Downtime so far0 h

The numbers are a teaching model, not a measurement of real attacks: real dwell times, spread rates and recovery times vary enormously. The rules are the point: speed matters, scope before you cut, evidence before you wipe, coordinate before you announce, and fix the way in.

Ransomware resilience

Backups the attacker can't reach

Ransomware turns an intrusion into an emergency by encrypting the files a business runs on. For several years now it has usually come with a second threat: the attackers copy data out first and threaten to publish it. CISA's guide calls the combination double extortion, and notes that some groups skip the encryption and only threaten to leak. That matters for planning: good backups solve the encryption half, but they do nothing about data that has already left. This page gives no guidance about dealing with the criminals; CISA's and the FBI's advice is to report the incident and seek help, starting from the pages listed under Communication below.

The encryption half is where you have the most control, and the core idea is old: keep copies. CISA's backup paper states the 3-2-1 rule: three copies of any important file (the original and two backups), on two different media types, with one copy offsite. It was written for floods, fires and failing disks. Ransomware adds a new hazard, because the thing destroying your data can also log in. Attackers who reach administrator level routinely look for backups and delete or encrypt them before launching the main attack; CISA's guide says exactly that, and it is why the guide stresses that backups be kept offline. MITRE ATT&CK catalogs the behaviour as T1490, Inhibit System Recovery, next to T1486, Data Encrypted for Impact.

So the modern reading of 3-2-1 adds conditions about reachability:

  • At least one copy is offline or immutable. Offline means physically or logically disconnected: a rotated drive in a drawer, a tape on a shelf. Immutable means the storage itself refuses to delete or change a backup until a set date, whoever asks, which many cloud object stores and backup appliances offer as a "lock" setting. CISA notes immutable storage must be configured carefully and may not meet every compliance need, so read the fine print.
  • The backup system does not trust the domain. If the backup console accepts the same administrator account that runs every server, then whoever steals that account owns your backups too. Separate credentials, with their own MFA, ideally not joined to the main directory at all.
  • Backups live on their own network segment, reachable only from the backup servers that need them (Chapter 5 covers segmentation).
  • Retention is longer than an attacker's likely dwell time. If someone was inside for three weeks and you keep two weeks of backups, every copy you have contains their foothold.
  • Restores are tested. CISA's first recommendation is to "regularly test the availability and integrity of backups in a disaster recovery scenario." A backup you have never restored is a hope.

Two numbers describe how much a backup scheme can promise. The recovery point objective (RPO) is how much data you can afford to lose, measured in time: if your last good backup is from midnight and you are hit at 3 a.m., you lose three hours of work. The recovery time objective (RTO) is how long you can afford to be down. The arithmetic is short:

Data lost = time of attack − time of last clean, surviving backup
The worst case for a schedule is one full interval: back up every 24 hours and you can lose up to 24 hours of work. But "surviving" and "clean" carry most of the weight: a backup the attacker deleted, or one taken after they arrived, doesn't count.
Restore time ≈ data size ÷ restore throughput + fetch time + rebuild & verify time
2,000 GB at 200 MB/s is 10,000 seconds, under 3 hours of pure copying. Rebuilding servers, fetching media from offsite and checking everything before reconnecting usually take longer than the copy.
RPO and RTO on a timeline A timeline with the last clean backup, then the attack, then service restored. The gap from backup to attack is data lost, compared with the RPO. The gap from attack to restoration is downtime, compared with the RTO. last clean backup attack service restored DATA LOST · compare with RPO DOWNTIME · compare with RTO RPO looks backward from the attack; RTO looks forward
Instrument 2 · Backup survival simulator

Will any copy survive?

Configure the primary data plus up to three backup copies, then let a simulated ransomware event happen on day 100 at the hour you choose. The attacker arrived dwell time days earlier. The simulator lists every snapshot each copy still holds, works out which the attacker could destroy (online copies whose console trusts the stolen domain administrator account), which were taken after the intruder arrived and so contain their foothold, and finds the best clean restore point. Try the presets first.

Best clean restore point–
Data lost (RPO achieved)–
Back in service (RTO achieved)–
Snapshots surviving · clean–

    Model assumptions, stated so you can disagree with them: backups run on a fixed schedule from midnight; the attacker holds domain admin and destroys every online copy whose console accepts it; online copies with separate credentials survive here but remain reachable over the network; immutable and offline copies survive; any snapshot taken after the intruder arrived is treated as tainted. Cloud restores run at half the local throughput; offsite removable media adds 4 hours to fetch.

    Go deeper: "tainted" doesn't mean useless

    The simulator treats a snapshot taken after the intruder arrived as tainted because restoring a whole system image from it brings back whatever they installed. But the business data inside it may be perfectly good, and much newer than the last clean copy. Responders often combine the two: rebuild systems from clean images or the last clean backup, then carefully restore data files only from a later snapshot, after scanning them and with executables and scripts excluded. CISA's guide warns that ransomware is sometimes dropped to hide other activity, and that responders must identify any dropper malware before rebuilding from backups. That is a judgment call for the people doing the investigation; the point here is that long retention gives them options.

    Communication

    Who to tell, and when

    Communication is a workstream with its own owner, not something the technical lead does between commands. It has four audiences.

    • Inside the organization. Leadership needs short, regular, honest status: what we know, what we don't, what we are doing, when the next update is. Staff need to know what to do (don't use the VPN, expect a password reset) without being told details an attacker reading the mail could use. Use the out-of-band channel for anything sensitive.
    • Customers and partners. If their data or service is affected, they will hear eventually. Plain, accurate, early statements age better than reassuring ones that have to be withdrawn. Don't guess at scope in public before you know it.
    • Regulators and contracts. Many organizations have legal or contractual duties to report certain incidents, especially ones involving personal data, often within fixed deadlines. Which duties apply depends on your sector, your location and the data involved. This page can't tell you yours: check your obligations with counsel before an incident, and write the deadlines into the runbook.
    • Law enforcement and national agencies. In the United States, CISA takes reports at cisa.gov/report and the FBI at ic3.gov; in the United Kingdom, the NCSC at report.ncsc.gov.uk. CISA's ransomware checklist advises consulting federal law enforcement even if you can recover on your own, because researchers sometimes find flaws that make decryption possible for some ransomware families.
    Post-incident

    The blameless review

    The cheapest security improvements an organization will ever get come out of an honest review of its last incident. The review has a fixed shape: a timeline of what happened and what people did, assembled from the logs and the incident notes; the contributing causes, which are almost never a single root cause; what went well; and action items, each with an owner and a due date, tracked until they are closed.

    "Blameless" is the part people misunderstand. It does not mean nobody is accountable. It means the review assumes people acted reasonably given what they knew at the time, and asks why the system let a reasonable action cause harm. Google's site reliability engineering book describes blameless postmortems this way: focus on contributing causes without indicting individuals, because people who expect punishment stop reporting problems, and an organization that stops hearing about problems stops fixing them. If the review concludes "Ana clicked a phishing link," it has found nothing: the useful questions are why one click was enough, why it took four hours to notice, and why the backup console trusted the same account.

    Rev. 3 adds a point the old loop drew poorly: lessons shouldn't wait for the end. If on day two you discover a gap in logging, fix it on day two. The review collects and prioritizes; it isn't the first time anyone writes a lesson down.

    At home and in a small office

    The same ideas, scaled down

    None of this needs a security team. An individual or a five-person business can do the parts that matter most in an afternoon.

    Before anything happens

    • Write one page: who to call (your IT helper, bank, insurer), where your backups are, how to restore them.
    • Keep 3-2-1 with one copy disconnected: an external drive you plug in weekly and unplug afterwards, plus a cloud backup with versioning.
    • Turn on MFA for email, banking and the backup service itself, with a different password from your computer login.
    • Restore one file from each backup every few months, so you know it works.

    When something seems wrong

    • Disconnect the machine from Wi-Fi or the network; don't rush to power it off or wipe it.
    • From a different device, change the passwords of your email and money accounts first.
    • Write down what you saw and when, with photos of any messages on screen.
    • Report it (links above), and to your bank if money or cards are involved.
    • Rebuild from a clean install and restore your files from a backup taken before the trouble began.

    The UK NCSC's Small Business Guide covers the same ground at this scale, including backups and incident planning, and its Incident Management collection grows from there to larger organizations.

    Where this comes from

    Sources