Most days a security team does not fight anyone. It reads. Hundreds of advisories arrive every week, a few of them about software it actually runs, and one of those may be about to be used against it. The watch floor is the habit that turns that stream into a short, ordered list, gets the right fixes in fast, proves they landed, and then checks whether anyone got in before the door was shut.
Common mix-up: a patch ends the exposure, not the incident. Installing the fix closes the door from that moment on; it says nothing about who walked through it before. A watch floor treats "patched" as the start of a second job: verify the fix is really live everywhere, then hunt back through the logs for anything that happened while it was not.
Before you start: this chapter assumes the basics of CVE, CVSS and KEV from Threats, risk & attack surface, logging and alerting from Logs, detection & ATT&CK, and the incident lifecycle from Incident response & recovery. For the networking side of "is this service reachable from outside?", read The troubleshooting toolkit on networks.jelia.nyc first.
The film version of a security operations room has dark walls, red numbers and someone shouting. The real thing is quieter. A watch floor is a standing habit of attention, held by people who take turns holding it: somebody is always responsible for reading what came in, deciding what it means here, and seeing the decision through. You can run one from a single desk if the habit is real.
This chapter follows one public example closely. RedPatch is the security-intelligence site of LIBCSystems LLC, a Long Island infrastructure firm. Its front page tracks vulnerabilities being exploited in the wild and turns them into sourced fixes for macOS, Windows 11, Windows Server and Linux; every recommendation links to the vendor's own documentation, feeds refresh twice a day, and CISA's Known Exploited Vulnerabilities catalog drives the watch list, with an "exploited right now" panel of new KEV entries. Its blog, THE WATCH FLOOR at blog.redpatch.us, is where the reasoning is written down, in dispatches such as "What a Security Watch Floor Actually Does" and "How to Read a Threat Feed Without Panicking".
Those dispatches make three points that hold for any team, of any size.
Underneath those habits sits a loop that every vulnerability goes around. The rest of the chapter walks it in order.
Each advisory goes around once. The center is what makes every step possible: an inventory of what you run and a baseline of what normal looks like.
| Step | Question it answers | Output |
|---|---|---|
| Intake | What changed in the world today, and does any of it touch software we run? | A list of advisories matched against the inventory |
| Triage | How exposed is it, how likely is it to be used, and how much does the affected machine matter? | An ordered queue with a deadline on each item |
| Patch | Can we fix it now, safely, with a way back? | A change record, a rollout and a rollback plan |
| Verify | Is the fixed version really what is running, everywhere? | Evidence per host, not a "should be done" |
| Hunt | Did anyone use the weakness before we closed it? | A finding: evidence of compromise, or none found |
| Report | Who needs to know what, and what is still open? | A status note, including exceptions |
A floor that runs around the clock changes people, and every shift change is a chance to drop context. The RedPatch dispatch on the subject calls the handoff the most important fifteen minutes of the day: the outgoing watch briefs the incoming one on what is open, what is odd, and what has been quiet for suspiciously long. A good handoff is written, not just spoken, so it survives the person going home.
A one-person floor hands off between today and tomorrow; a running list of what is open, with dates, is its shift brief. The enemy the dispatch names is complacency, the drift from "it is always nothing" to "I stopped looking", which is why checks should be scripted rather than remembered.
A new advisory arrives carrying several labels from several organizations. Each answers a different question, and the commonest mistake on a watch floor is to treat one of them as if it answered all five. The basics are covered in chapter 1; here is how they are used together.
| Signal | Who publishes it | The question it answers | What it cannot tell you |
|---|---|---|---|
| CVE record | A CVE Numbering Authority (CNA): often the vendor itself. PHP is its own CNA. | Which flaw is this, which versions are affected, which versions fix it? | How urgent it is for you |
| CVSS v4.0 | The CNA, sometimes NIST's National Vulnerability Database; maintained by FIRST | How severe is the flaw in the abstract, from its technical properties? | Whether anyone is using it, or whether you are exposed |
| EPSS | FIRST, recomputed daily for every published CVE | How likely is exploitation activity in the next 30 days? | Impact; anything about your network |
| KEV | CISA | Is there reliable evidence it has been exploited in the wild? | Anything before the evidence arrives |
| SSVC fields | CISA's Vulnrichment program adds them to CVE records | Exploitation status, whether attacks can be automated, and technical impact (partial or total control) | Your exposure and asset value; you supply those |
The CVE record is the anchor: affected product, version ranges, and usually a link to the vendor's advisory. You match the version ranges against your inventory, and without an inventory nothing else here works. The supply-chain chapter covers software bills of materials, which automate that matching for code you build.
CVSS v4.0 scores severity from 0.0 to 10.0. The specification splits its metrics into four groups: Base (the flaw's intrinsic properties), Threat (how mature exploitation is), Environmental (your own requirements and controls) and Supplemental (extra context such as whether it is automatable, which does not change the score). Version 4.0 also asks publishers to say which groups were used: a score labeled CVSS-B used only the base metrics, CVSS-BT added threat, CVSS-BE environment, CVSS-BTE all three. Almost every score you see in a feed is CVSS-B, and the base metrics assume the worst about threat. That is a reasonable default, not a fact about this week.
EPSS, the Exploit Prediction Scoring System, is a model run by FIRST that gives each CVE a probability between 0 and 1 that exploitation activity will be observed in the next 30 days, updated daily, along with a percentile against all other CVEs. FIRST's own FAQ explains the meaning with the number 0.05: among many vulnerabilities scored around 0.05, expect about 5 percent to see exploitation activity. A low EPSS is not "safe"; it is a base rate.
KEV, CISA's Known Exploited Vulnerabilities catalog, adds an entry when the flaw has a CVE ID, there is reliable evidence of active exploitation in the wild, and there is a clear remediation such as a vendor update. Each entry carries a required action, a federal due date and a ransomware-use flag. KEV is the strongest signal here and also the latest: it arrives after exploitation has been seen.
SSVC, Stakeholder-Specific Vulnerability Categorization, is a decision-tree method from Carnegie Mellon's CERT/CC that CISA uses. Instead of a number it asks plain questions: is it being exploited (none, public proof of concept, active), can it be automated at scale (yes or no), and how much control does it give (partial or total)? CISA's Vulnrichment program attaches those answers to new CVE records, and SSVC's outcomes are words: Track, Track*, Attend and Act.
The two PHP2Shell records, CVE-2026-109618 and CVE-2026-109619, carry the same CVSS v4.0 vector from the PHP project. Read left to right it is a short, precise description of the flaw.
CVSS:4.0/AV:N/AC:H/AT:P/PR:N/UI:N/VC:H/VI:H/VA:H/SC:N/SI:N/SA:N → 9.2 Critical (CVSS-B)
| Metric | Value | Plain English |
|---|---|---|
| AV:N | Network | Reachable over the network; no local access needed |
| AC:H | High complexity | Success depends on getting around security measures built into the target, not just sending one request |
| AT:P | Attack requirements present | Some deployment condition must hold for the attack to work |
| PR:N · UI:N | None · None | No login and no victim action required |
| VC/VI/VA:H | High, High, High | Full loss of confidentiality, integrity and availability on the vulnerable system: code execution |
| SC/SI/SA:N | None | The score itself claims no direct impact on other systems |
The same day, CISA's Vulnrichment entries for both records said Exploitation: none, Automatable: no, Technical Impact: total. The next day EPSS put them at about 0.053 and 0.058, the 93rd percentile, and neither was in KEV. So: very severe, total control if it works, not yet seen in the wild, not easy to automate, and a one-in-twenty chance of exploitation activity within a month. Every one of those labels is accurate, and none of them says what to do. That depends on the next step.
CVSS describes the flaw, EPSS the world's behavior, KEV the evidence, SSVC a few facts in words. They are meant to disagree. A Critical CVSS with a low EPSS is the commonest case: attackers concentrate on a few flaws that are cheap to exploit against many targets.
The disagreement only causes harm when one number is used as a queue order. Sorting by CVSS fills the week with severe bugs nobody is using; sorting by EPSS alone ignores the size of the blast; waiting for KEV means acting after the evidence of attacks, which for an internet-facing system is late. All three are inputs. The decision needs two more things only you know: whether the vulnerable code is reachable, and what the machine is worth.
Signals also move: EPSS is recomputed daily, KEV grows on evidence, Vulnrichment fields are updated. Re-check open items against the latest values each day.
Triage turns labels into an order. The version used on this page is deliberately simple: three questions, each answered on a small scale, multiplied together.
The two-week exploitability figure comes from treating EPSS's 30-day probability as a steady daily rate: the chance of no exploitation in 14 days is (1 − EPSS) raised to the power 14/30. For EPSS 0.05 that gives about 2.4 percent; for 0.9 it gives about 66 percent.
| Advisory (made up) | Exposure | Exploitability | Value | Severity | Priority |
|---|---|---|---|---|---|
| VPN gateway, auth bypass | 1.0 (internet) | 1.0 (in KEV) | 3 | 0.93 | 2.79 |
| Database, privilege escalation | 0.3 (internal) | 0.005 (EPSS 0.01) | 3 | 0.88 | 0.004 |
| Mobile SDK, cert check | 0 (we do not ship it) | 1.0 (in KEV) | – | 0.91 | 0 |
The gateway wins by a factor of several hundred. The database flaw is severe but needs an attacker already inside with a login, and nobody is using it: normal patch cycle. The SDK is in KEV and scored 9.1, and irrelevant, because it is not in the inventory; the triage note is "not ours, checked against the inventory on this date". Order matters too, because crew hours are finite: a one-hour fix that removes a lot of risk should usually go before a day-long one that removes slightly more.
Imagine each open advisory leaks risk at a steady rate until it is patched, and the crew can only work on one at a time. The total risk you accumulate is the sum, over every advisory, of its rate multiplied by the time it stays open. A classic result from scheduling theory says this total is smallest when you work in order of rate divided by effort, largest ratio first, a rule known as weighted shortest job first. Swapping any two neighbors that are out of that order lowers the total, so the sorted order cannot be beaten.
Real risk is not perfectly steady (exploitation pressure grows over time, which is the next instrument's subject) and real patches can run in parallel, so the rule is a strong default rather than a theorem about your week. Its practical lesson survives: a quick fix on an exposed, exploited system should almost never wait behind a long project on a quiet one.
Ten made-up advisories have landed, each with its CVSS, EPSS, KEV status, exposure, asset value and the crew hours its fix takes. Your crew has a fixed number of hours a day for 14 days. Order the queue with the arrow buttons and mark anything you do not run as "Not ours". The chart shows the risk you carry each day, against doing nothing and against the triage formula.
Risk is in "risk-days": each open advisory's priority, summed over the days it stays open. The deadline yardstick is the federal table in the next section. Try "Highest CVSS first", then "Watch-floor triage", then try to beat it by hand.
For each advisory the instantaneous risk on day t is exposure × value × (CVSS ÷ 10) × p(t), where p(t) is 1 for a KEV entry and otherwise 1 − (1 − EPSS)t/30, the chance exploitation activity has started by then. Software marked "Not ours" in the scenario contributes nothing, whether or not you mark it. Fixes are worked one at a time in queue order; an item finishes when the crew has spent its hours on it, at your chosen hours per day. Anything unfinished at day 14 stays open to the end of the chart.
The "formula order" line drops what is not in the inventory and sorts the rest by the priority formula (with two-week exploitability) divided by hours of effort. It is a good plan, not a guaranteed best one, which is why a careful hand ordering can sometimes edge past it. The numbers are illustrative: real exposure, value and effort come from your own inventory and your own people.
A watch floor works to written deadlines, sometimes called patch service-level agreements: how many days an item of each kind may stay open. Writing them down in advance is what lets the person on duty act without asking. The most detailed public example is the one US federal civilian agencies must follow. CISA's Binding Operational Directive 26-04, "Prioritizing Security Updates Based on Risk", issued June 10, 2026, replaced the older BOD 22-01 and BOD 19-02. It keeps the KEV catalog and its entry criteria, and sets remediation timelines from four yes-or-no facts, two of which (Automatable and Technical Impact) CISA publishes through Vulnrichment for every CVE.
| Row | Publicly exposed? | In KEV? | Automatable? | Technical impact | Deadline (calendar days) |
|---|---|---|---|---|---|
| 1 | Yes | Yes | Yes | Total | 3 days & forensic triage |
| 2 | Yes | Yes | Yes | Partial | 3 days |
| 3 | Yes | Yes | No | Total | 3 days & forensic triage |
| 4 | Yes | Yes | No | Partial | 14 days |
| 5 | Yes | No | Yes | Total | 3 days |
| 6 | Yes | No | Yes | Partial | 14 days |
| 7 | Yes | No | No | Total | 14 days |
| 8 | Yes | No | No | Partial | 60 days |
| 9 | No | Yes | Yes | Total | 3 days & forensic triage |
| 10 | No | Yes | Yes | Partial | 14 days |
| 11 | No | Yes | No | Total | 14 days |
| 12 | No | Yes | No | Partial | 14 days |
| 13 | No | No | Yes | Total | 60 days |
| 14 | No | No | Yes | Partial | 60 days |
| 15 | No | No | No | Total | Fix on system upgrade |
| 16 | No | No | No | Partial | Fix on system upgrade |
Three details are worth copying at any size. "Forensic triage": in the worst cases, patching is not enough; the machine must also be checked for compromise. The deadline is dynamic: taking a system off the internet moves it out, a KEV listing moves it in. And the clock starts at the KEV listing or the agency's own discovery, whichever is first.
The highlighted row is where PHP2Shell sat on its first day: public web servers, not in KEV, not automatable according to Vulnrichment, total control. The federal deadline was 14 days. LIBCSystems took under three hours. A government deadline is a ceiling for the slowest acceptable response, not a target, and the patch clock below shows why fast matters most in exactly the cases where the labels are still catching up.
Normal change control (scheduled window, review, approval) stops well-meant changes breaking things, and is too slow for an exploited flaw on an exposed system. NIST's patch-management guide, SP 800-40 Revision 4, asks for a written maintenance plan for four scenarios: routine patching, emergency patching, emergency mitigation (a temporary fix when no patch is ready), and assets that cannot be patched at all. The emergency lane is a pre-agreed shortcut with its own guard rails, written before it is needed:
# Emergency change record (illustrative template) id: EC-2026-10-21-01 trigger: CVE-2026-109618, CVE-2026-109619 (PHP); exposed: yes; impact: total scope: PHP runtime only; application versions pinned and unchanged fix: PHP 8.2.35 / 8.3.36 / 8.4.27 / 8.5.12, from checksummed sources canary: one low-traffic site, then remaining services rollback: previous images retained; config backed up verify: version check per host, health checks, login pages render follow-up: compromise hunt back to before disclosure; status note approved by: the accountable owner (named), time-stamped
Sometimes the fixed version does not exist for your platform yet, or the system cannot be restarted today. Then the job is to cut one of the three triage factors. Reduce exposure: take the service offline, restrict it to known networks, or put it behind authentication. Reduce exploitability: apply the vendor's documented mitigation, disable the affected feature, or add a filter in front that rejects the shape of request the advisory describes. Reduce value at risk: move sensitive data or credentials off the box. Every mitigation is recorded as temporary, with the real fix still on the list. In the PHP2Shell post, the governments that took sites offline before patching were making exactly this choice. SP 800-40 calls it emergency mitigation, and lists unpatchable assets as a fourth case needing its own plan: isolate them, watch them closely, and schedule their replacement.
After disclosure, attacks do not arrive at once: someone builds a working exploit, then automated scanning ramps up. This instrument models that as attack pressure, the daily chance an exposed, unpatched machine is successfully hit, and races three teams against it, finishing in hours, days and weeks. Each fleet is patched gradually, between half and one-and-a-half times its stated time. Press Play or drag the time slider.
The "chance at least one machine was hit" figure in the verdict is the reason for the hunt in the next section: even the fast team is rarely at zero. Numbers are a toy model with made-up rates, built to show the shape of the race, not to forecast any real flaw.
Attack pressure is a smooth step: λ(t) = λmax × σ((t − Te) / w), where σ is the logistic function, Te is when a working exploit appears and w sets how quickly scanning ramps up. The chance that one machine left unpatched until time τ has been hit is 1 − e−Λ(τ), where Λ(τ) is the area under λ from 0 to τ. Each team's machines are patched at times spread evenly over its window, so the expected number lost is the fleet size times the average of that chance across the window.
The asymmetry is the lesson. Before the exploit exists, patching a day sooner buys almost nothing; after it exists, every day costs a lot. Since you rarely know which side of that line you are on, and the public signals (KEV, EPSS) usually lag the real curve, the only safe policy for exposed, total-control flaws is to act as if you are already on the steep side.
"Deployed" is a claim; verification turns it into evidence. Check on every machine that the running process is the fixed version, not just the package on disk: long-running services keep old code in memory until restarted, and containers run the old image until recreated. Check the service still works for real users by loading pages and sign-in screens, not just that a port answers. Then re-scan from outside.
Then comes the part that is easy to skip: the compromise hunt. The window of risk did not open at disclosure. It opened whenever the vulnerable code went live, and the attacker may have known before you did. The hunt looks back over that window, as far as your logs reach, for signs the weakness was used. It is the "forensic triage" that BOD 26-04 demands for its worst rows, and it is the bridge from this chapter to detection and, if anything turns up, to incident response.
The hunt window starts as far back as your logs go, not at the disclosure date. Each lane is a different kind of evidence an intruder tends to leave.
A hunt can only look back as far as its evidence goes. If web logs are kept for seven days and the flaw was in use for three weeks before disclosure, the most a team can say is "no evidence in the last seven days". That is why the RedPatch dispatch on the defensive mindset calls logs institutional memory: logs from before an event are the baseline that makes the event visible at all.
Retention is cheap compared with the cost of an unanswerable question. Keep request logs, authentication logs and change records for long enough to cover a realistic dwell time, keep them somewhere the server being investigated cannot rewrite, and keep known-good checksums of what should be on disk. The detection chapter covers what to log and how to search it.
On October 21, 2026, the PHP project published two critical flaws in PHP itself, the language runtime behind a large share of the web. CVE-2026-109618 ("RCE via OOM corruption") and CVE-2026-109619 ("RCE via stale global state") are both memory-handling bugs inside the interpreter: in some situations request memory is left uninitialized after an out-of-memory error, or global state keeps pointing at memory that has already been freed, and later processing reads it. Both records describe remote code execution, and both list the same fixed versions: PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35. (The memory-safety chapter explains this class of bug.)
LIBCSystems published an account of its response the next day on its blog, "PHP2Shell: How We Patched Everything in Three Hours While Governments Took Sites Offline". Citing Reuters and CNBC reporting, the post describes organizations, including several European government services, taking websites offline rather than wait. What follows is what the post says the company did, step by step, mapped onto the loop.
Record times are from the CVE records on cve.org and CISA's Vulnrichment entries; the build time and everything on the right-hand side is from LIBCSystems' post.
| Loop step | What the post reports | The general lesson |
|---|---|---|
| Intake | Fourteen of the company's self-hosted services run PHP: mail, cloud files, webmail, blogs, admin panels. | The inventory answered "are we affected?" in minutes. |
| Triage | Internet-facing, unauthenticated code execution, Critical 9.2: patch the same day, not in the 14-day window. | Exposure and impact decided it before KEV or EPSS caught up. |
| Patch | Official patched container images were not yet published, so the team built them from Docker's official recipes, checking every PHP download against php.net's published checksums. Fixed builds committed at 16:34 UTC, about three hours after the fix was released. | Do not wait for someone else's packaging if you can build safely; verify what you download. |
| Patch | Only PHP changed. Nextcloud, Roundcube, phpMyAdmin and WordPress stayed pinned to the exact versions already running, with the same extensions, checked old against new. | Change only what is broken. |
| Patch | Old images kept, every changed file backed up, the whole change committed. | A way back for everything. |
| Verify | The automated estate check passed 91 of 91 sites and services; every cloud reported healthy; every login page confirmed to render. | Evidence, not assumption, and checks that test what users see. |
| Hunt | About 2.7 million web requests reviewed back to October 6; no crashed PHP workers, planted files, rogue processes, new scheduled jobs or keys; WordPress and Nextcloud checksums matched; every account audited and every new one traced to the team's own work. | Look back well before disclosure, across several kinds of evidence. |
| Hunt | Scanners had sent requests matching the attack's shape to eleven sites; each was refused by the web server's request-size limit before PHP saw it. | "Attempts seen, all blocked" names the control that worked. |
| Report | Verdict: no evidence of compromise. Older housekeeping found by the audit (over-broad permissions, a stale admin account or two) fixed the same day. One exception named: the architect's own Mac, where Homebrew still shipped PHP 8.5.11 that day; a mitigation config was applied, its web server listens only to the machine itself, and Homebrew published 8.5.12 the next morning. | "Everything is patched" should mean everything. Name the exception and its mitigation. |
Two things stand out for a defender. First, the public signals said "wait": not in KEV, Vulnrichment exploitation none, EPSS around 0.05. The decision to move at once came from the local factors, exposed and total control on systems that hold mail and files, which is exactly the triage formula's argument. Second, the post's own closing advice is the loop in four lines: find every copy of PHP you have, including forgotten ones on admin machines; upgrade to a fixed version (a WordPress auto-update will not do it, because this is the runtime, not a plugin); check logs, files and accounts for anything before the patch; and if you cannot patch yet, take it offline.
The post also says who did the work: one architect decided and signed off, and Claude Code, an AI agent, carried out the rebuild, verification and hunt. Whatever the tools, the shape is the one this chapter teaches: a named person accountable, a written record, evidence at every step.
The last step of the loop is a short, accurate message to the people who need it. Different readers need different things, and the content should be written once and trimmed for each.
Good status notes give times with a time zone, separate facts from assessments, say what was checked rather than only the conclusion, and list the exceptions, because the exception is what will matter if anything goes wrong later. The LIBCSystems post led with the one machine still waiting, and why it was safe to wait.
# Status note (illustrative)
Subject: PHP critical flaws (CVE-2026-109618/-109619): patched, no evidence of compromise
Affected: 14 internet-facing services ran a vulnerable PHP version.
Action: All 14 upgraded to fixed PHP by 16:34 UTC Oct 21; application versions unchanged.
Verified: Version checks and health checks on every service; login pages render.
Hunt: Web, file, process and account review back to Oct 6. No evidence of compromise.
Attack-shaped requests to 11 sites were refused by a request-size limit.
Open: 1 internal laptop awaiting its package manager's fixed build; mitigated, not exposed.
Next check: Re-verify after the laptop update; review the KEV and EPSS status daily for 2 weeks.
None of the loop needs a big team. A one-person floor can do it with four habits. Keep an inventory, even a plain list, of what runs where and which version. Subscribe to the KEV catalog's updates and your key vendors' security advisories, and read them against that list once a day. Write a two-line deadline policy: exposed and exploited, same day; everything else, the next routine window. And after any emergency patch, spend an hour on the look-back before writing "done".
The RedPatch front page is built around the same idea for individuals: start with what is being exploited now, find the fix in the vendor's own documentation for the platform you run, and apply it.