comdivision: blog

How Many Changes Can You Absorb?

Written by Yves Sandfort | Nov 22, 2026, 10:25:06 AM

 

A few weeks ago I wrote about whose clock we are patching on. This week the question changed its shape. It is no longer only about who sets the date, it is about how many changes a platform, a team and a change process can actually absorb before the fix itself becomes the outage. AI found more holes this week than ever, and at the same time the reported reason for some of the most visible outages in the US was not an attacker at all. It was the rush to close those holes.

What did this week actually show us?

Start with the outages. According to Fortune, a spokesperson for Senator Mark Warner's team linked "the botched implementation of those patches" to an outage of an internal communications system at the NSA and to the "banking outages." Officials, Fortune writes, have made clear that the outages are "not a result of a coordinated cyberattack." The Washington Sun, as summarized by Slashdot, quotes an anonymous congressional official: "They're trying to push through a lot of patches all at once -- more than the system was designed for." To be fair, the picture is not clean. One of the affected banks spoke of "a software bug in on-premise hardware systems" and said it did not find or fix that bug with AI, the NSA has said routine maintenance caused its network problems, and the Washington Sun report rests on anonymous sources. Christiaan Beek of Rapid7 put it well: "Complex systems fail in boring ways far more often than because of an adversary." Nikesh Arora, CEO of Palo Alto Networks, added that "some of the recent outages are a consequence of a deluge of patches," and in a separate post that "most organizations do not have capacity to patch at the speed needed, especially the smaller ones."

Then the largest office cloud. In its preliminary post-incident report, as reported by The Register, the provider traced a day-long email outage to a defect introduced in a July update, a lock contention problem that got worse on machines with more cores. "Our load testing tended to focus on weaker hardware with fewer cores," the report says, and in the end more than 80 percent of the servers were unresponsive. The sentence I keep coming back to is a different one: "A straightforward rollback was also not possible because the defect was months old."

And then, closer to home for everyone running VMware. Broadcom published VMSA-2026-0009 for an authentication bypass in vCenter with a CVSS score of 9.8. According to Rapid7, Mandiant and Google Threat Intelligence confirmed in-the-wild exploitation by a state-linked actor "as early as September 2026," Broadcom lists no workaround, and "the patches cannot be applied using vCenter Quick Patch." Fixes exist for the 9.1, 9.0 and 8.0 lines and for VCF 5.2. In parallel, VMSA-2026-0010 covers an Apache mod_http2 flaw in VCF Operations, Aria Operations and VCF Operations HCX, where for now there are only workarounds that disable HTTP/2. And the Broadcom KB for VCF Operations says it plainly: "Every upgrade also replaces the shipped vcops-photon-apache.conf," so after each upgrade you repeat the workaround on every node.

Is finding still the hard part?

Honestly, no. Look at what else happened in the same seven days. Google's threat intelligence group wrote that in the past quarter it "identified threat actors using LLM frameworks to fully automate vulnerability discovery and exploitation without a human in the loop for the first time." The researchers who found Java4Shell, the new OpenJDK flaw, report via Qualys that they found it "with AI assistance in about an hour," had a first working exploit two hours later, for "under $200 in AI token costs." And the NGINX HTTP/2 flaws disclosed on Wednesday were "independently uncovered" by five different teams, one of them using an AI harness.

Five teams, the same bug, the same week. That is what cheap discovery looks like, and it means our queue gets longer every week while the change window stays exactly the same size.

Qualys adds one detail that every operations team should print out: "Many products and container images bundle their own JDK," which "operating system updates do not touch." So even the number of things you have to change is larger than your patch tool believes.

Can the patch be the outage?

Yes, and we always knew it. Every change carries three costs that do not show up in a CVE score. It needs someone to test it, it needs a window in which something can break, and it needs a way back. When findings arrive at machine speed and changes are still done at human speed, one of those three gets skipped. Usually the testing. Sometimes the way back.

The office cloud report is the quiet lesson here. A change that runs unnoticed for months is no longer a change you can roll back, it has become the platform. A rollback plan has an expiry date, even if nobody wrote one on it. And the bank and NSA stories, whatever the final root cause turns out to be, show the other side: when the pressure is high enough, organizations push changes faster than their systems were designed for, and the outage is self-inflicted.

Yes I know, the answer cannot be to patch slower. A vCenter that was exploited for two months before the advisory is not something you leave for the next quarterly window. The answer is to treat change capacity as what it is: a scarce resource that somebody has to manage on purpose.

Which four dials decide your change capacity?

The way I look at it, there are four dials, and most organizations have never set any of them consciously.

1. Triage: what happens today. Not everything is "patch now." For each finding there are three honest options: patch today, shield today and patch in the next window, or accept the risk for a defined time with a name and a date attached. The management plane belongs in the first group. Something that is only reachable from a jump host may be fine in the second. What does not work is a fourth, unofficial option called "we'll see."

2. Blast radius: how much breaks if it goes wrong. A change that hits everything at once has no test except production. Staging by domain, by cluster, by a handful of hosts first is not bureaucracy, it is how you find the July bug in July instead of in November.

3. Reversal: how long the way back stays open. Before the change, decide how long you can still roll back and what the plan is after that point. Snapshots, backups of the management components, a tested restore, and a clear owner for the moment when "back" is no longer an option.

4. Capacity: how many changes the team can carry. If the team can safely test and roll out ten meaningful changes a week, then the eleventh is a decision, not a ticket. Someone has to say which one waits. If nobody says it, the change advisory board says it by accident, or the outage says it for you.

So why would I run this on VCF?

Because all four dials are much easier to turn when the management plane, the hypervisor, storage and network share one lifecycle instead of five tools with five calendars. VCF does not take the decisions away from you, but it gives you the places to make them.

A few concrete points, as Broadcom describes them. On staging: in VCF 9.1 you upgrade the management domain first, and workload domain upgrades "can be performed at any time after the management domain of the respective instance is already upgraded," even later "as a Day-N procedure." That is dial two built into the platform. Within a cluster upgrade, "you can now select specific hosts rather than the entire cluster." On preparation: Broadcom's documentation has long required a precheck before applying an update, and in 9.1 "prechecks are now conducted using native VCF component capabilities" rather than relying solely on SDDC Manager. On pace: Express Patches "are released monthly and are downloaded and applied through VCF Operations," described as "modular, targeted updates that address critical CVEs and bug fixes between major rollups," and for eligible patches "the host applies eligible patches without a reboot." That last part matters for dial four, because a change that does not need a reboot is cheaper to carry.

None of this replaces the decision. The vCenter fix this week cannot go in via Quick Patch, so it needs a real window, and the mod_http2 workaround has to be put back after each upgrade. That belongs in the runbook, not in somebody's memory.

How do we approach this at comdivision? With the same three steps we use for every VCF platform: Architect, Educate, Operate. In this context that means designing the lifecycle into the architecture from day one, domains, staging order, backup and restore of the management components, so that the four dials exist before the first critical advisory. It means enabling the internal team to run that lifecycle confidently instead of depending on one person who knows the order. And it means supporting operations when the advisories pile up. [ERGÄNZEN: konkretes Vorgehen von comdivision bei Lifecycle und Patch-Management in VCF, z. B. Upgrade-Planung, Runbooks oder Betriebsunterstützung, nur belegte Angaben] We are highly specialized in the VMware ecosystem, especially VCF, and deliver clearly defined results at a fixed price rather than open-ended hours, and a lifecycle you can actually execute under pressure is exactly that kind of result.

What should IT leadership ask this week?

  • Is the vCenter fix from VMSA-2026-0009 in, and if not, who decided that it can wait, and until when?
  • How many changes can our team safely test and roll out per week, and who decides which one waits when the list is longer?
  • For our last three critical changes, how long could we still have rolled back, and did anybody know that date in advance?
  • Where are workarounds in our environment that the next upgrade will silently remove, and are they in the runbook?
  • How many Java runtimes and other bundled components do we run that our patch tool does not see?

Decide or go home...

This week showed both ways to lose. You can lose because you patched too slowly, and a management plane was open for two months. And you can lose because you patched too fast, and your own change took down the thing you wanted to protect. The difference between the two is not courage and not tooling. It is whether somebody set the four dials before the advisories arrived.

Finding is solved. Changing is the job now. Decide how much change you can carry, decide what comes first, and make the platform carry the rest. #DecideOrGoHome

At comdivision we spend most of our time helping organizations design and run VMware Cloud Foundation, from the target architecture to day-two lifecycle and patching. If you want to talk about how much change your platform can absorb, reach out to me.

Sources

  • Emily Forlini, "Behind the recent wave of IT outages: AI is driving an unprecedented number of software patches", Fortune, 18.11.2026: fortune.com
  • BeauHD, "Bank CEOs, NSA Tell Congress That AI-Driven Patches Are Behind Outages", Slashdot, 18.11.2026: slashdot.org
  • Ana-Maria Stanciuc, "Banks and the NSA reportedly blame US outages on patches for AI-found flaws", The Next Web, 18.11.2026: thenextweb.com
  • "AI's ability to find cyber flaws outpacing firms' capacity to fix them: Palo Alto Networks' Nikesh Arora", YourStory, 11/2026: yourstory.com
  • Richard Speed, "Microsoft blames monster email outage on software defect introduced in July fix", The Register, 16.11.2026: theregister.com
  • Rapid7, "CVE-2026-123890: Critical Authentication Bypass in VMware vCenter", 19.11.2026: rapid7.com
  • Sulochana, "Chinese Hackers Exploited vCenter Zero-Day CVE-2026-123890 Months Before Broadcom Patch", Cyber Kendra, 20.11.2026: cyberkendra.com
  • Simon Sharwood, "Flaws in VMware and KVM need attention, if you have time while fixing Java, PHP, and everything else", The Register, 20.11.2026: theregister.com
  • Broadcom KB 463005, "Manual interim workaround for CVE-2026-119618 (Apache mod_http2 vulnerability) on VCF Operations", accessed 21.11.2026: knowledge.broadcom.com
  • Aurora Blum et al., "GTIG AI Threat Tracker: Agentic Threat Activity Crosses a Significant Threshold", Google Cloud Blog, 16.11.2026: cloud.google.com
  • Saeed Abbasi, "How to Find, Mitigate, and Patch the Critical Java4Shell Flaw (CVE-2026-114938) Now", Qualys, 17.11.2026: blog.qualys.com
  • Nir Ohfeld, Tomer Schwartz, "Unauthenticated Remote Code Execution in NGINX: CVE-2026-123633 and CVE-2026-123007", Wiz, 18.11.2026: wiz.io
  • Jonathan McDonald, "Navigating the AI Threat Era: Upgrading to VMware Cloud Foundation 9.1 and Applying Express Patches", VMware Cloud Foundation Blog, 01.10.2026: blogs.vmware.com
  • Broadcom TechDocs, "Upgrading to VMware Cloud Foundation 9.1.x", accessed 21.11.2026: techdocs.broadcom.com
  • Broadcom TechDocs, "What's New in VCF Operations 9.1", accessed 21.11.2026: techdocs.broadcom.com
  • Broadcom TechDocs, "Perform Update Precheck in SDDC Manager", accessed 21.11.2026: techdocs.broadcom.com