Why Large Organizations Struggle With Disruption, and What to do About It

Seemingly overnight, disruption has allowed challengers to threaten the dominance of companies and government agencies as many of their existing systems have now been leapfrogged. How an organization reacts to this type of disruption determines whether they adapt or die.

---

I’ve been working with a large organization whose very existence is being challenged by an onslaught of technology (AI, autonomy, quantum, cyberattacks, access to space, et al) from aggressive competitors, both existing and new. These competitors are deploying these new technologies to challenge the expensive (and until now incredibly effective) legacy systems that this organization has built for decades. (And they are doing it at speed that looks like a blur to this organization.) But the organization is also challenged by the inaction of its own leaders, who cannot let go of the expensive systems and suppliers they built over decades. It’s a textbook case of the Innovators Dilemma.

In the commercial world creative destruction happens all the time. You get good, you get complacent, and eventually you get punched in the face. The same holds true for Government organizations, albeit with more serious consequences.

This organization’s fate is not yet sealed. Inside it, I’ve watched incredibly innovative groups create autonomous systems and software platforms that rival anything a startup is doing. They’ve found champions in the field organizations, and they’ve run experiments with them. They’ve provided evidence that their organization could adapt to the changing competitive environment and even regain the lead. Simultaneously, they’ve worked with outside organizations to complement and accelerate their internal offerings. They’re on the cusp of a potential transformation – but leadership hesitates to make substantive changes.

The “Do Nothing” Feedback Loop
I’ve seen this play out time and again in commercial and government organizations. There’s nothing more frustrating for innovators than to watch their organization being disrupted while its senior leaders hesitate to take more than token actions. On the other hand, no one who leads a large organization wants it to go out of business. So, why is adapting to changed circumstances so hard for existing organizations?

The answer starts at the top. Responding to disruption requires action from senior leadership: e.g. the CEO, board, Secretary, etc. Fearful that a premature pivot can put their legacy business or forces at risk, senior leaders delay deciding – often until it’s too late.

My time with this organization helped me appreciate why adopting and widely deploying something disruptive is difficult and painful in companies and government agencies. Here are the reasons:

Disconnected Innovators – Most leaders of large organizations are not fluent in the new technologies and the disruptive operating concepts/business models they can create. They depend on guidance from their staff and trusted advisors – most of whom have been hired and promoted for their expertise in delivering incremental improvements in existing systems. The innovators in their organization, by contrast, rarely have direct access to senior leaders. Innovators who embrace radically new technologies and concepts that challenge the status quo and dogma are not welcomed, let alone promoted, or funded.

Legacy – The organization I’ve been working with, like many others, has decades of investment in existing concepts, systems, platforms, R&D labs, training, and a known set of external contractors. Building and sustaining their existing platforms and systems has left little money for creating and deploying new ones at the same scale (problems that new entrants/adversaries may not have.) Advocating that one or more of their platforms or systems are at risk or may no longer be effective is considered heresy and likely the end of a career.

The “Frozen Middle” – A common refrain I hear from innovators in large organizations is that too many people are resistant to change (“they just don’t get it”.) After seeing this behavior for decades, I’ve learned that the frozen middle occurs because of what’s called the“Semmelweis effect” – the unconscious tendency of people to stick to preexisting beliefs and reject new ideas that contradict them – because it undermines their established norms and/or beliefs. (They really don’t get it.) This group is most comfortable sticking with existing process and procedures and hires and promotes people who execute the status quo. This works well when the system can continue to succeed with incremental growth, but in the face of more radical change, this normal human reaction shuts out new learning and limits an organizations’ ability to rapidly adapt to new circumstances. The result is organizational blinders and frustrated innovators. And you end up with world-class people and organizations for a world that no longer exists.

Not everyone is affected by the Semmelweis effect. It’s often mid-grade managers / officers in this same “middle” who come up with disruptive solutions and concepts. However, unless they have senior champions (VP’s, Generals / Admirals) and are part of an organization with a mission to solve operational problems, these solutions die. These innovators lack alternate places where the culture encourages and funds experimentation and non-consensus ideas. Ironically, organizations tend to chase these employees out because they don’t conform, or if forced to conform, they grow disillusioned and leave for more innovative work in industry.

Hubris is managerial behavior of overconfidence and complacency. Unlike the unconscious Semmelweis effect, this is an active and conscious denial of facts. It occurs as some leaders/managers believe change threatens their jobs as decision-makers or that new programs, vendors or ideas increase the risk of failure, which may hurt their image and professional or promotional standing.

In the organization I’ve been working with, the internal engineering group offers senior leaders reassurances that they are responding to disruption by touting incremental upgrades to their existing platforms and systems.

Meanwhile because their budget is a zero-sum game, they starve innovators of funds and organizational support for deployment of disruptive new concepts at scale. The result is “innovation theater.” In the commercial world this behavior results in innovation demos but no shipping products and a company on the path to irrelevance or bankruptcy. In the military it’s demos but no funding for deployments at scale.

Fear of Failure/Risk Aversion – Large organizations are built around repeatable and scalable processes that are designed to be “fail safe.” Here new initiatives need to match existing budgeting, legal, HR and acquisition, processes and procedures. However, disruptive projects can only succeed in organizations that have a “safe-to-fail” culture. This is where learning and discovery happens via incremental and iterative experimentation with a portfolio of new ideas and failure is considered part of the process. “Fail safe” versus “safe-to-fail” organizations need to be separate and require different culture, different people, different development processes and risk tolerance.

Activist Investors Kill Transformation in Commercial Companies
A limit on transformation speed unique to commercial organizations is the fear of “Activist Investors.”  “Activist investors” push public companies to optimize short-term profit, by avoiding or limiting major investments in new opportunities and technology. When these investors gain control of a company, innovation investments are reduced, staff is cut, factories and R&D centers closed, and profitable parts of the company and other valuable assets sold.

Unique Barriers for Government Organizations
Government organizations face additional constraints that make them even slower to respond to change than large companies.

To start, leaders of the largest government organizations are often political appointees. Many have decades of relevant experience, but others are acting way above their experience level. This kind of mismatch tends to happen more frequently in government than in private industry.

Leaders’ tenures are too short – All but a few political appointees last only as long as their president in the White House, while leaders of programs and commands in the military services often serve 2- or 3-year tours. This is way too short to deeply understand and effectively execute organizational change. Because most government organizations lack a culture of formal innovation doctrine or playbook – a body of knowledge that establishes a common frame of reference and common professional language – institutional learning tends to be ephemeral rather than enduring. Little of the knowledge, practices, shared beliefs, theory, tactics, tools, procedures, language, and resources that the organization built under the last leader gets forwarded. Instead each new leader relearns and imposes their own plans and policies.

Getting Along Gets Rewarded – Career promotion in all services is primarily driven by “getting along” with the status quo. This leads to things like not cancelling a failing program, not looking for new suppliers who might be cheaper/ better/ more responsive, pursuing existing force design and operating concepts even when all available evidence suggests they’re no longer viable, selecting existing primes/contractors, or not pointing out that a major platform or weapon is no longer effective. The incentives are to not take risks. Doing so is likely the end of a career. Few get promoted for those behaviors. This discourages non-consensus thinking. Yet disruption requires risk.

Revolving doors – Senior leaders leave government service and go to work for the very companies whose programs they managed, and who they had purchased systems from (often Prime contractors). The result is that few who contemplate leaving the service and want a well-paying job with a contractor will hold them to account or suggest an alternate vendor while in the service.

Prime Contractors – are one of our nation’s greatest assets while being our greatest obstacles to disruptive change. In the 20th century platforms/weapons were mostly hardware with software components. In the 21st century, platforms/weapons are increasingly software with hardware added. Most primes still use Waterfall development with distinct planning, design, development, and testing phases rather than Agile (iterative and incremental development with daily software releases). The result is that primes have a demonstrated inability to deliver complex systems on time. (Moving primes to software upgradable systems/or cloud-based breaks their financial model.)

As well, prime contractors typically have a “lock” on existing government contracts. That’s because it’s less risky for acquisition officials to choose them for follow-on work– and primes have decades of experience in working through the byzantine and complex government purchasing process; and they have tons of people and money to influence all parts of the government acquisition system—from the requirements writers to program managers, to congressional staffers to the members of the Armed Services and Appropriations committees. New entrants have little chance to compete.

Congress – Lawmakers have incentives to support the status quo but few inducements to change it. Congress has a major say in what systems and platforms suppliers get used, with a bias to the status quo. To keep their own jobs, lawmakers shape military appropriations bills to support their constituents’ jobs and to attract donations from the contractors who hire them. (They and their staffers are also keeping the revolving door in mind for their next job.) Many congressional decisions that appear in the National Defense Authorization Act (NDAA) and in appropriations are to support companies that provide the most jobs in their districts and the most funds for their reelection. These come from the Prime contractors.

What to Do About It?
It starts at the top. Confronted with disruptive threats, senior leaders must actively work to understand:

• The timing of the threat – disruption never comes with a memo, and when it happens its impact is exponential. When will disruption happen that will make our core business or operating concepts/force design obsolete? Will our competitors get there first?

• The magnitude of the threat – will this put a small part of our business/capabilities at risk or will it affect our entire organization?

• The impact of the threat – will this have a minor impact or does it threaten the leadership or the very existence of the organization. What happens if our competitors/adversaries adopt this first?

• The response to the threat- Small experiments, department transformation, and company or organization-wide transformation – and its timeline.

Increase Visibility of Disruptive Tech and Concepts/Add Outside Opinions

• To counter disruptive threats, the typical reporting relationship of innovators filtered through multiple layers of management must be put aside.

  • Senior leaders need a direct and unfiltered pipeline to their internal innovation groups for monthly updates and demos of evidenced-based experiments in operational settings.

  • And the new operating concepts to go with it.

• Create a “Red Team” of advisors from outside their organization.

  • This group should update senior leaders on the progress of competitors

  • And offer unbiased assessment of their own internal engineering/R&D progress.

• Stand up a strategic studies group that can develop new business models/ new strategic concepts usable at the operational level – ensure its connection with external sources of technical innovation

• Create a “sensing” and “response” organization that takes actual company/agency/service problems out to VC’s and startups and seeing how they would solve them

  • However, unless senior leaders 1) actively make a point of seeing these first hand (at least biannually), and have the mechanism to “respond” with purchase orders/ OTA’s, this effort will have little impact.

Actively and Urgently Gather Evidence

• Run real-world experiments – simulations, war games, – using disruptive tech and operating concepts (in offense and defense.)

• See and actively seek out the impact of disruption in adjacent areas e.g. AI’s impact on protein modeling, drones in the battlefield and Black Sea in Ukraine, et al.

• Ask the pointy end of the organization (e.g the sales force, fleet admirals) if they are willing to take more risk on new capabilities.

These activities need happen in months not years. Possible recommendations from these groups include do nothing, run small experiments, transform a single function or department, or a company or organization-wide transformation.

What Does Organization-wide Transformation look like?

• What outcome do we desire?

• When do we need it?

• What budget, people, capital equipment are needed?

  • What would need to be divested?

• How to communicate this to all stakeholders and get them aligned?

• In the face of disruption/ crisis/ wartime advanced R&D groups now need a seat at the table with budgets sufficient for deployment at scale.

  • Consider organizing as an ambidextrous organization (e.g. SpaceX Falcon 9 operational execution versus Starship disruptive experimentation) (see this HBR article.)

  • In addition to mangers of process, create rapid innovation teams (technologists, visionaries, senior champions)

• Finally, encourage more imagination. How can we use partners and other outside resources for technology and capital?

Examples of leaders who transformed their organization in the face of disruption include Microsoft CEO Satya Nadella and Steve Jobs from Apple, in defense, Bill Perry, Harold Brown and Ash Carter. Each dealt with disruption with acceptance, acknowledgment, imagination and action.

Much more to be said about transformation in future posts.