Design Phase Corruption: Why the Flaws You Cannot See Are the Ones That Sink You
Boeing's 737 MAX was not a manufacturing failure. It was a design phase corruption that propagated invisibly. The same pattern is emerging in AI-augmented decision-making.
KEY TAKEAWAYS
- Design phase corruption is a reasoning flaw introduced at the earliest stage of a decision - where assumptions form and structural logic is set - that propagates invisibly through every downstream system
- Boeing's 737 MAX was not a manufacturing failure - it was a design phase corruption that passed through testing, certification, and training without detection, ultimately killing 346 people
- The 100x cost multiplier (IBM, NASA) confirms that flaws caught in design cost exponentially less than the same flaws discovered in production
- Research shows a negative correlation between frequent AI usage and critical thinking - the mechanism by which design phase corruption enters founder decisions
- Four practical countermeasures protect the design phase: independent framing, reasoning audits, adversarial checkpoints, and selective cognitive load
The Flaw That Passed Every Test
On 29 October 2018, Lion Air Flight 610 crashed into the Java Sea thirteen minutes after takeoff. 189 people died. Less than five months later, on 10 March 2019, Ethiopian Airlines Flight 302 crashed six minutes after takeoff. 157 people died.
Both aircraft were Boeing 737 MAX 8s. Both crashes were caused by the same system: the Manoeuvreing Characteristics Augmentation System (MCAS). Most analyses focus on the software, the sensors, the regulatory failures. They miss the structural fact: MCAS was not a manufacturing defect. It was not an assembly error. It was not a maintenance failure.
It was a design phase corruption. And that distinction changes everything.
Boeing had mounted larger, more fuel-efficient engines on the 737 airframe. The new engine placement created an aerodynamic tendency for the nose to pitch upward under certain conditions. Rather than redesign the airframe - which would have required expensive recertification - Boeing chose a software fix. MCAS would automatically push the nose down when sensors detected excessive pitch.
Here is where the corruption entered. Boeing originally designed MCAS with a narrow scope, using inputs from two Angle of Attack (AOA) sensors. Then - late in development, under schedule pressure - engineers expanded the system’s authority and reduced its sensor inputs from two to one. One sensor. One point of failure. That decision, made in a design review and not on a factory floor, propagated through testing, through certification, through pilot training manuals that never mentioned MCAS existed, and ultimately through two cockpits where pilots fought a system they did not know was operating.
MCAS was not a manufacturing defect. It was a design phase corruption that propagated through testing, certification, and training without detection.
The design phase is where the structural logic of a system gets set. Everything downstream - manufacturing, testing, deployment, operation - builds on that logic. When the logic is sound, the downstream processes work. When the logic is corrupted, the downstream processes faithfully execute a flawed design.
346 people died because a design phase corruption passed every downstream checkpoint without triggering a single alarm.
In The Handoff Tax, I described the cost that accumulates at the boundaries between human and AI work - context degradation, intent drift, knowledge leakage. But reducing the Handoff Tax only helps if what crosses the boundary is sound. Design Phase Corruption is the upstream problem: the structural flaw that enters before the first handoff and propagates through every one after it.
The 100x Rule: Why Early Flaws Cost Exponentially More
Engineering has known this principle for decades. Research from IBM’s Systems Sciences Institute and confirmed by NASA’s technical reports established what is commonly called the cost multiplier rule: a defect caught during the design phase costs roughly 100 times less to fix than the same defect discovered in production.
The exact multiplier varies by domain - software, aerospace, industrial systems all show different numbers. But the direction of the curve has been confirmed repeatedly: the cost of fixing a flaw increases exponentially the later it is discovered. A requirements error caught during design review costs hours. The same error caught in deployment costs weeks. Caught by a customer - or a crash investigation - it costs careers, reputations, and sometimes lives.
The reason is structural, not bureaucratic. A design phase flaw becomes embedded in everything built on top of it. Fixing the flaw means unwinding every layer that assumed the flaw was a feature. The later you find it, the more layers you have to unwind.
The old frame treats quality as a production concern. Inspect the output. Test the product. Catch defects at the end of the line. This is the quality control model, and it works for manufacturing defects - a misaligned bolt, a contaminated batch, a miscalibrated instrument.
The new frame treats quality as a design concern. The most expensive defects are never manufacturing errors. They are reasoning errors that enter at the design phase and propagate undetected because every downstream process assumes the design is sound. This is the quality architecture model, and it is the one that matters for AI-augmented decisions.
The Pattern Repeating in AI-Augmented Decisions
Now here is the parallel that should keep you up at night.
When you use Artificial Intelligence (AI) to frame a problem - to define the variables, identify the constraints, generate the options, analyse the data - you are using AI at the design phase of your decision. Think about that. The AI is not executing a decision you already made. It is shaping the structural logic that every subsequent action will build on.
If the AI’s framing contains a reasoning flaw - a missed variable, an incorrect assumption, a subtly biased analysis - that flaw becomes your design phase input. And if you accept the framing without independent verification, the corruption propagates. Your hiring decision builds on it. Your resource allocation builds on it. Your strategy shift builds on it. Every downstream action faithfully executes a flawed design.
This is not a theoretical risk. Research published in the European Journal of Information Systems found that when AI systems provide feature attributions - explanations of why they reached a conclusion - the explanations did not improve human-AI complementary decision quality. Instead, they increased people’s tendency to adopt AI advice regardless of its correctness. The explanation created a false sense of audit. The human believed they had verified the reasoning. They had not. They had verified the presentation of the reasoning.
Feature explanations did not improve decision quality. They increased adoption of AI advice regardless of correctness. The explanation created a false sense of audit.
Boeing’s engineers did not intend to create a single point of failure. They made a series of individually reasonable decisions - use software instead of hardware, simplify the sensor input, expand the authority envelope - that collectively produced a design phase corruption. No single decision looked dangerous in isolation. The corruption was in the compound effect.
The same dynamic operates when a founder makes a series of individually reasonable AI-assisted decisions. Each one looks sound. The AI provided data. The analysis seemed thorough. The recommendation felt grounded. But if the foundational framing was flawed - if the AI missed a variable or introduced a subtle bias at the problem-definition stage - every subsequent decision faithfully compounds the error.
Automation Complacency: The Mechanism of Entry
The question is not whether AI produces flawed reasoning. It does - compositional generalisation failures, missing dependencies, contradictory outputs across multi-step inference chains are well-documented limitations of current Large Language Model (LLM) systems. The question is why founders accept the flawed reasoning at the exact stage where it matters most.
The answer has a name: automation complacency. And the research on it is uncomfortable reading.
A study published in Frontiers in Psychology found a significant negative correlation between frequent AI usage and critical thinking ability. The mechanism is cognitive offloading - when AI performs reasoning tasks, humans practise them less. Over time, engagement drops. The cognitive habits that sustain independent analysis weaken. Not because you chose to stop thinking. Because the architecture of your workflow stopped requiring it.
Here is the part that should concern you: a study examining 998 researchers found that high immersion in Generative AI (GenAI) intensified the negative impact of cognitive strain rather than reducing it. Over-reliance on AI amplified mental burden. The tool designed to reduce cognitive load was increasing it - because the cognitive work shifted from doing the analysis to managing the AI’s output, verifying its claims, and reconciling its framing with reality.
Prolonged reliance on automation fosters both complacency and confirmation bias. The cognitive effort shifts from independent analysis to confirming the machine’s output. You stop asking “is this right?” and start asking “does this look right?” Those are fundamentally different questions.
For a founder running a company with 10 or 20 people, this complacency operates at the design phase of nearly every strategic decision. You are not using AI to execute decisions made through independent reasoning. You are using AI to frame the decisions in the first place. And the complacency that accumulates through repeated positive AI experiences - the “it has been right before, it is probably right now” instinct - means you are progressively less likely to audit the framing at the exact moment when audit is most valuable.
Why the Founder-Operator Is Asymmetrically Exposed
In an enterprise, design decisions pass through layers of review before reaching execution. A strategic recommendation generated with AI assistance gets reviewed by the executive team, challenged by the board, stress-tested by the risk committee. Each layer is an opportunity to catch a design phase corruption before it propagates.
A founder running a 12-person company does not have those layers. Your framing becomes your company’s strategy - directly, without intermediation. Your AI-assisted analysis becomes the basis for your resource allocation. Your problem definition becomes the constraints within which your team operates. No filter. No second opinion. No committee to push back.
This is the asymmetric exposure. The enterprise can absorb a design phase corruption because the corruption passes through multiple checkpoints before it reaches execution. You cannot absorb it because you are the checkpoint. And if automation complacency has degraded your critical evaluation of AI-assisted framing, the checkpoint is not functioning.
Boeing had review layers. They had Failure Mode and Effects Analysis (FMEA) processes. They had the Federal Aviation Administration (FAA). And the design phase corruption still propagated through all of them - partly because Boeing had delegated portions of the certification process to its own engineers, collapsing the independence between design and oversight.
When you use AI to both frame and analyse your strategic decisions, you are doing the same thing. You are collapsing the independence between the system that generates the reasoning and the system that audits it.
When you use AI to both frame and analyse your strategic decisions, you collapse the independence between the system that generates the reasoning and the system that audits it.
The Three Propagation Channels
Design phase corruption in AI-augmented decisions propagates through three distinct channels. Understanding them is the first step toward building structural defences.
Channel 1: Framing propagation. The AI frames the problem. You accept the frame. Every subsequent conversation, analysis, and decision operates within that frame. If the frame excluded a critical variable or defined the problem too narrowly, the exclusion compounds. You cannot find what you were never looking for.
Channel 2: Assumption propagation. The AI makes implicit assumptions in its analysis - about market conditions, about team capability, about competitive dynamics. These assumptions enter your mental model without being explicitly stated or examined. They become the invisible scaffolding on which your strategy rests.
Channel 3: Confidence propagation. Each positive outcome from AI-assisted decisions increases your confidence in the AI’s framing. This is the automation complacency mechanism. Confidence propagates faster than evidence, and it creates the conditions for a larger failure - because by the time the flawed framing produces a visible negative outcome, it has been compounded across dozens of downstream decisions.
Boeing’s MCAS passed every test - in simulation. The confidence in the design propagated through the organisation. The single-sensor vulnerability was a known compromise, but confidence in the overall system made the vulnerability invisible to the people who could have caught it.
Protecting the Design Phase: A Practical Architecture
This principle was tested under conditions that burned away everything non-structural. Between 2008 and 2018, I rebuilt my own operating architecture from paralysis - twice. What survived that reconstruction was structural, not motivational. The central lesson: the architecture of your decision process determines the quality of every decision that flows through it. Protect the design phase, and the downstream execution takes care of itself.
Four countermeasures protect the design phase from AI-augmented corruption. They are listed in order of structural importance.
1. Separate the Framing Phase From AI Assistance
Before you consult AI on any strategic decision, form your own structural view of the problem. Write down: what are the variables? What are the constraints? What does success look like? What would failure look like?
Then consult the AI. Compare its framing to yours. The differences are where the corruption risk lives - not because the AI is wrong, but because the delta between your framing and its framing is the territory that needs deliberate examination.
The mechanism: this preserves your independent reasoning at the design phase. Cognitive offloading research shows that the critical thinking loss occurs when you delegate the framing, not when you delegate the analysis. Frame first. Delegate second.
The diagnostic: On your last three strategic decisions, did you form your own view before asking the AI? If the AI framed the problem for you, you have no independent baseline against which to detect corruption.
2. Audit AI Reasoning at the Design Phase, Not Just the Output
Most founders evaluate AI output: does this analysis look right? Does this recommendation make sense? This is output-quality checking. It catches manufacturing defects - obvious errors in the final product.
Design phase audit is different. It means examining the AI’s assumptions, its problem definition, its variable selection - the structural inputs that shaped the output. What did the AI assume about your market? What did it leave out? What framing did it adopt without stating it?
The mechanism: this is the FMEA process applied to AI-assisted decisions. Boeing’s failure was not that MCAS produced bad output in testing. MCAS worked as designed. The failure was that the design assumptions were flawed, and no one audited the assumptions.
The diagnostic: Can you name three assumptions your AI made in its most recent strategic recommendation? If you cannot, you audited the output but not the design.
3. Build Adversarial Checkpoints
Use a second AI system - or a different prompting approach - to challenge the first system’s framing. Ask it to find flaws in the analysis. Ask it what variables were excluded. Ask it to argue the opposite position.
The mechanism: this restores the structural independence that collapses when you use AI to both frame and evaluate. Boeing’s review failure occurred because the company had been delegated the authority to certify its own designs. An adversarial checkpoint creates separation between design and audit.
The implementation can be as simple as a second conversation with a different AI, prompted to act as a sceptical reviewer of the first conversation’s conclusions. The cost is minutes. The structural protection is significant.
The diagnostic: Do your AI-assisted strategic decisions ever face structured opposition before you act on them? If not, you have a design-and-certification collapse.
4. Maintain Cognitive Load on Strategic Framing - Offload Execution
The cognitive offloading research points to a precise intervention: delegate execution to AI. Do not delegate framing. The tasks that benefit most from AI assistance - data processing, formatting, scheduling, research compilation - are execution tasks. The tasks that suffer most from AI delegation - problem definition, variable selection, strategic framing - are design phase tasks.
The mechanism: this maintains the cognitive habits that sustain critical thinking at the exact stage where critical thinking has the highest leverage. The 100x cost multiplier means that every unit of independent thinking invested at the design phase saves 100 units of correction cost downstream.
The diagnostic: What percentage of your AI usage is execution (formatting, processing, compiling) versus framing (defining problems, selecting variables, setting strategy)? If framing exceeds 30%, you are offloading the highest-leverage cognitive work.
The Design Phase Is Your Competitive Advantage
Here is the calculation that reframes the entire discussion.
The 100x cost multiplier means that the design phase is the highest-leverage point in any system. A founder who invests 30 minutes of independent critical thinking before consulting AI on a strategic decision is investing at the cheapest point in the cost curve. A founder who delegates the framing to AI and spends 30 minutes evaluating the output is investing at a point 10x to 100x further up the same curve.
The first founder catches design phase corruption before it enters the system. The second founder catches it - if at all - after it has propagated through weeks of downstream execution.
As AI gets better at execution, the relative value of human judgment at the design phase increases. The machines can process, analyse, format, and compile with improving accuracy. What they cannot do - reliably - is frame problems with the contextual richness that a founder brings to their own business. Your knowledge of your clients, your team’s real capabilities, the competitive dynamics that no dataset fully captures - this is design phase intelligence that AI cannot replicate.
Protecting that intelligence is not a defensive posture. It is the actual competitive advantage - the one asset that compounds rather than commoditises as AI gets better at everything else.
As AI gets better at execution, the relative value of human judgment at the design phase increases. Protect it.
What Design Phase Corruption Means for Your Operation
-
The most expensive flaws are invisible at the point of entry. Design phase corruption propagates through every downstream system because every downstream system assumes the design is sound. Boeing’s 346 deaths were caused by a design decision, not a production defect.
-
AI-assisted framing carries design phase risk. When AI shapes your problem definition, its reasoning flaws become your foundational assumptions. The research shows that explanations increase AI adoption without improving decision quality - a false sense of audit.
-
Automation complacency degrades the checkpoint you need most. Frequent AI usage correlates negatively with critical thinking ability. The more you rely on AI for framing, the less equipped you become to detect corruption in that framing.
-
Founder-operators absorb the full propagation. Enterprise teams have review layers. You are the review layer. Protect the independence between your design phase reasoning and AI-assisted analysis.
-
Frame first. Delegate second. The highest-leverage intervention is the simplest: form your own structural view before consulting AI. The cost is minutes. The protection spans every downstream decision.
How Sound Is Your Decision Architecture?
The Sovereignty Index measures your structural independence across 10 dimensions - including the design phase integrity and cognitive sovereignty that determine whether your strategic framing is genuinely yours. 10 questions. 10 minutes. 1 answer. A structural diagnostic, not a personality quiz.