Exhaustive Testing Remains Impossible Even as LLMs Find More Bugs

# What the author observed LLMs are already finding bugs in mature, stable software. That shows exhaustive testing has always been impractical. Generative models accelerate discovery of faults, but they do not remove the fundamental constraint: you cannot observe every possible input, state, or interaction of a nontrivial system.

# Why exhaustive testing is impossible

# What LLMs actually add LLMs help by generating test cases, surfacing unfamiliar failure modes, and suggesting fixes. They are effective with text, so they can accelerate generation of hypotheses about what might go wrong. They do not, however, verify by themselves that a solution is correct in the real-world sense: they lack the external context necessary to answer whether the product fits the customer problem or whether the delivered behavior matches requirements.

# Two distinct challenges remain

• Verification: confirming the system implements the specified behavior. This is partly addressable with tests and automated checks, but it still requires selecting what to verify.
• Validation: confirming the system meets user needs and solves the intended problem. Validation requires business and market context outside the code and cannot be delegated to models alone.

# Practical implications for teams Adopting LLM-generated code at speed without accompanying mental models and risk assessment erodes the human learning loop. If developers and product people defer decisions to models, the team loses the ability to steer, prioritize risk, and interpret trade-offs.

Risk-based testing remains the right approach. Decide which behaviors and scenarios matter to your product and market, and generate or run tests that align with those priorities. Treat LLM outputs as accelerants for hypothesis generation, not as final authorities.

# Recommendations

• Keep humans in the loop for decision-making about what to test and why. Developers, product managers, founders, and testers all participate in this work.
• Define risk profiles tied to business goals and use those profiles to prioritize test generation and execution.
• Use LLMs to expand coverage where they suggest plausible failure modes, but require human review of proposed changes and mental-model validation before merging.
• Maintain feedback loops: when a model-produced test or fix finds a bug, capture the reasoning and update the team's understanding of system risks.

# Bottom line