Entropy in AI and Human Thinking: Why Certainty Is Not Accuracy

# What the story argues This piece separates three meanings of entropy and explains why treating certainty as a proxy for accuracy leads to mistakes in human reasoning and AI usage. It begins with a diagnostic example: an engineer and an AI assistant both give the same confident migration plan, yet a compatibility check shows the plan is unsupported. Familiarity and repeatable outputs made the decision feel safe without improving alignment with reality.

# Three distinct entropies

• Thermodynamic entropy: a physical quantity tied to microscopic states compatible with a macroscopic description. It governs energy, heat exchange, and the second law in isolated systems. Local entropy can decrease when compensated by the surroundings (refrigerator example).

• Shannon (information) entropy: a measure of uncertainty in a specified probability distribution. It quantifies unpredictability given a model, but does not prove that the most probable outcome is true or useful.

• Psychological entropy: a theoretical framework for uncertainty among competing interpretations and actions. As presented, it remains a model about perception and decision-making, not a direct physical law governing minds.

Each meaning requires a defined system and measurement. Claiming an organization has "high mental entropy" is metaphorical unless states, probabilities, and methods are specified.

# Why confidence can mislead The core practical point is that reduced variability or increased certainty does not confirm correctness. The article provides two mechanisms that create this illusion:

• Repetition and rehearsed procedures can make decisions feel familiar without incorporating missing constraints or new evidence.

• In probabilistic language models, lowering sampling temperature or otherwise reducing output diversity makes responses more predictable but does not verify facts or improve alignment with the environment.

The example of an engineer and an AI assistant shows identical downstream failure despite apparent readiness: both systems produced consistent answers yet missed an incompatibility revealed only by validation.

# What to do instead The author presents a different operational objective. Practical systems—whether organizations, human teams, or AI-augmented workflows—should prioritize:

• Evidence-supported decisions: tie recommendations to observable, testable evidence rather than to confidence alone.

• Corrective feedback loops: validate outputs against the environment and adjust models, procedures, or training data accordingly.

• Controlled action with explicit validation: before an output drives change in a live system, run compatibility checks and confirm assumptions.

• Clear boundary accounting: when treating parts of a system (for instance, a team or a model), include the processes that maintain conditions rather than assuming a closed-system behavior.

# Practical framing for technical leaders

# Bottom line Certainty and consistency are signals about a model's behavior, not proof of correctness. Use the right measures for the right domain, insist on validation, and design systems that convert predictable outputs into verified, evidence-backed actions.