When an LLM Analyzed Three Tabular Files It Repeated the Same Kind of Mistake

The useful part

It Made the Same Mistakes Every Time The review pass fixed a row count and approved two wrong conclusions. what's our average delivery time, which region is our best performer, how many athletes are in this file. We handed the model its own answer back and told it the numbers were going into an exec deck, so verify everything.

How it works

• The Data First, we use the shipment_tracking datatable, which is used in this interview question.
• Working on shipment_tracking, we asked for the average delivery time, and this is the calculation that came back: df['delivery_days'] = (df['delivered_date'].
• Both metrics are real and they measure different things: ship-to-door tells you how the warehouse is performing, and order-to-door tells you how long customers wait.
• How to Catch the Error Read the question, then read the column names in the calculation underneath it.
• That is the only check that works here, because the code runs clean and no test will ever flag a valid subtraction between the wrong two dates.

What to take from it

What makes the sentence dangerous is that 50 minus 22 is 28, so it is internally consistent and externally false. How to Catch the Error Ask whether the code actually ran, and check that every number in the prose appears somewhere in the output, because two of our runs presented code they never executed. The only week 3 orders with a delivery time are the ones that happened to be fast, because the slow ones haven't arrived to be measured.

Example or evidence

• By Nate Rosidi, KDnuggets Market Trends & SQL Content Specialist on September 3, 2026 in Artificial Intelligence --> We ran an experiment: three small datasets, one AI model, and the questions a business...
• One review pass caught a wrong row count and put a checkmark next to a conclusion that was backwards.
• The other invented a correction and turned a right answer into a wrong one.

Details worth keeping

We used GPT-5.6 Terra for the fast first pass and GPT-5.6 Luna for a separate set of unhurried runs on the same files. The second file we are dealing with is regional_sales, used in this interview question. region_name year sales latam 2012 230.62 us_west 2010 163.94 us_east 2012 270.63 emea 2010 150.00 … … … europe_north 2020 300.00 That grain is broken in two ways, and neither one is visible in the column names.

Related coverage

• Medium: Three assistants sold as alternatives failed in the same window on 3 September. Continue reading on Medium »
• Medium: Maybe the real question isn't which AI is smarter… Continue reading on Medium »
• Artificialintelligence News: Advertising inside ChatGPT arrived with a promise that the assistant already knows what the user wants.