Beyond the price per token: How to pick the right OpenAI model on Amazon Bedrock for real workloads

# What the post tests

# What they measure The harness reports three concrete metrics:

• Single-call accuracy and cost-per-correct-answer on benchmarks that still separate top models (AIME math, GPQA Diamond science, MMLU-Pro).
• Multi-turn agent trajectory cost on live web-research tasks, where turn count and conversation growth matter.
• Rubric-graded professional deliverables scored with deterministic checks plus a frozen prompting LLM judge (gpt-5.5) whose prompts are hashed and recorded.

Each run emits a timestamped JSON with every number used to generate the post's charts and tables.

# Key patterns the results show Capability tiers are visible in these samples: higher-tier models (Sol in the reported runs) achieve materially higher accuracy on the academic benchmarks than cost-optimized minis and nanos. That accuracy gap translates into fewer attempts needed per correct answer and therefore lower observed cost per correct answer despite higher nominal token prices.

Agentic workloads amplify billing differences. Every additional turn in a web-research agent re-sends the growing conversation and multiplies billed tokens. The harness measures end-to-end agent trajectory cost rather than per-turn token price to show the real bill impact of agent design and model choice.

# Practical guidance Run the harness on your real tasks. The comparison in the post is intentionally a practical deployment comparison: Bedrock runs had reasoning disabled for these models while the OpenAI API baselines used default settings. That means the numbers reflect deployment configurations, provider infrastructure, and model-specific settings rather than a controlled intrinsic-capability test.

# Where to start Clone and run the open-source harness (openai-on-aws/benchmarks-openai) against the models and settings you plan to deploy. Use your real prompts, agent design, retry and backoff strategy, and rubric definitions so the measured cost per correct answer and deliverable quality reflect your product's requirements.

The post's core recommendation is procedural: evaluate models by outcomes (cost per correct answer, trajectory cost, and professionally graded deliverables) using identical client paths across backends, and base selection on replicated results for your workload.