How to add LLM-powered DAG failure analysis to Amazon MWAA

# What this plugin does

This post shows how to build a custom Apache Airflow plugin for Amazon MWAA that provides on-demand root cause analysis for failed DAG tasks. Instead of manually trawling logs, engineers trigger an Analyze Task action in the Airflow UI. The plugin gathers context, enriches it with operator-aware code retrieval, calls Amazon Bedrock for analysis, and returns a structured diagnostic report that appears inside the Airflow UI.

# Why this approach

Airflow orchestrates complex ETL and analytics workflows across services such as AWS Glue, Amazon EMR, Amazon Athena, and Amazon Redshift. When a task fails, teams typically read logs, inspect DAG files, and try to map error messages to code. The plugin reduces that manual work by collecting the relevant artifacts and passing them to a foundation model that produces a clear, actionable diagnosis and recommended remediation steps.

# Core architecture and flow

• Based on operator type, the plugin fetches operator-specific scripts or queries (for example, PySpark scripts referenced by a Glue job definition or SQL text embedded in an Athena operator).
• The enriched context is posted to Amazon Bedrock (Anthropic Claude in the example) for analysis.
• A structured diagnostic report is returned and displayed in the Airflow UI, with root cause identification, step-by-step resolution, and prevention recommendations.

# Operator-aware context collection

Including the exact script or query allows analysis to correlate stack traces or error messages with lines of code.

# Authentication and credentials

All AWS API calls (Amazon Bedrock, S3, Glue) use the aws_default Airflow connection. On Amazon MWAA, that connection typically has no static credentials and boto3 falls back to the environment execution role. This removes the need to manage long-lived keys. If a different identity is required, supply credentials or a dedicated IAM role in the aws_default connection to use instead of the execution role.

# Deployment and resources

The post points to a GitHub repository named sample-aws-mwaa-llm-powered-plugin containing the complete source code and deployment examples. The plugin integrates into MWAA, stores artifacts and DAGs in S3, and delegates model inference to Amazon Bedrock.

# Practical considerations

• Operator-aware retrieval depends on operators being configured with accessible script locations or inline code. If code lives outside accessible locations, retrieval will be limited.
• Using the execution role simplifies credential management, but cross-account or alternate-role scenarios require configuring the aws_default connection.

# Bottom line