How the FDA built HALO: a secure, AI-ready data foundation on Databricks for government

# What happened

The FDA built HALO (Harmonized AI and Lifecycle Operations for Data), an enterprise-grade data platform on Databricks running in AWS GovCloud. The goal was to replace a fragmented landscape of siloed data, duplicated effort, inconsistent pipelines, and high operational overhead with a single, governed foundation for AI and analytics that meets strict federal requirements.

# Why it mattered

# How they approached it

• Make compliance a gating factor: FedRAMP High authorization sponsorship and a move to AWS GovCloud came first. These steps unlocked deployment options and allowed the agency to host export-controlled and sensitive workloads.
• Use a hardened deployment model: The team adopted Terraform-based infrastructure-as-code patterns and a security reference architecture that includes PrivateLink and customer-managed keys to support FedRAMP and IL5-level controls.
• Adopt a unified governance layer: Unity Catalog became the single, open governance layer for data and AI. It provided centralized discovery, access controls, lineage, and a model for secure sharing across centers.
• Maintain multi-tenant isolation: The platform uses a multi-tenant model—described internally as an apartment complex—so each regulatory center keeps its own locks, spaces, and policies while sharing infrastructure.

# Outcomes and scale

The migration and modernization feats were executed without disrupting the FDA's mission. Notable outcomes reported by the agency include:

• Migration of more than 5,000 users and more than 8,000 jobs and pipelines with zero downtime.
• Refactoring of more than 1,000 data pipelines and more than 4,000 notebooks.
• Platform-wide measurable gains: a 30% increase in SQL query speed, a 20% reduction in compute costs, and a 75% decrease in time spent on data provisioning and sharing.

# Technical building blocks

Databricks emphasized a combination of platform and security controls needed to support secure AI in regulated environments:

• Infrastructure-as-code patterns using Terraform for repeatable, auditable deployments.
• Network and storage controls such as PrivateLink and customer-managed keys for data protection and isolation.
• Unity Catalog for a unified governance layer enabling secure discovery, lineage, and managed sharing of data and AI assets.
• A deployment target of Databricks on AWS GovCloud to handle FedRAMP High, DoD IL5, ITAR/EAR, and other sensitive workloads.

# Practical lessons for other agencies or regulated organizations

• Start with compliance requirements: authorization and the correct cloud environment need to be in place before rolling out a platform for sensitive workloads.
• Combine centralized governance with per-tenant isolation to reduce duplication while preserving policy boundaries.
• Use infrastructure-as-code to make hardened deployments repeatable and audit-ready.
• Plan migrations to avoid downtime: the FDA's migration shows that large-scale user and pipeline moves can happen without interrupting mission work.

# Where this sits now

HALO is presented as a production, enterprise-grade data platform that supports secure, auditable AI and analytics across FDA centers. Unity Catalog provides the governance surface, while the underlying deployment patterns and cloud choice deliver the compliance posture required for regulated, mission-critical workloads.