Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs. The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue.
Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs.
The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue.
In this tutorial, we will look at an alternative solution that addresses both problems at their root: using Netflix Maestro, a horizontally scalable workflow orchestrator open-sourced by Netflix in July...
The page is ready to read now. The fuller skim-friendly version will appear here automatically.
Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs. The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue. In this tutorial, we will look at an alternative solution that addresses both problems at their root: using Netflix Maestro, a horizontally scalable workflow orchestrator open-sourced by Netflix in July 2024, along with Apache Iceberg, a standard table format for analytics on object storage.
Open the app view to save this story, compare related coverage, and continue from the same source.