A practical briefing of the experiment that measures how rewriting many tiny Iceberg data files into a few larger ones affects SQL query workloads — including the problem, the toolchain, the compaction operation used, and the experimental approach.
The author built a local, reproducible test: an Iceberg table with 50 million rows spread across 1,000 tiny files, then used Iceberg’s default bin-pack rewrite to produce 6 files and benchmarked three SQL workloads before and after.
The article documents a minimal dev environment and exact software choices so readers can reproduce the experiment locally without cloud services.
When you deal with large analytical datasets stored in table formats such as Apache Iceberg, there is a known issue called the small files problem. This happens when data is written in lots of tiny files rather than a smaller number of reasonably sized ones. This increases metadata overhead and can slow down query planning and execution.
Iceberg supports atomic changes, schema evolution, partition evolution and time-travel queries without converting the underlying data into a proprietary storage format. Adaptive Query Execution is disabled because Spark might otherwise combine our deliberately small tasks. I'm focussing on Apache Iceberg as it's rapidly growing into one of the leading open table formats for large-scale analytical data, bringing features such as schema evolution, time travel, partition evolution and reliable transactions to data stored in object storage or distributed file systems.
I Compacted 1,000 Apache Iceberg Files Into 6. Importantly, although Iceberg supports compaction it doesn't do it for us automatically. But, really, does compaction make that much of a difference?
В феврале этого года проект с красивым именем Polaris незаметно стал полноценным проектом фонда Apache. Новость прошла как-то мимо, ну стал и стал, мало ли. Между тем это одна из тех новостей, которые лет через пять, возможно, будут называть поворотной точкой. Потому что Polaris — это каталог. А каталог в мире Iceberg
Accelerate slow, repetitive Apache Spark analytical queries on Apache Iceberg tables without rewriting any SQL. This post shows how automatic query rewrite in Amazon EMR and AWS Glue uses Iceberg materialized views in the AWS Glue Data Catalog to transparently substitute matching query plans, and how to design material
Many database (DB) queries rely on indexing, sorting, and temporary storage during query execution. These elements need to be well-aligned to ensure queries return results quickly and reduce unnecessary CPU, memory, and disk usage. For IT professionals managing production systems, understanding these metrics is a pract
In many analytics platforms, there are performance issues that do not always come from complex transformations. Sometimes the bottleneck is much simpler: the same large datasets are being read repeatedly from remote storage. This pattern is common in shared analytics environments. A data engineering job reads a curated
Инженеры данных построили агрегат — маленькую таблицу «продажи по магазинам по дням». Отчёт из неё собирается за доли секунды. А сводная в Excel всё равно ждёт двадцать секунд и читает миллиард строк. Разбираемся на живом ClickHouse, почему база не видит предагрегат, который для неё построили, какая форма запроса это л
For the past two decades, most enterprise data engineering systems have been built on one default assumption: People understand the system. The system executes the pipeline. Engineers understand the business context, break a requirement into steps, write SQL, Spark jobs, shell scripts, synchronization tasks, and schedu
Loading more related stories...
Open the app view to save this story, compare related coverage, and continue from the same source.
