Apache Iceberg is an open table format designed for huge analytic datasets in data lakes. Think of it not as a storage system or a query engine, but as a universal translator or a set of rules for how data is organized. It allows different query engines like Spark, Trino, Flink, and Presto to work with the same large datasets reliably, just as if they were interacting with a traditional SQL table.
For most users, Iceberg is completely invisible. You run your SQL queries, and they just work—only faster and more reliably. You only need to peek under the hood if you’re a data engineer or administrator setting up the infrastructure.
Why Was Iceberg Created? A Little History
In the age of Big Data and the Internet of Things (IoT), companies needed to store and analyze massive amounts of data. Traditional data warehouses were often too rigid and expensive. This led to the rise of data lakes, which could store vast quantities of data in various formats like Parquet, ORC, and Avro.
However, the original table format used in these data lakes (most notably the Hive table format) had significant problems:
- Slow Performance: Listing all the files for a large table could take minutes.
- Lack of Reliability: It was difficult to safely write to a table while others were reading it.
- Complex to Manage: Evolving the table schema over time was a painful process.
Iceberg was created at Netflix to solve these exact problems by introducing a more robust and efficient way to manage table metadata.
How Does Iceberg Work? The Magic Layers
Iceberg’s power comes from its clever, layered metadata structure. Imagine it as a tree, where each layer points to the next, ultimately leading to the actual data.
- Iceberg Catalog: This is the starting point. It’s a simple lookup service that tells a query engine where to find the current metadata file for a specific table.
- Metadata File: This file is a snapshot of the table at a specific point in time. Every time a change is made to the table (like adding or deleting data), a new metadata file is created, pointing to the complete list of files for that version. This is the key to Iceberg’s “time travel” and versioning capabilities.
- Manifest List: The metadata file points to a manifest list. As its name suggests, this file is a list of all the manifest files that make up that snapshot. It also stores summary information, like the range of values in certain columns, which helps query engines skip reading unnecessary files altogether.
- Manifest File: Each manifest file tracks a group of data files. It stores detailed statistics for each data file, such as the number of rows, lower and upper bounds for columns, and more. This granular information allows for extremely efficient query planning.
- Data Files: Finally, we have the actual data, stored in efficient columnar formats like Parquet, ORC, or Avro.
When you run a SELECT query, the engine follows this path:
Catalog → Current Metadata File → Manifest List → Relevant Manifest Files → Specific Data Files.
This structure allows Iceberg to quickly find the exact files it needs without having to scan through massive directories, making analytics fast, reliable, and scalable.