Apache Iceberg brings the reliability, performance, and governance capabilities of a data warehouse to data lakes. It does this through a metadata-first architecture that changes how analytics engines interact with data.
Metadata as the core innovation
Open table formats improve legacy approaches by adding a metadata layer on top of raw files. What distinguishes Iceberg is how it structures that metadata and what it enables.
Apache Iceberg organizes every table into three layers: a catalog that serves as the entry point for all table operations, a metadata layer that tracks every file and every change, and the data files themselves. The catalog points to the current metadata, ensuring that any compatible engine can locate and safely operate on the same table without coordination outside the format.
Each Iceberg table maintains three layers of metadata:
- Metadata files capture the table's complete state at a specific moment, including the schema, partition layout, sort order, and full history of snapshots.
- Manifest lists reference groups of manifest files and record the changes introduced by each write operation.
- Manifest files contain detailed information about individual data files, including file locations, row counts, partition values, and column-level statistics like minimum and maximum values.
When a user submits a query, Iceberg evaluates the manifest metadata first. It reads the manifest-level partition summaries and per file column-level statistics within each manifest to determine which data files contain relevant records and only opens those files. This means the engine skips irrelevant files entirely, scanning only the data that matters. In cloud environments where cost is tied to the amount of data scanned, this translates directly into cost savings and faster response times.
Learn how Amazon Ads achieved a 22% reduction in processing time, 20% decrease in compute costs, and 32% savings in storage costs by leveraging Iceberg’s file layout optimizations on Amazon S3.
ACID transactions
Atomicity, consistency, isolation, and durability (ACID) transactions are what make data lakes suitable for workloads that previously required a data warehouse. Every write operation in Iceberg produces a new immutable snapshot, which represents the state of a table at a given moment. Readers always see a consistent state of the table and never encounter partial writes, even when multiple processes are writing concurrently. When conflicts occur, Iceberg detects them and fails cleanly rather than corrupting data.
Time travel
Because every write produces an immutable snapshot, Iceberg naturally preserves a complete history of table states. This means you can query the exact state of a table as it existed at any point, enabling audit trails, historical analysis, and reproducible analytics. When an erroneous write corrupts a table, you can roll back to a previous state in seconds rather than rebuilding from scratch.
Tables accumulate snapshots over time, and while this history enables powerful capabilities, it requires active management like expiring old snapshots and removing orphan files to maintain table health and control storage costs.
Schema and partition evolution
Data structures rarely remain static. Iceberg was designed to allow schemas to evolve in a safe, consistent, and forward-compatible way by managing schema and partition changes through metadata rather than data rewrites. Columns can be added, removed, renamed, or retyped without reprocessing existing files. Partition strategies can shift while maintaining full compatibility with historical data, allowing your data architecture to evolve alongside your business.
Hidden partitioning
Traditional data lakes require users to know the partition scheme to write efficient queries, meaning you must explicitly filter on those derived columns. Iceberg removes this requirement by using partition transforms to automatically derive partition values from source columns and prunes data based on the values in the query itself. Users write natural filters on original columns, and Iceberg handles the rest.
Row-level update and delete operations
Iceberg supports MERGE INTO, UPDATE, and DELETE operations directly on data lake tables without full table rewrites. This enables streaming pipelines, regulatory deletion requests, and corrections at scale.
Open at every layer
Apache Iceberg is an open source project guided by community governance, not controlled by any single vendor. Its openness runs through every layer of the architecture: data is stored in open file formats (Parquet, ORC, Avro), the Iceberg REST Catalog provides a standardized API for table discovery, and any compatible engine can work with the same tables without reformatting or duplication. Organizations are free to build their own data architectures without fear of vendor lock-in or committing to any single proprietary format.
