Edmondo's Vault

Apache Iceberg Catalogs

May 18, 20265 min read

Introduction

Iceberg provides a catalog interface that requires the implementation of a set of func tions, primarily ones to list existing tables, create tables, drop tables, check whether a table exists, and rename tables.

The primary high-level requirement for a catalog implementation to work as an Iceberg catalog is to map a table path (e.g., db1.table1) to the filepath of the metadata file that has the table’s current state. Not all implementations are suitable for being used in production: only the one that support atomic operations for updating the current metadata pointer should be used to avoid data loss

The examples in here are for Spark (see Apache Iceberg with Spark for setup), but similar concept applies to Apache Iceberg with Flink

Main catalogs

The Hadoop Catalog

The Hadoop Catalog maps the table path to its current metadata file by listing the contents of the table’s metadata directory and choosing the most recently written metadata file based on the timestamp listed for each file in the filesystem. It is not recommended to be used in production for three reasons:

  • not all filesystems provide atomic updates
  • it can only be used with one warehouse folder
  • listing the catalog tables require listing the filesystem

Warehouse folder

A warehouse is essentially a root directory under which multiple Iceberg tables are stored. The Hadoop catalog cannot handle tables that are not under a single folder hierarchy

Tip

Generally, however, engines will write a file called version-hint.txt in the table’s meta‐ data folder containing a single number that the engine/tool then uses to retrieve the indicated metadata file by name—for example, v{n}.metadata.json.

To launch Spark with the Hadoop Catalog, besides the SparkSessionExtensions defined above, the following extra conf are needed:

--conf
spark.sql.catalog.my_catalog1=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.my_catalog1.type=hadoop \
--conf
spark.sql.catalog.my_catalog1.warehouse=<protocol>://<path>

The Hive Catalog

The Hive catalog maps a table’s path to its current metadata file by using the location table property in the table’s entry in the Hive Metastore, setting it to the absolute path in the filesystem where the table’s current metadata file can be located. Although it is compatible with many tools, the Hive catalog is often not a good choice for two reasons:

  • it requires an external dependency/component
  • it doesn’t support multitable transactions

Compared to the Hadoop catalog, to use it one needs to change spark.sql.catalog.catalog_name.type from hadoop to hive and set spark.sql.catalog.catalog_name.uri to a thrift address for the metastore rather than a warehouse

The Glue Catalog

The Glue catalog is fully managed by AWS, reducing operational overhead. However, it still doesn’t support multitable transactions and is very coupled with AWS. To use it, one needs to set additional properties when starting spark

--conf spark.sql.catalog.my_catalog1.warehouse=s3://<path> \
--conf spark.sql.catalog.my_catalog1.catalog-impl=org.apache.ice-
berg.aws.glue.GlueCatalog \
--conf spark.sql.catalog.my_catalog1.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
--conf spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY \
--conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY

The Nessie Catalog

With Nessie, a table’s path is mapped to its current metadata file using a table property called metadataLocation stored within the table’s entry in Nessie similarly to what is done with the Hive catalog. Nessie is a great choice:

  • it supports multitable transactions
  • it provides a Git-like experience to data lakes and enables the concept of “data as code”

The REST Catalog

The service implementing the REST catalog interface can choose to store the mapping of a table’s path to its current metadata file in any way it chooses. It could even store it in one of the other catalogs and provide multitable transactions.

Warning

Not all tools support the REST Catalog, and there is no available implementation. The only existing one is managed and is provided by Tabular

The JDBC Catalog

The JDBC catalog is backed by a relational database such as PostgreSQL or MySQL and is relatively lightweight to operate. However, it doesnt support multitable transactions

Other catalogs

A popular catalog is Snowflake, but also the Unity Catalog

A note on SparkSessionCatalog

The SparkSessionCatalog is a more specialized implementation that wraps around Spark’s built-in session catalog, adding support for Iceberg tables. This could benefit scenarios when you want to use Iceberg tables seamlessly alongside non-Iceberg tables in your Spark session: all the non-Iceberg tables are managed by the built-in Spark catalog, while the tables specific to Iceberg are managed separately through the SparkSessionCatalog

Important

Running create table as select(CTAS) in the context of Apache Iceberg is an atomic operation only when using the SparkCatalog class. If you use the SparkSessionCatalog class, CTAS is supported but is not atomic, which may cause inconsistencies when

Migrations

Migrations can happen in two ways:

  • using the Catalog Migrator CLI
  • using the Engine, such as Spark
java -jar iceberg-catalog-migrator-cli-0.2.0.jar migrate \
--source-catalog-type GLUE \
--source-catalog-properties warehouse=s3a://bucket/gluecatalog/,io-
impl=org.apache.iceberg.aws.s3.S3FileIO \
--source-catalog-hadoop-conf
fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY,fs.s3a.access.key=$AWS_ACCESS_KEY_ID \
--target-catalog-type NESSIE \
--target-catalog-properties uri=http://localhost:19120/api/v1,ref=main,ware-
house=s3a://bucket/nessie/,io-impl=org.apache.iceberg.aws.s3.S3FileIO \
--identifiers db1.nominees

The Migration CLI offers two main commands:

  • migrate which effectively will remove the table from the input catalog
  • registerwhich will have the table registered in both catalogs with update to the table through the new catalog not being reflected in the old catalog

An alternative is to use the engine call to invoke register_table, which effectively means the same table is reigstered in two catalogs, or snapshot, which creates a non-owning registration of the table(i.e. expire_snapshots is not possible)

CALL target_catalog.system.register_table(
	'target_catalog.db1.table1', 
	'/path/to/source_catalog_warehouse/db1/table1/metadata/xxx.json'
)

or so:

CALL target_catalog.system.snapshot(
	'source_catalog.db1.table1',
	'target_catalog.db1.table1'
)

Graph View

Backlinks