Apache Paimon is a lakehouse table format in the same family as Apache Iceberg: both turn files in object storage or distributed filesystems into transactional tables with snapshots, metadata, schema evolution, and query-engine integration.
For a Fluss reader, the relevant distinction is that Iceberg is mainly an analytic table metadata layer over immutable files, while Paimon includes more of the streaming-table runtime in the format and client library. A Paimon primary-key table stores bucket-local sorted runs and changelog files; Paimon writers, readers, and compaction jobs know how to turn those files into an updateable table.
Prerequisites
- Introduction to Apache Iceberg - open table formats, snapshots, manifests, and table metadata
- Log Structured Merge Tree - the sorted-run structure Paimon uses for primary-key tables
- Introduction to Flink - the streaming engine that commonly writes and reads Paimon tables
Why Paimon Is More Than Parquet Files
Plain Parquet files are just columnar files. A directory full of Parquet does not define which files form the current table version, how to atomically add files, how to represent deletes, how to time travel, or where a streaming reader should resume.
A Paimon commit creates a new snapshot. The snapshot points to manifest metadata, and the manifests point to immutable data and changelog files.
Paimon adds a table metadata layer:
| Concept | Role |
|---|---|
| Snapshot | A point-in-time table version. Readers start from a snapshot to discover the files that make up the table state. |
| Manifest list | The list of manifest files referenced by a snapshot. |
| Manifest | File-level changes for the snapshot: data files or changelog files added/deleted, with partition, bucket, statistics, and sequence information. |
| Data file | The actual table data, usually Parquet, ORC (Optimized Row Columnar), or Avro. |
| Changelog file | A data-file-shaped file that records row changes for a primary-key table. |
| Commit protocol | A two-phase-style write path that lets distributed writers publish a coherent new table snapshot. |
The table abstraction matters because engines need a single entry point. A query engine asks the catalog for the current snapshot, follows the snapshot to the manifests, then scans the files listed by those manifests.
Where The Runtime Lives
Paimon is not a database server in the RocksDB sense. There is no always-on Paimon TabletServer holding a MemTable for every table. The always-durable object is still a lake table: directories, immutable data files, changelog files, snapshots, and manifests.
The LSM-like behavior lives in the compute jobs that use the Paimon library:
| Piece | Where it lives | What it does |
|---|---|---|
| Writer buffer | Inside the Flink/Spark/Paimon writer task | Holds incoming rows before flushing them as new sorted data files. |
| Data files / sorted runs | In the table storage directory | Persist the bucket’s immutable runs. These are the durable part of the LSM-shaped layout. |
| Snapshot / manifests | In the table metadata directory | Decide which files are visible for a table version. |
| Merge engine | In the Paimon reader/writer library | Resolves multiple records for the same primary key: latest row wins, delete suppresses the row, or another configured merge rule applies. |
| Compaction | In writer-side compaction threads or a dedicated compaction job | Merges sorted runs and marks superseded files as deleted in new metadata. |
Iceberg readers normally scan the files listed by the current snapshot. Paimon primary-key readers must do more: for a bucket, they may need to merge several sorted runs before emitting the current row for each primary key. Paimon is therefore closer to a lake storage engine implemented by client-side writer/reader/compaction code, while Iceberg is closer to a table metadata protocol over immutable files.
Primary-Key Tables
A Paimon primary-key table supports inserts, updates, and deletes. Paimon partitions the table, then subdivides each partition into buckets (the smallest read/write storage unit). Each bucket is its own storage unit: it has an LSM-shaped set of data files and, when configured, changelog files.
In merge-on-read mode, a write publishes a new sorted run for a bucket. A table read reconstructs the current row for each primary key by merging the visible runs from the chosen snapshot.
Write Path
For a primary-key table in the default merge-on-read (MOR) mode, a write does not search all old files and rewrite the old row in place. The writer task behaves like the mutable front end of the storage engine for the buckets it is writing.
- The writer receives records with a primary key and row kind: insert/update/delete.
- Paimon chooses the bucket by hashing the bucket key, usually the primary key.
- Inside that writer task, Paimon buffers incoming records in memory. This buffer is not a durable server-side MemTable; it is task-local state used while preparing the next file commit.
- At a flush/checkpoint boundary, the buffer is sorted by primary key and written as new immutable data files in a new sorted run. A sorted run is one or more data files whose key ranges do not overlap within that run.
- The commit creates snapshot/manifest metadata that makes the new run visible. Older runs remain visible too until compaction or snapshot expiration removes them.
If key 42 is written as pending, then later paid, then later shipped, those versions may live in different sorted runs. The table state is not stored as one physically rewritten row. It is the logical result of merging the visible runs.
Changelog files are separate from the current-state data files. With an input changelog producer, Paimon stores the input row changes in changelog files so streaming readers can see the event stream. Without a changelog producer, a table scan can still recover the latest table state, but downstream consumers may not get a complete old-value/new-value changelog.
Read Path
A read starts from a snapshot, not from “whatever files happen to be in the directory.” The snapshot points to manifests; manifests identify the data files and changelog files visible for that table version.
For a primary-key table scan through the Paimon reader:
- The planner prunes partitions and buckets. A primary-key filter can often identify the relevant bucket because the bucket is derived from the key hash.
- For each selected bucket, the reader opens the sorted runs visible in the snapshot. This step is why a generic file reader is not enough for a fresh Paimon primary-key table.
- Each run is scanned in primary-key order. Different runs may contain the same key.
- The reader performs a multi-way merge across runs. When records share a primary key, the table’s merge engine decides the surviving row.
- The default
deduplicatemerge engine keeps the latest record for the key. If the latest record is a delete, the key disappears from the table result.
This is the concrete meaning of merge-on-read: the write path avoids old-file rewrites, and the read path pays by combining several sorted runs before it can emit the current row. Compaction reduces that read cost by merging runs in the background and marking superseded key versions as deleted in table metadata.
Paimon also has two read/write tradeoff modes that move work out of reads:
| Mode | What changes on write | What changes on read |
|---|---|---|
| MOR (merge-on-read) | Write a new sorted run; minor compaction may run separately | Merge visible runs by primary key |
| COW (copy-on-write) | Perform full compaction during writes | Read mostly pre-merged files |
| MOW (merge-on-write) | Query existing data during write and create deletion vectors for older rows | Use deletion vectors to skip invalidated rows instead of fully merging them |
The Small-File Tax
Paimon does not remove the object-store small-file problem. It moves the problem into a storage engine that can manage it.
Object stores are immutable at the file/object level: a writer cannot patch one row inside an existing Parquet file. A streaming writer therefore has to publish new files over time. In a Paimon primary-key table, each Flink checkpoint can flush new L0 data files for the buckets touched by that checkpoint. If checkpoints are frequent, writer parallelism is high, partitions are many, or bucket count is too large, the table can accumulate many small files.
Paimon’s answer is compaction:
- Writer-side compaction can merge recent small sorted runs while the streaming job writes.
- Dedicated compaction jobs can run separately when compaction would otherwise backpressure writers or when multiple writers make inline compaction conflict-prone.
- Table options such as target file size and small-file compaction thresholds decide which files are worth rewriting.
- Full compaction can periodically rewrite a bucket or partition into larger files, trading write/maintenance cost for cheaper reads.
This is still an object-store maintenance burden. Compaction reads small files, writes larger replacement files, and commits new metadata that marks old files as deleted. Old files may remain physically present until snapshot expiration and cleanup remove them. The win is not “streaming writes without small files”; the win is that Paimon gives the table format a built-in way to track, merge, and expire those files while preserving snapshot isolation.
Bucket sizing is the lever that makes this tradeoff visible. More buckets increase write/read parallelism, but each bucket has its own runs and compaction work. Too many buckets means many small files; too few buckets means each bucket is large, reads have less parallelism, and a single bucket’s merge work gets heavier.
Paimon vs. Iceberg
Paimon and Iceberg overlap because both are open lakehouse table formats. For Apache Fluss, the key difference is where each format puts the streaming/update machinery.
| Question | Iceberg | Paimon |
|---|---|---|
| Core mental model | Analytic table metadata over immutable data/delete files | Lake storage engine: sorted runs, merge engine, changelog files, and compaction |
| Runtime | Query engines and catalogs interpret Iceberg metadata; the format itself has no table-owned writer/compactor runtime | Paimon client code inside Flink/Spark/writer jobs performs buffering, merging, and compaction |
| Updates | Represented through snapshots, data files, delete files, and engine-specific merge behavior | Built into primary-key tables through sorted runs, merge engines, and changelog files |
| Changelog reads | Possible through integrations and table-change features, but not the core storage primitive | Native concept for primary-key tables |
| Best fit | Broad ecosystem interoperability and large analytic datasets | Flink-style streaming updates, primary-key tables, and incremental consumption |
This is not “Paimon good, Iceberg bad.” Fluss supports both. Iceberg is often the safer interoperability choice; Paimon is often the more direct fit when the tiered table needs to preserve upsert and changelog semantics.
Why Not Just Sorted Iceberg Files?
Writing sorted files to Iceberg is not the same as writing a Paimon primary-key table.
Suppose key 42 changes three times:
| Offset | Key | Status |
|---|---|---|
| 10 | 42 | pending |
| 20 | 42 | paid |
| 30 | 42 | shipped |
If those rows are written to Iceberg as ordinary data files, Iceberg records that the files belong to a snapshot. Sorting by key or offset can make scans cheaper, but the table format does not automatically interpret the three rows as “only offset 30 survives.” A query still needs a deduplication rule such as row_number() over (partition by key order by offset desc) = 1, or a writer must execute a merge/upsert operation that rewrites data/delete files according to engine-specific logic.
In a Paimon primary-key table, the replacement rule is part of the table’s storage semantics. The sorted runs may contain all three versions, but the Paimon reader’s merge engine treats them as versions of the same primary key and emits the surviving row. Compaction later rewrites the bucket’s files so older versions stop burdening reads, and the new snapshot metadata marks the superseded files as deleted.
That is the core difference: Iceberg can store sorted versioned facts; Paimon primary-key tables define how those facts collapse into current keyed state.
Why Fluss Tiers To Paimon
Apache Fluss keeps hot data in Fluss for sub-second reads, lookup joins, and changelog consumption. Historical data is tiered into a lakehouse format so analytical engines can scan it efficiently.
Paimon helps Fluss preserve stream position and table semantics in the cold layer:
- A Fluss LogTable can become a Paimon append-style table with stream-position system columns.
- A Fluss PrimaryKeyTable can become a Paimon primary-key table whose cold copy still understands updates and deletes.
- Paimon snapshots give Fluss a durable boundary between “cold data is committed” and “hot data still needs to be read from Fluss.”
- Paimon changelog files let the cold table remain useful for incremental consumers, not just batch scans.
Fluss-created Paimon tables add system columns such as __bucket, __offset, and __timestamp. Those columns let Fluss and query engines align lake snapshots with Fluss log offsets when performing union reads across cold Paimon data and hot Fluss data.
See Also
- Apache Fluss - streaming storage that can tier hot data into Paimon
- Introduction to Apache Iceberg - the most common comparison point
- Log Structured Merge Tree - the storage shape behind Paimon primary-key tables