Chuck's laws of Apache Iceberg

I gave a 3 minute talk about these about a year ago, then 6 months ago, then they showed up in an incident retro very recently. They haven’t changed the whole time, so I’m ready to promote them.

If you’re going to do anything with Iceberg, you need to know these:

  • Chuck’s first law of Iceberg: The only easy way to use Iceberg is Spark/Flink. Any documentation that leads you to believe otherwise is a lie.
    • Chuck’s first lemma of Iceberg: Iceberg is just files and files don’t do anything. Any documentation that leads you to believe otherwise is a lie.
    • Chuck’s second lemma of Iceberg: Iceberg requires a database or process. Any documentation that leads you to believe otherwise is a lie.
  • Chuck’s second law of Iceberg: Iceberg assumes large files, written infrequently, with a (relatively) fixed schema, accessed by a single tenant. In general, none of those are safe assumptions for SaaS telemetry customers.

Iceberg seems to be universally hailed as the solution to all data storage and retrieval needs. It’s just files in an S3 bucket! What could be easier?

Flow diagram of an Iceberg write: an INSERT/UPDATE/DELETE statement leads to 1. write new data files to storage (e.g., Parquet), 2. create new manifest files referencing the data files, 3. create a new manifest list as a new snapshot, and 4. an atomic commit that updates metadata.json to point at the new snapshot. An optimistic concurrency check after the commit either succeeds or sends the writer back to step 1 to retry. Look how easy it is? It’s like 4 steps!

Iceberg is complicated. Not like OAuth complicated, but still, it’s complicated. It relies on a 2-phase write system where something (foreshadowing) writes out data, often Parquet files, to a storage location, often S3, in a reasonable location (I don’t really know, it’s magic). That’s not the hard part; stay with me.

Next you construct new “manifest” files which contain a list of all the new files that something just wrote to disk, as well as statistics about those files and maybe even some stats about their columns. So far so good.

None of this is particularly hard, but now we get to the commit process (I know - some of you felt the cold hand of death pass through your soul). The commit process is where all hell breaks loose.

During the commit process we do a couple of very simple things:

  1. Read the old pointer file which contains the location of the start of the graph
  2. Create a new pointer file which points at the old start location and the new one we just made
  3. Do any schema migrations that need to happen
  4. Handle down migrations / column deletions on your new data (hopefully you didn’t write any data to those columns!)
  5. Copy and edit files that you requested migrations to
  6. Determine which kind of writing mode you need to use (Copy-on-Write, Merge-on-Read, etc)
  7. Deal with any write conflicts that may have come up (very similar to git, so we know how easy that is)
  8. Validate everything and switch the old pointer for the new pointer.
  9. If anything went wrong, go back to the beginning.

Iceberg masterfully compresses all the hardest things about databases into one tight little nugget, then makes sure that query performance is the thing that breaks if anything goes wrong.

Repeat after me:

Iceberg is just files. Iceberg is just files.

Let’s look at a couple of Iceberg settings:

write.parquet.row-group-size-bytes: this is basically how many rows we want in a “chunk” of data. Don’t worry if that doesn’t mean much to you, it’s exciting to me.

When I started working on Federated Logs, this was one of the most exciting settings for me. It meant that Iceberg could take care of handling all the Parquet files, how they’re sized, making sure that they are optimized, etc. It’s very cool.

But Iceberg is just files and files don’t do anything. Processes do things, but files just sit there being little shits, refusing to manage the size of my Parquet files.

In hindsight, it should have been obvious.

There are 85 settings on the Iceberg settings page and exactly zero of those settings affect Iceberg! 100% of those settings are for engines that operate on Iceberg. Why aren’t they Spark settings? I couldn’t tell you.

But you’ll never guess which of those don’t work in the iceberg-go library?! Actually, you will have to guess, because those settings don’t indicate which engines/libs honor each setting either.

I’m not grumpy. You’re grumpy. I don’t know what you’re talking about.

Iceberg requires a database/process

So yeah, Iceberg is just files. And you know what else files can’t do? Hold mutexes and manage write contention.

You remember that whole commit process I described above? Guess what files can’t do! That!

So here’s the thing to remember:

Any time you see an article about Iceberg, substitute the word Iceberg for “Spark or Flink” and it’ll be correct.

Oh. Unless you don’t use Spark or Flink. Then you’re effed.

Okay, breathe… I’m doing fine. How are you?

Moving to the query side, I have good news! Iceberg is just files! I mean, yeah, you have to use a database to acquire a read lock too, but after that point, it’s just files!

NRDB/Dirac are exceptionally good at reading files from S3, so it was a natural move to extend them to do Parquet. And it’s working great.

But when you think about Spark, do you think “scrappy, agile, data retrieval system”? No, you think “I need something that can combine all the compliance files for 6 years together across both clouds and our closet in the Portland office that’s mostly full of wine bottles and GC Jenkins”

We aren’t using Spark for query, but what have we learned about Iceberg? Any time we see the word Iceberg, we can replace it with Spark/Flink. So even though we’re using NRDB, a scrappy, agile, multi-tenant masterpiece, IcebergSpark/Flink’s influence casts a dark shadow across the horizon of our hopes and dreams.

Federated Logs architecture diagram. In the customer environment, agents feed the Pipeline Control Gateway, which writes Parquet files to an S3 bucket and sends a message via SQS to Flink; Flink updates the Glue/Iceberg catalog and runs compaction and cleanup jobs. In New Relic's regions, the query path runs from the NRQL Query Gateway through the TI Router and Query Cell Router to query workers (Arrow file readers) that read the Parquet files directly from S3. Fed Logs architecture diagram (Red is query, Gray is ingest)

Above is the Federated Logs architecture diagram. It took us a while, but we eventually figured out that in order to write we really had to have Spark/Flink in there. It’s still slightly non-standard, but it’s working fine for now.

Thought experiment 1

Let’s say you have a system like the one above. You need to get data from the Agents/Forwarders, to the customer’s browser, in 30-45 seconds.

If transiting through Pipeline Control Gateway, SQS, and Flink takes about 15 seconds, how many seconds of data can you fit in a file and still have some buffer?

If you said “about 15 seconds”, you deserve a pat on the back.

Thought experiment 2

If you needed to load an hour worth of data into memory, spread across hundreds of workers, who each individually operate on their files and then combine the results, how much time do you think you would like to have in each file?

If you said “an hour”, you get a second pat on the back.

Thought experiment 3

If you needed to load years of data into memory, spread across a few, beefy nodes, and you don’t care if it takes an hour to compute the results, how much time would you like to have in each file?

If you said “as long and as big as possible”, you get a third pat on the back.

If you went through the thought experiments above, you may be beginning to get an inkling of some of the problems we’re facing.

So if you’re running a system that assumes that files are large, written infrequently, have a (relatively) fixed schema, and are accessed by a single tenant, how much more okay are you with having literally all your risk in one, tight little commit process? I’d be pretty okay with it. “It’s not like we do it more than like 20 times a day, right?”

Wrong. We’re trying to do operational logs, so we need that data to get written every 15s, then we need to combine that data in sequential series of aggregations until it’s at a reasonable file size, which all means we do 5,760 writes per day, minimum. I can assure you, it’s a lot more than that, because people have different regions, different tables with different data, different accounts, different use cases and different S3 buckets.

Conclusion

You might come to the conclusion that Fed Logs isn’t going well, and you’d be wrong. It’s going shockingly well. We have one of the largest Logs customers at NR sending a staggering amount of data through a frankly insane cluster of pipeline control gateway daemonset pods, who get their data processed and in the browser in well under 45s, all day, every day. And they can query it too.

Yes, there are issues with combining files. Yes, there are things we still need to clean up. It’s not always pretty down here in the vegan sausage factory.

But holy cow… We’re. Doing. It!

I hope you learned some things. ❤️