Growing Tips
Growing Tips
Apache Iceberg & Spark 101
17 articles · \~2h 46m · 227 snippets
\[0\]Foundations
Storage, compute, and where Iceberg and Spark fit.
[\[0\]The Basic Building Blocks of Data SystemsEvery garden needs soil and sunlight. Every data system needs storage and compute. Learn the two pieces everything else grows from.\~6 min read](https://oleander.dev/grow/the-basic-building-blocks-of-data-systems)[\[1\]Introduction to Apache IcebergApache Iceberg is the raised bed of the data garden: an open table format that keeps files in object storage organized as tidy, reliable tables.\~8 min read](https://oleander.dev/grow/introduction-to-apache-iceberg)[\[2\]Introduction to Apache SparkApache Spark is the heavy machinery of the plot: a distributed compute engine that works your data at any scale.\~7 min read](https://oleander.dev/grow/introduction-to-apache-spark)
\[1\]Getting started
Core concepts, a working Spark setup, and your first Iceberg tables.
[\[3\]Apache Iceberg: Concepts and TerminologyCatalogs, snapshots, schemas, and partitions. Learn the botanical names of your Iceberg garden so you can talk shop with any gardener.\~7 min read](https://oleander.dev/grow/apache-iceberg-concept-and-terminology)[\[4\]Apache Spark: Getting StartedSet up Spark locally, start a session, and run your first commands against an Iceberg catalog.\~5 min read · 11 snippets](https://oleander.dev/grow/apache-spark-getting-started)[\[5\]Working with Apache Iceberg Tables in SparkCreate, load, update, and inspect Iceberg tables from Spark. Daily care for healthy tables.\~5 min read · 19 snippets](https://oleander.dev/grow/working-with-apache-iceberg-tables-in-spark)[\[6\]Apache Spark: Concepts and TerminologyDrivers, executors, jobs, stages, and tasks. How Spark actually divides up the work in the field.\~6 min read · 1 snippet](https://oleander.dev/grow/apache-spark-concept-and-terminology)
\[2\]Analyzing clickstream data
Summaries, funnels, and joins over a retail clickstream table.
[\[7\]Setting Up a Retail Clickstream Table with Apache IcebergBreak ground on the retail clickstream table we will cultivate for the rest of the series.\~2 min read · 4 snippets](https://oleander.dev/grow/setting-up-retail-clickstream-table)[\[8\]Summarizing Clickstream Events with SparkTurn raw clickstream events into daily summaries with Spark SQL and the DataFrame API. Your first basket of produce.\~17 min read · 26 snippets](https://oleander.dev/grow/summarizing-clickstream-events-with-spark)[\[9\]Analyzing Product Funnels with AggregationsFollow sessions from product view to purchase with aggregations, the way a gardener tracks seed to fruit.\~13 min read · 17 snippets](https://oleander.dev/grow/analyzing-product-funnels-with-aggregations)[\[10\]Measuring Cart Abandonment with JoinsSome carts never ripen into purchases. Use joins to find what wilted on the vine and when.\~13 min read · 19 snippets](https://oleander.dev/grow/measuring-cart-abandonment-with-joins)[\[11\]Understanding Join Types in SparkInner, left, full, semi, and anti joins. Choosing the right graft when joining two tables together.\~13 min read · 30 snippets](https://oleander.dev/grow/understanding-join-types-in-spark)
\[3\]Production techniques
Retry-safe writes, partitioning, time travel, streaming, and compaction.
[\[12\]Making Batch Pipelines Retry-Safe with MERGE INTORetries happen. MERGE INTO keeps a re-run pipeline from double-planting the same rows.\~14 min read · 22 snippets](https://oleander.dev/grow/making-batch-pipelines-retry-safe-with-merge-into)[\[13\]Improving Query Performance with PartitioningPartition your tables so queries only walk the rows they need instead of trampling the whole field.\~9 min read · 10 snippets](https://oleander.dev/grow/improving-query-performance-with-partitioning)[\[14\]Querying Historical Data with Time TravelIceberg keeps a journal of every change. Query your table as it was, compare seasons, and roll back mistakes.\~10 min read · 21 snippets](https://oleander.dev/grow/querying-historical-data-with-time-travel)[\[15\]Streaming Clickstream Events into IcebergSwap the watering can for drip irrigation: stream clickstream events into Iceberg with Structured Streaming.\~18 min read · 29 snippets](https://oleander.dev/grow/streaming-clickstream-events-into-iceberg)[\[16\]Managing Small Files with CompactionFrequent commits leave lots of tiny files. Prune them back with compaction so the table stays healthy as it grows.\~13 min read · 18 snippets](https://oleander.dev/grow/managing-small-files-with-compaction)
You are loved. Always water & happy growing...