An analytics pipeline

Play this lesson

A share from a tunnel

A player shares a level at 17:58 on a train, in a tunnel, with no signal. The event reaches the studio at 19:20.

Which hour should it count in? Theo, the studio's data analyst, says the answer depends on the pipeline: the path every event takes from a tap to a table.

It has five stages:

Apache Beam, an open-source data tool, calls data of a fixed size bounded, handled in batch, and data that keeps growing unbounded, handled as a stream.

Batched to save battery

Firebase's docs say events are generally batched together over about an hour and uploaded together. Sending one request per tap would drain batteries and data plans.

On a developer's test phone in debug mode, events upload with minimal delay, so Remy can check a new event at once.

So analytics is not live to the second. For questions like "how many levels were shared this week", it doesn't need to be.

Quick check

With Firebase's default settings, Theo sees a player's event the moment they tap.

Show the answer

False

Events are generally batched over about an hour and uploaded together. Only debug mode on a test device sends them with minimal delay.

Event time and processing time

Apache Beam names two clocks:

Theo counts by event time, because the question is when players did things, not when the network delivered them.

Late data and the watermark

Beam defines a watermark as a guess of when all the data for a window of time has arrived. An event that turns up after the watermark, with a timestamp inside that window, is late data.

A pipeline can update the old numbers when late data arrives, or set it aside. Either way, yesterday's numbers can still change a little today. Theo's dashboard marks the newest day "still filling".

Read the numbers

level_shared, counted on Monday
Counted at    08:00  12:00This Sunday   3,900  5,100Last Sunday   5,600  5,600

On Monday morning, Sunday's shares look 30% down. By noon the gap is much smaller. What is happening?

  1. Late events from batched and offline phones are still arriving
  2. Players stopped sharing on Sunday morning
  3. The pipeline is counting by processing time on purpose
Show the answer

Late events from batched and offline phones are still arriving

Events arrive up to hours late, so the newest day fills in slowly. Comparing it with a complete day makes it look like a drop. Wait for the watermark, or mark the day as still filling.

Sent twice

Many delivery systems promise at least once: every event arrives, and some arrive twice. Google Cloud's Pub/Sub docs call these redeliveries valid, the system working as intended.

So each event carries an event ID, a random ID made once, on the device, when the event happens. If the phone retries, the same ID goes again. The pipeline keeps one row per ID. This is deduplication.

Quick check

Why is the event ID made on the device, not when the event reaches the server?

  1. A retry resends the same ID, so the copy can be spotted
  2. Server IDs would be too long
  3. Phones make random numbers faster
Show the answer

A retry resends the same ID, so the copy can be spotted

If the server made the ID, two copies of one event would get two different IDs and both would count. An ID made once at the source travels with every copy.

Raw first, then tables

Theo always keeps the raw events, unchanged. If a transform has a bug, Theo fixes it and runs it again over the raw data, instead of losing a week of numbers.

Bolt says: Next: where those raw events live, a warehouse.

Fact-checked 11 October 2026