Exactly-Once Log Summaries with Sentinel data lake and Fabric - Part 1: The Idea

Summarization and Current Limitations

Firewall, DNS and proxy logs produce millions of events a day. Keeping them all in the Sentinel analytics tier is expensive; keeping them only in the data lake means no analytics rules can run on them. Summarization is the middle ground: the raw events stay in the lake, and an aggregated version goes back to the analytics tier for detection and reporting.

A summary groups events by a few keys and keeps a count and the time range they cover. For example, 400 firewall events of a website visit from the same host within 15 minutes become a single row:

Analytics rules alert on the count, and FirstSeen-LastSeen tells an analyst where to look for the raw events.

Microsoft’s built-in options - Summary Rules and KQL Jobs - make this easy, but neither can guarantee that every event is counted exactly once (no event missed, none counted twice). Both can wait before processing a time range so late events have time to arrive. KQL Jobs can also overlap their runs instead, so late events get a second chance. So to handle late events, you either pay with delayed processing or with potential duplicates. And either way, anything later than the wait or the overlap is still missed. This post shows how a Fabric notebook can count every event at least once, however late it arrives, and exactly once at query time with a small deduplication step.

Why summaries break

Logs are never queryable the moment they are generated, so an event’s TimeGenerated field and the time it was ingested can differ. Usually the gap is small, but slow sources or collection outages can stretch it to hours.

This is a known problem for Analytics rules. Microsoft’s recommendation is to let rule runs overlap on TimeGenerated - the time field Sentinel filters on by default - so late events stay in scope, and then filter on ingestion_time() - the time the record was ingested - so the overlap doesn’t produce duplicates.

The data lake doesn’t have this option: ingestion_time() is on the short list of functions KQL jobs don’t support. Microsoft’s guidance is again to overlap the queries, but without ingestion_time() nothing resolves the duplicates. The overlap also only catches events delayed by less than the overlap itself: with 15 minutes of overlap, an event 20 minutes late is still missed. And widening it costs money, because querying data in the lake is billable.

A job running every 15 minutes with a 30-minute lookback: on-time event A falls into two windows and is counted twice, while late event B arrives only after both runs that cover it have finished, so it is never counted.

To sum up, with KQL jobs you get:

• No at-least-once processing: an event that arrives later than the overlap covers is never counted.
• No at-most-once processing: an event that arrives on time is counted by every run that overlaps it.
• Paying for the same data more than once: the more delay you want to handle, the more data each run scans again.

Fabric’s solution: process by arrival, not by timestamp

Instead of asking “which TimeGenerated range should this run cover?”, ask “what arrived since the last run?”. Sentinel can’t answer that in the data lake - a Delta table in Fabric can.

The Mirrored Azure Monitor (preview) lets Fabric read Sentinel’s tables in place, without moving or copying them. It “doesn’t introduce extra storage or pipeline costs” - you only pay for the Fabric compute.

Log Analytics already stores its data in Delta format, where every write is a new, numbered commit in arrival order. A firewall log generated 45 minutes ago that landed 5 minutes ago sits in one of the newest commits. Spark in Fabric reads a Delta table commit by commit and remembers the last commit it processed in a checkpoint, so every run picks up exactly what arrived since the previous one - however late a log arrives, it is never skipped. These are exactly the two signals KQL jobs lack: the commit order replaces ingestion_time(), and the checkpoint remembers what the last run processed.

By timestamp, the late event lands in a range that was already processed. By arrival, it lands in a new commit after the checkpoint

Bins, not ranges

A summary row typically groups events by a few keys - source IP, destination IP, port - and adds a count and the first and last TimeGenerated of the group. That range is how an analyst finds the raw events behind the row. But one run processes recent events together with the occasional late one, so a single late event would stretch the range to hours. So instead, events are grouped into fixed 15-minute bins, and a late event gets its own row in its real bin:

The same batch as summary rows: one late event stretches the min/max row to 57 minutes, while each bin row keeps a 15-minute lookup range

Each row’s TimeGenerated is the processing time (or ingestion time); the event time is kept in BinStart and BinEnd. Since Analytics rules filter on TimeGenerated, this ensures a late 13:30 row falls into the next Analytics rule run’s window, not behind one that already ran.

The trade-off: a late event adds a second row for an old bin. So one 15-minute bin will have multiple entries with different TimeGenerated values. Your parser/rule has to be prepared to handle this.

Making it exactly-once

Spark processes the new commits in numbered batches. Every batch goes through two steps:

1. Summarize the batch and send it to Sentinel through the Logs Ingestion API.
2. Spark marks the batch as done in the checkpoint - automatically, once step 1 finishes without an error. This is a built-in capability.

Batch: a set of logs processed together, made up of one or more commits.

If a run dies before step 2, the checkpoint still points to the previous batch, so the next run replays the failed one - with the same batch ID and the same records. Nothing is lost. But the Fabric docs warn: “If you use custom sinks or foreachBatch, make the write logic idempotent because Spark might rerun a batch after a failure.” The Logs Ingestion API isn’t idempotent: if the run died after the summary was (partly) sent, the replay sends those rows again.

A crash between sending and checkpointing replays the batch - the duplicate shares its BatchId, so it is easy to drop

That only happens when a run dies at exactly the wrong moment. But duplicates are easy to spot: replayed rows have the same BatchId as the originals, while a genuine late event always comes from a different batch. This function handles the duplicates:

FwFlowSummary_CL
| summarize arg_min(TimeGenerated, *)
    by BatchId, BinStart, SourceIP, DestinationIP, DestinationPort, Computer, Protocol

So how does this compare to KQL jobs?

• At-least-once processing: yes. Every log that reaches the workspace is processed, regardless of how late it is.
• At-most-once processing: not on its own. A crash during or right after sending can send a batch twice - rare, and filtered out by the deduplication function above. Preventing it inside the pipeline (e.g. checking Sentinel before sending) is possible but adds complexity, so I left it out.
• Paying for the same data twice: no. Reading through the mirror incurs no data lake query charges, so there is no overlap to pay for. Fabric compute isn’t free, though: with an existing Fabric capacity, the extra cost is small; without one, a new capacity and the effort to run it can make it the more expensive option.

So, compared to KQL jobs, Fabric doesn’t miss late events, makes rare duplicates easy to remove, and avoids the trade-off between delayed processing and duplicates: no arbitrary waiting for late data, no overlapping runs.

Trying it yourself

The notebook is surprisingly small: Spark reads only the new commits of the mirrored table, a function summarizes each batch into bins and sends it to Sentinel, and the checkpoint tracks what’s done. I walk through the code and setup in Part 2.

Closing thoughts

Summary rules and KQL jobs only see TimeGenerated, so they have to guess how late the data is. Fabric processes by arrival, so with a checkpoint every event is sent at least once, and the small deduplication function makes sure it’s counted only once.

Notebooks aren’t new to Sentinel, but most SIEM engineers have happily avoided them. Now some things can only be done in Fabric notebooks. It is a forced move, but a useful one: data engineering is exactly what security teams will need going forward.