← Back to the archive
Analysis on X/Full text

The Intelligence Data Lake: When Technology Changes Analysis

The shift from pushing intelligence to pulling it, and the human relationship between collectors and analysts.

English translation of the Hebrew original.

Omri tells a news story that sounds deeply troubling to the average listener. But, at least as I understand it, he identifies the wrong problem in that story.

I will try to explain briefly why.

First, a clarification:

Nothing I write here comes from inside information. It comes solely from connecting publicly available information published after October 7.

Based on Omri’s wording, I suspect the system said to have been down on the night of October 7 was probably the “intelligence lake” described in Ben Caspit’s investigation published on February 24, 2024.

The intelligence lake—or data lake—is essentially a database into which the various collection agencies “pour” intelligence arriving from different theaters and sources. The analysis units, primarily Military Intelligence, then retrieve the information they want using different queries and analyze it.

In the past, before the intelligence lake existed, collection personnel in Unit 8200 were also responsible for delivering information to the various analysis units. As a result, the relationship between the network intelligence officer—the collection official—and the intelligence analyst was personal and frequent.

Among other things, that officer understood the role not only as collecting intelligence but also as providing warning. When reports with a strong warning component appeared, or reports the analysis unit ought to notice, the officer would call the relevant section and make sure its analysts were aware of the information.

Since the intelligence lake was introduced, the personal connection and warning function of collection personnel have been significantly eroded.

Because information is no longer pushed to analysis units but pulled by them, collection personnel have become accustomed to thinking their job ends once they feed the information into the lake. That is the problem Omri’s story points to.

An intelligence lake is an enormous database. Like any database handling countless sophisticated queries simultaneously—particularly one in the IDF—it is prone to malfunctions and crashes.

The correlation between the malfunction and the events of October 7 is probably not a causal relationship, and we should not look for one there.

What should concern us is how warning or near-warning information could accumulate among collection personnel on the night of October 7, without anyone bothering to pick up the phone and make sure the analysts who were supposed to receive it actually knew about it.

This is primarily a story about technology, and how it changes the way we work together.


Images and video attached to the original publication: Media 1 · Media 2 · Media 3 · Media 4 · Media 5 · Media 6 · Media 7