A security data lake is a central repository that stores large amounts of security data, such as logs, alerts and telemetry from firewalls, endpoints, cloud services and identity systems, on low-cost storage for long periods. It applies the general data lake approach to security: keep raw or lightly processed data in one place, separate storage from the tools that analyze it, and search or analyze it when needed. It is not a standardized product category; the term covers do-it-yourself builds on cloud storage, vendor platforms and lakes built into security tools.
At a glance
- A security data lake keeps security data in one place, usually on cloud object storage, at a lower cost per gigabyte than typical SIEM ingestion.
- Storage is separated from analytics, so several tools can query the same data.
- It is mainly used for long retention, investigations, threat hunting and compliance.
- It usually complements a SIEM rather than replacing its real-time detection.
- Savings depend on query and compute charges as well as storage, and on having the skills to run it.
What problem it solves
Security teams want more data, for longer. Investigations often need logs from months ago, and attackers can sit undetected for a long time. But many SIEMs price by the volume of data ingested, so high-volume sources such as DNS, proxy, network flow and cloud audit logs get expensive quickly. The usual result is that teams drop sources or cut retention to stay within budget, which leaves gaps exactly when an incident needs them.
A security data lake changes the economics. Data lands in cheaper storage, often in open formats, and is analyzed only when needed. That lets teams keep a much fuller record without paying SIEM rates for all of it. It also gives one place to bring together data from many security tools, instead of each tool holding its own silo.
How it works
Collection. Logs and telemetry are shipped from sources such as firewalls, endpoint agents, identity providers, cloud platforms and email security into the lake. Some organizations route data through a pipeline tool first that filters, reshapes and decides which data goes to the SIEM and which to the lake.
Storage and format. Data is stored on low-cost storage, usually in a public cloud. Many approaches normalize data into a common schema so that a firewall log and a cloud log describe users, devices and addresses in the same way. Open schemas and formats make it easier to use several tools on the same data.
Analysis. Analysts run queries for investigations and hunting, often through SQL-style tools, notebooks or a search interface. Some SIEM, extended detection and response (XDR) and analytics products can query the lake directly or run detections against it on a schedule.
Retention and access. Policies set how long each type of data is kept, who can access it and how it is protected, which matters because security logs often contain personal data.
When it matters for buyers
- When SIEM renewal quotes jump. Moving high-volume, low-alert data to a lake is a common way to control cost. See our SIEM overview for the wider set of options.
- When compliance requires long retention. Some regulations and contracts require logs to be kept for a year or more; check what applies to you.
- When investigations keep hitting missing data. If incident responders ask for logs you no longer have, retention is the problem.
- When choosing a managed provider. A co-managed SIEM or MDR provider may run the lake, require their own, or work with yours.
- When you want to avoid lock-in. Data in open formats on storage you control can outlast any one analytics tool.
Questions to ask vendors
- Where is the data stored, in whose cloud account, and in what format?
- How are storage, ingestion, query and compute priced, and can you model our expected volumes and search patterns?
- Which schema do you use, and can other tools read the data directly?
- Can detection rules run against lake data, or is it search-only?
- How fast are searches over months of data?
- How do you handle retention, deletion and access controls for personal data in logs?
- If we leave, how do we get our data out and in what form?
How it differs from SIEM
A SIEM is a detection and response tool: it collects logs, correlates them in near real time, raises alerts and supports investigations, usually with storage bundled into its pricing. A security data lake is mainly a storage and analysis layer: it holds more data, for longer, at lower cost, but on its own offers limited real-time alerting or case management. Basic log management collects and retains logs too, but usually without a lake’s scale or the ability for many tools to query the same data. In practice the lines are blurring, as some SIEMs now run on a lake and lakes gain detection features, so compare what each platform actually does rather than its label.