SIEM vs. Data Lake: Why We Ditched Traditional Logging?

SIEM vs. Data Lake: Why We Ditched Traditional Logging?

In this episode, Cliff Crosland, CEO & co-founder of Scanner.dev, shares his candid journey of trying (and initially failing) to build an in-house security data lake to replace an expensive traditional SIEM.

Cliff explains the economic breaking point where scaling a SIEM became "more expensive than the entire budget for the engineering team". He details the technical challenges of moving terabytes of logs to S3 and the painful realization that querying them with Amazon Athena was slow and costly for security use cases .

This episode is a deep dive into the evolution of logging architecture, from SQL-based legacy tools to the modern "messy" data lake that embraces full-text search on unstructured data. We discuss the "data engineering lift" required to build your own, the promise (and limitations) of Amazon Security Lake, and how AI agents are starting to automate detection engineering and schema management.


Guest Socials -⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cliff's Linkedin

Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels:

-⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Podcast- Youtube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

If you are interested in AI Cybersecurity, you can check out our sister podcast -⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ AI Security Podcast⁠


Questions asked:

(00:00) Introduction(02:25) Who is Cliff Crosford?(03:00) Why Teams Are Switching from SIEMs to Data Lakes(06:00) The "Black Hole" of S3 Logs: Cliff's First Failed Data Lake(07:30) The Engineering Lift: Do You Need a Data Engineer to Build a Lake?(11:00) Why Amazon Athena Failed for Security Investigations(14:20) The Danger of Dropping Logs to Save Costs(17:00) Misconceptions About Building Your Own Data Lake(19:00) The Evolution of Logging: From SQL to Full-Text Search(21:30) Is Amazon Security Lake the Answer? (OCSF & Custom Logs)(24:40) The Nightmare of Log Normalization & Custom Schemas(28:00) Why Future Tools Must Embrace "Messy" Logs(29:55) How AI Agents Are Automating Detection Engineering(35:45) Using AI to Monitor Schema Changes at Scale(39:45) Build vs. Buy: Does Your Security Team Need Data Engineers?(43:15) Fun Questions: Physics Simulations & Pumpkin Pie

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(357)

Shadow AI & Sandbox Escapes: Why You Need an Agentic Control Plane?

Shadow AI & Sandbox Escapes: Why You Need an Agentic Control Plane?

When Claude Cowork hits a roadblock, it doesn't give up, it writes a custom Python script and downloads an untrusted NPM package just to bypass its restrictions and finish its goal. Are your security ...

18 Aug 32min

How Adobe Uses AI Agents for building a WAF Pipeline?

How Adobe Uses AI Agents for building a WAF Pipeline?

When vulnerability disclosures shrink the wild exploit window down to less than 24 hours (or even minutes), traditional manual patching and WAF rule creation simply cannot keep up.In this episode, Ash...

4 Aug 47min

Why Runtime Agents Are Replacing Static Posture Checks

Why Runtime Agents Are Replacing Static Posture Checks

The attack window from the discovery of a vulnerability to its exploitation has shrunk to less than 24 hours and sometimes down to just 25 minutes. Is your security team prepared for the speed of AI-d...

28 Juli 44min

The Hidden Cost of BlackBox AI: Bridging Cloud and Code Security

The Hidden Cost of BlackBox AI: Bridging Cloud and Code Security

Traditional SCA and SAST tools are notorious for drowning security teams in false positives, historically flagging nine out of ten alerts incorrectly. But while generative AI seems like a magic bullet...

9 Juli 42min

Who Governs Your AI Agents? Identity, Offboarding & Open Standards

Who Governs Your AI Agents? Identity, Offboarding & Open Standards

Are overprivileged AI agents the biggest emerging threat in cybersecurity? In a recent high-profile attack, a vibe-coding company had its entire source code stolen because an attacker exploited a long...

2 Juli 34min

AI-Powered Forensics: How Attackers Automate Breaches

AI-Powered Forensics: How Attackers Automate Breaches

AI isn't necessarily creating impossible new attacks, but it is drastically lowering the technical barrier to entry for cybercriminals. In this episode, Ashish Rajan speaks with Simon Biggs, Cyber Inc...

23 Juni 39min

The 4 Pillars of AI SOC:From Threat Hunting to Vibe Hunting

The 4 Pillars of AI SOC:From Threat Hunting to Vibe Hunting

Threat hunting has officially evolved into "vibe hunting". However, if your AI security tools lack the right semantic context, they might be doing more harm than good. In this episode, Ashish sits dow...

16 Juni 46min

Native Cloud Firewalls Falling Short in a Multicloud World

Native Cloud Firewalls Falling Short in a Multicloud World

As enterprises expand across multiple cloud environments, on-premise data centers, and dynamic AI workloads, traditional perimeter defenses and siloed cloud-native tools are no longer enough to secure...

11 Juni 36min

Populärt inom Teknik

uppgang-och-fall
market-makers
skogsforum-podcast
rss-elektrikerpodden
rss-laddstationen-med-elbilen-i-sverige
rss-uppgang-och-fall
rss-en-ai-till-kaffet
natets-morka-sida
hej-bruksbil
bli-saker-podden
garagehang
 och-bilen-gar-bra
bosse-bildoktorn-och-hasse-p
under-femton
gubbar-som-tjotar-om-bilar
elbilsveckan
klocksnack-tillsammans-med-nymans-ur-1851
rss-milpodden
bilar-med-sladd
rss-fabriken-2