Why Databricks serverless exists, and which of your jobs should stay on classic clusters

Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story.

Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines that are barely breathing for most of the run. Wait, or waste, and there was never a third option.

In this episode:

- Why the cluster wait was only the receipt, and what classic Databricks compute never actually solved

- Why Databricks can keep compute warm when your own team never could, and what that shows up as in the rate

- What "serverless" means here, jobs and notebooks, and why a serverless SQL warehouse is a different conversation

- The thing a fixed cluster was quietly doing for your budget that serverless stops doing, and why the policy you're thinking of doesn't cover it

- Three tests to run on your own workloads this week, plus the ones that can't move at all no matter what the math says

This episode is for Databricks data engineers who have serverless sitting in a dropdown and no real argument either way. Whether you're defending a migration in an architecture review or explaining next month's bill, you'll walk away able to say which of your jobs should move, which should stay on classic, and why.

---

Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every week.

LinkedIn: linkedin.com/in/jrlasak

Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(26)

How Databricks engineers lose the final round (and the 4 places it happens)

You cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough decision. You go back looking ...

14 Sep 10min

The real reason your Databricks week never got simpler - it isn't the lakehouse

Your org finished the migration. Two systems became one. The nightly copy nobody wanted to own is gone, and so is the quarterly meeting where finance and the data science team each showed up with a re...

7 Sep 11min

The Databricks incident where the most senior engineer never touched the keyboard

Thursday, just after two in the afternoon. The first message doesn't come from an alert, it comes from a person in revenue operations: the biggest region jumped overnight and nobody sold anything new....

31 Aug 11min

9 Databricks decisions that reveal your real seniority (and one that only looks senior)

Three in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun button is right there. What you do in the next ten seconds says more about your ...

24 Aug 11min

The Databricks job market in 2026: there are two of them, and only one answers

One engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and hearing nothing. Same month, same platform, sometimes the same city.That sounds like no...

17 Aug 11min

Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)

Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee...

10 Aug 10min

Build the platform or use the platform: the two Databricks data engineer tracks nobody names

Two Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year go by, and they are not doing the same job.Both get rated strong. Both get told th...

4 Aug 11min