Databricks: Strengths, Weaknesses, and When Not to Use It

Databricks is an excellent platform that a lot of companies should not be using, and the honest assessment has to hold both of those at once. We build on it, we have migrated clients onto it, and we have talked clients out of it. Here is where it wins, where it hurts, and how to tell which case you are in.

Where it genuinely wins

Machine learning and data science at scale. Notebooks, MLflow, feature management and Spark on one platform over the same data. If ML is central to the business, Databricks is often the right centre of gravity.

Streaming and batch together. Structured Streaming and batch pipelines on a single engine, without maintaining two systems.

Open table formats. Delta Lake, and increasingly Iceberg, mean the data is not locked into a proprietary storage layer.

Genuine scale. Terabytes to petabytes, varied and unstructured data, workloads that would strain a conventional warehouse.

Where it hurts

Cost. Pricing scales with compute, and without cluster policies, auto-termination and workload governance, bills climb quickly. It rewards platform discipline and punishes casual use.

Operational weight. It assumes a capable platform team. Cluster configuration, job tuning and permissions are real work, and a company without someone to own them will struggle.

Analyst friction. Teams comfortable in a SQL warehouse find the environment heavier than they need. SQL warehouses within Databricks have narrowed this, but the gap is real.

Over-engineering gravity. Modest workloads placed on Databricks tend to acquire Spark jobs they did not need. The platform's power pulls projects toward complexity.

When not to use it

When your data is gigabytes, not terabytes. When the workload is reporting and dashboards and there is no ML or streaming. When the team is analysts rather than engineers. When simplicity of operation matters more than depth. In those cases Snowflake or BigQuery will be cheaper and simpler, and for a surprising number of companies a well-run Postgres with dbt on top is the honest answer.

Databricks or Snowflake

Databricks where ML, streaming and unstructured data are central and the team can run a Spark platform. Snowflake where the workload is SQL analytics and operational simplicity wins. Many organisations run both. The mistake is choosing on brand rather than workload, and a partner who recommends one platform for every client is not assessing your workload.

Two clients, two answers

A logistics company came to us running a few hundred gigabytes of shipment and telematics data, with an analyst team that lived in SQL and a leadership team that wanted dashboards. They had been advised to move to Databricks. We put them on a managed Postgres warehouse with dbt on top, and a year later their reporting is faster than it was and the platform costs a fraction of the original proposal. A fintech client with terabytes of transaction data, a fraud-detection model retrained weekly and a streaming requirement went the other way, and Databricks was right for them. The difference was not ambition or budget. It was the workload. The mistake in both cases would have been choosing on brand, and the logistics company came close to making it.

Cost controls that matter

If you do adopt Databricks, most of the cost stories that make it into conference talks come from a short list of missing controls. Cluster policies that cap instance sizes and enforce auto-termination, so a notebook left open on Friday does not run until Monday. Job clusters rather than interactive clusters for scheduled work. Photon and serverless SQL warehouses for the analyst workload, sized to actual concurrency rather than peak guesses. Tagging by team and project so the bill is attributable and someone owns each line. And a monthly review where the platform team looks at the top ten jobs by cost and asks whether each still needs to run. None of this is advanced. It is the operational discipline the platform assumes you have, and the bills climb for companies that do not.

Migration off, when it comes to that

We are occasionally asked to move a company off Databricks, usually because the workload turned out to be smaller than planned or the team that could operate it left. It is less painful than people fear, mainly because of the open table formats. Delta tables can be read by other engines, and Iceberg even more so, which means the data does not have to be re-exported. The work is in the pipelines: notebooks that grew into production jobs need to become versioned dbt models or scheduled tasks somewhere else, and that is the part to scope carefully. The same open formats are also the best argument for adopting Databricks if it does fit, because you are not locking the data into anything. Choose the platform for the workload you have, keep the data in an open format, and the decision stays reversible.

Questions to settle before you commit

Answer these honestly and the platform question mostly answers itself. How much data do you actually have, measured in the warehouse rather than estimated in a slide? Is there a machine learning workload today, with a model someone retrains, or is there a plan to have one? Does anything need to be processed as it arrives, or can everything wait for a nightly batch? Who on the team will own the platform, and have they run Spark before? What does the finance team want to see, and would a SQL warehouse give it to them by Friday? If the answers are gigabytes, no, no, nobody and yes, you do not need Databricks yet, and the money is better spent on the pipelines and the data quality that every platform depends on. If the answers are terabytes, yes, yes, an engineer who has and no, it is probably the right choice, and the cost controls above are the first thing to set up.

A note on the sales process

Databricks, like every platform vendor, has a sales team, and that team is measured on consumption. The proof of concept they offer will be well run and will show the platform at its best on your data. What it will not show is the operational cost of running it with your team after the vendor's solution architects go home. So take the proof of concept, and also ask two things they cannot answer for you: who on your side will own cluster policies and job tuning, and what the bill looks like in month six with real workloads rather than the demo dataset. If you have those two answered honestly, the platform decision is straightforward. If you do not, the proof of concept is the beginning of a relationship you have not priced.

The short answer

If your data is large, your workloads are mixed between engineering and machine learning, and you have or will hire people who can run Spark, Databricks is a strong choice and its cost is justified. If your data fits comfortably in a warehouse, your workloads are SQL, and your team is analysts rather than engineers, it is an expensive way to get what a warehouse plus a scheduler would give you. Most teams we meet are in the second group and have been told they are in the first. The proof of concept will not tell you which you are. Your team's skills and your bill in month six will.

How to decide

Inventory the actual workloads: volume, variety, ML, streaming, who consumes the output. Match the platform to that, not to the roadmap slide. Our data engineering practice does this assessment before recommending a platform, and our guide to choosing a data engineering company covers what else to ask. For the modernisation path off a legacy warehouse, see our data engineering services page.