Databricks Cost Monitoring - KRATTI GenAI Agent solution

Jan 13

3 min read

Unexpected cost spikes caught you off guard and caused budgeting panic again?

Have you received a message from the controller after a long time? What a nice surprise. But wait a second, your smile fades as soon as you see the content — costs have exceeded budgets and the cost spikes came as a complete surprise. Now, you need to get things under control quickly, prepare for countless internal meetings to go over the situation and ensure that this doesn’t happen again in the future. And all of this takes time away from business development. The very thing that brings in revenue, instead of just firefighting...

You're not alone in facing these challenges. Our own Databricks costs also skyrocketed unexpectedly on a monthly basis, completely wiping out our budgets. The culprit? An LLM model we were hosting had been left running accidentally. In just a couple of days, we managed to burn through an entire month's budget. And within a week, several months’ worth. To make sure this never happens again, something had to be done.

Almost everyone has encountered similar situations, especially when adopting new technology. Databricks is certainly no exception. As functionalities and user adoption evolve at a rapid pace, costs inevitably increase as well. That's why precise cost monitoring becomes crucial — not only for budgeting but also for calculating the ROI of different data solutions. But in today's world, who really has the time for cost monitoring and optimization when data professionals are already drowning in an ever-growing stream of new projects.

Welcome GenAI Agents

Our very own GenAI Agent, KRATTI, was born to meet this need. We built a solution that autonomously monitors costs at the Databricks Account level, automatically reporting any anomalies. Every morning, I receive a summary in Slack of the previous day's costs, allowing me to check the situation at a glance on my phone as soon as I wake up. Since implementing KRATTI, our costs have never spiraled out of control and I know exactly where every euro is going on a monthly basis. With KRATTI, we can develop freely without worry — because if anything unexpected happens from a cost perspective, we’ll be the first to know.

KRATTI in action

New fun features

We also realized that daily cost monitoring alone wasn’t enough, so we enhanced KRATTI with the ability to analyze and optimize interactive clusters on a daily basis and SQL warehouse behavior on a weekly level. For security reasons, KRATTI doesn’t modify anything itself but instead provides clear, actionable recommendations on how to optimize a cluster based on its actual usage and how much money the adjustment would save. This has allowed me to fine-tune clusters where default settings were incorrectly left in place, such as an improperly configured auto idle time.

In addition, KRATTI can monitor components in real-time, including DLT, Clusters, Workflows, and SQL Warehouses, to detect any resources that may have been left running unintentionally — whether due to an infinite loop or simple oversight. These kinds of incidents happen all the time, and in the worst case, a DLT pipeline could be stuck running for weeks in a development environment before anyone even notices. But now, thanks to KRATTI, we no longer have to worry about these issues.

Databricks Cost Monitoring on a new level

So, Where can I learn more about KRATTI? This question has surely crossed your mind. You can learn more about KRATTI through this link: KRATTI | Cost Guardian

Ikidata is a pioneer in GenAI Agent automation, providing deep insights into this emerging technology from technical, architectural, and business perspectives. We make it simple to bring your ideas to life.