- • Organizations running Databricks receive two separate bills: one from the cloud provider for infrastructure and one from Databricks for processing (DBUs).
- • Leadership could not easily see what each team or product actually costs, because combining the two bills manually is error-prone.
- • Glisten built a FinOps (cloud cost management) platform on Databricks that runs identically on AWS and Azure, productized for client delivery.
- • A single cost view combining cloud infrastructure spend and Databricks DBU spend.
- • Clean-up of misspelled, inconsistent and missing resource tags so every cost can be attributed.
- • Anomaly detection for forgotten always-on compute, cost spikes and storage growth.
- • Fully automated CI/CD with quality checks before every release.
- • One codebase deployable to both AWS and Azure.
- • Secure access with no stored cloud credentials.
- • Fine-grained data governance so each business unit sees only its own costs.
- • Built a medallion architecture (Bronze, Silver, Gold) on Databricks to ingest and refine hourly cloud billing data.
- • Used Auto Loader for incremental ingestion from S3 (AWS) and ADLS Gen2 (Azure) with schema evolution.
- • Implemented tag normalization and typo mapping, with unattributed spend tracked as a KPI.
- • Defined jobs and environments with Databricks Asset Bundles, keeping everything as code in a single Git repository.
- • Built a CI/CD pipeline in GitHub Actions: pull requests run unit tests and bundle validation; merges deploy, run the pipeline and execute automated quality gates.
- • Configured keyless authentication using AWS IAM OIDC and Azure Entra workload identity federation, with OAuth service principals for Databricks.
- • Implemented Unity Catalog row filters, column masking and automatic end-to-end lineage.
- • Delivered AI/BI dashboards, budget alerts and a Genie space for plain-English cost questions.
- • Identical pipeline runs on AWS and Azure from the same codebase.
- • 56,000+ billing line items processed across 12 cloud services.
- • All planted cost anomalies detected, including a 15x network-cost spike.
- • Quality gate correctly blocked a bad release during the build.
- • Changes tested, deployed and verified in minutes with no manual steps.
- • Zero cloud credentials stored anywhere in the pipeline.