Modern enterprise data platforms face a key challenge: legacy codebases, isolated streaming feeds, unmanaged costs, and rigid retrieval tools struggle to support modern AI. Trapped mainframe logic and ungoverned streaming feeds create severe compliance gaps and ballooning expenses. Leading organizations solve this by modernizing legacy systems onto Delta Lake, enforcing governance at streaming commit time, automating FinOps, and deploying custom reinforcement learning agents. This edition highlights how code translation, streaming governance, operational observability, and advanced reasoning accelerate enterprise data operations.
In This Edition
- A Use Case Spotlight on how financial institutions and global enterprises leverage LLM-assisted translation pipelines on Databricks to convert legacy mainframe COBOL into governed PySpark and Delta Lake tables.
- A Partner in Focus on Unravel Data and how its AI-powered observability and FinOps platform delivers granular cost allocation, query tuning, and automated DBU optimization across Databricks workloads.
- A Featured Video demonstrating StreamNative and Databricks introducing catalog-based commits to bring real-time streaming feeds under native Unity Catalog governance without post-processing.
- From the Editor's Lens exploring KARL, a custom reinforcement learning agent framework from Databricks AI Research designed to deliver faster, lower-cost grounded reasoning over complex enterprise data.
Autonomous Mainframe Modernization & Legacy Code Translation Engine
Core banking engines, insurance claims systems, and retail logistics still run petabytes of daily business transactions on decades-old COBOL and PL/I codebases. Operating these monolithic environments incurs exorbitant mainframe licensing fees, risks severe developer knowledge loss, and creates data bottlenecks that delay real-time analytics. Converting millions of lines of procedural, undocumented legacy code into cloud-native pipelines without breaking business logic remains a critical engineering friction point.
The Databricks Solutions
Deploying an autonomous mainframe modernization engine built on the Databricks Data Intelligence Platform transforms legacy procedural code into modern, governed PySpark and SQL pipelines running on Delta Lake. The architecture utilizes Mosaic AI to parse legacy COBOL copybooks into structured Delta Live Tables definitions. Governed by Unity Catalog, raw EBCDIC datasets are converted into Medallion layers using serverless compute and the vectorized Photon execution engine, making historical data instantly available for downstream AI and operational analytics.
Who's Already Doing This
Global banking institutions, healthcare networks, and retail enterprises are accelerating legacy data modernization on Databricks:
TD Bank
Consolidated petabytes of enterprise data onto Azure Databricks in partnership with Microsoft and Accenture, as detailed in their cloud migration architecture study, eliminating legacy on-premises Hadoop constraints and powering scalable banking analytics.
Walgreens
Replaced legacy Teradata and Hadoop batch systems to build a unified pharmacy data engine, documented in their official Databricks retail customer story, enabling near real-time inventory forecasting across 9,000 locations.
HSBC
Modernized core financial processing by consolidating 14 isolated databases onto Delta Lake, highlighted in their PayMe mobile platform case study, cutting data processing times from hours to seconds and enabling real-time analytics.
Nasdaq
Modernized legacy financial reporting and core market intelligence infrastructure on Delta Lake, as documented in their Databricks capital markets customer story, eliminating legacy batch processing bottlenecks and accelerating complex financial analytics.
Why This Use Case Continues to Expand
Rising mainframe operating costs and demand for sub-second AI access make legacy batch windows untenable. Databricks serverless compute and automated code translation workflows allow teams to validate code continuously, turning high-risk rewrites into automated, low-risk pipeline migrations.
Who Should Care
This use case delivers the highest ROI for platform leaders who:
Maintain legacy IBM z/OS mainframes running COBOL or PL/I processing jobs that delay real-time reporting.
Struggle to extract, parse, and structure binary EBCDIC data files and mainframe copybooks for cloud consumption.
Require fine-grained access governance and automated data lineage tracking over converted datasets to maintain continuous regulatory compliance.
Key Takeaway
Replacing legacy mainframe batch windows with AI-assisted code translation and real-time lakehouse ingestion turns brittle legacy logic into cloud-native assets. Early adopters are reducing operational overhead, eliminating expensive mainframe MIPS processing charges, and unlocking decades of core business logic directly within a governed data platform.
AI-Powered DataOps, Observability, and FinOps Optimization with Unravel Data
Unravel Data provides an enterprise-grade DataOps, performance observability, and FinOps optimization platform built specifically to manage complex data stacks. As organizations expand their footprint across interactive SQL, streaming, and machine learning workloads, maintaining granular cost visibility and optimizing compute efficiency becomes paramount. Unravel’s deep integration with the Databricks Data Intelligence Platform empowers platform administrators and engineering teams to monitor, troubleshoot, and optimize Databricks Units (DBUs) automatically, eliminating wasteful cluster configurations and maintaining cost predictability across multi-cloud environments.
Partner Capability Snapshot
Strategic Engineering
Connects directly to Databricks workspaces, clusters, and SQL Warehouses using low-latency telemetry collectors, capturing job execution traces, query profiles, and resource utilization metrics in real time.
Developer Productivity
Delivers AI-driven recommendations and automated root-cause analysis to pinpoint failing jobs, slow-running SQL queries, memory bottlenecks, and data skew directly within a single operational console.
Certified Expertise
Integrates governance controls alongside Unity Catalog metadata, tracking DBU consumption by team, workspace, user, or project to enforce budget guardrails and financial accountability.
Add-ons/Accelerators
Features automated cluster right-sizing tools and cloud migration cost predictors to help teams optimize compute policies and scale lakehouse workloads cost-effectively.
Project Experience
Powers observability and cost governance for enterprise engineering teams across financial services, technology, healthcare, and retail globally, helping organizations optimize millions of DBUs.
Geographic Presence
Delivers native multi-cloud support across AWS, Azure, and Google Cloud environments, supporting global enterprise deployments across North America, Europe, and Asia-Pacific.
Native Streaming-to-Lakehouse Governance with Unity Catalog
Speakers
Benjamin Mathew
Product Manager, Databricks
Kundan Vyas
Director of Product & Partnerships, StreamNative
A Quick Summary
In this technical session from the Databricks Data+AI Summit, Benjamin Mathew and Kundan Vyas showcase how a native integration between StreamNative and Databricks solves the historic governance gap in real-time streaming architectures. The session introduces catalog-based commits, an architectural approach where real-time streaming data from Kafka and Pulsar topics written in Delta Lake or Apache Iceberg formats is registered directly inside Unity Catalog at the moment of commit. This eliminates manual metadata synchronization and ensures continuous access control across high-velocity event streams.
Key Topics Discussed
✓
Eliminating Streaming Governance Gaps: Solving the challenge of ungoverned external cloud storage writes by registering streaming topics directly inside Unity Catalog at commit time.
✓
Catalog-Based Commit Architecture: Understanding how real-time streaming writes achieve instant query-readiness and transactional consistency without post-processing steps.
✓
Unified Access Control and Lineage: Enforcing row- and column-level security policies and maintaining automated data lineage continuously across both batch and streaming pipelines.
✓
Automatic Storage Optimization: Utilizing Unity Catalog background processes like auto-compaction and file management directly on incoming streaming tables to prevent the small-file problem.
✓
Accelerating Real-Time AI Workflows: Streamlining the ingestion path from high-throughput streaming events directly into AI-ready, fully governed lakehouse tables.
Why It's Worth Watching
This presentation provides a practical blueprint for data architects and platform managers looking to unify real-time messaging systems with lakehouse governance. If your organization struggles to enforce consistent security policies, data lineage, and cost controls across streaming Kafka or Pulsar feeds, this demonstration delivers immediate technical value.
Meet KARL: A Faster Enterprise Knowledge Agent Powered by Custom Reinforcement Learning
A Quick Summary
Deploying enterprise AI agents for grounded reasoning over proprietary knowledge bases frequently encounters severe trade-offs between answer quality, response latency, and serving costs. Databricks AI Research introduced KARL (Knowledge Agents from Reinforcement Learning), a specialized agent architecture built to overcome these limitations. By applying custom reinforcement learning techniques rather than relying solely on prompting massive frontier LLMs, KARL matches top-tier model accuracy on complex multi-step retrieval tasks while reducing query latency and inference expenses.
Key Topics Discussed
✓
Custom Reinforcement Learning for Agents: Moving beyond basic prompt engineering by training agents using RL algorithms tailored specifically for enterprise retrieval and multi-step reasoning.
✓
High-Accuracy Grounded Reasoning: Enabling agents to cross-reference multiple evidence sources, synthesize complex document relationships, and reduce hallucinations on enterprise knowledge tasks.
✓
Latency and Serving Cost Reduction: Achieving high accuracy while significantly lowering inference costs and response times compared to standard general-purpose frontier models.
✓
Private Preview RL Capabilities: Exposing the underlying reinforcement learning framework to Databricks customers to allow teams to train custom, domain-specific enterprise agents.
✓
Enterprise Knowledge Integration: Bridging the gap between raw unstructured document repositories and interactive, highly accurate AI copilots governed under unified workspace permissions.
Why It's Worth Reading
This research milestone signals a key direction in enterprise AI: shifting from parameter-heavy generalist models to domain-tuned, RL-optimized agents. Data leaders and AI engineers should evaluate this development as a practical path to scaling internal knowledge assistants, operations QA bots, and support copilots without encountering prohibitive token costs or unacceptable latency.
Until Next Time
Whether translating legacy COBOL logic onto Delta Lake, optimizing DBU consumption with Unravel Data, enforcing catalog-based commits on live streams, or deploying RL-optimized knowledge agents with KARL, enterprise AI success relies on a performant, well-governed data foundation.
Your Action Item: Audit your legacy modernization and streaming ingestion entry points. Ensure streaming feeds are governed at commit time, optimize your compute clusters for cost efficiency, and verify that your enterprise AI agents operate over clean, well-governed lakehouse assets.
See you in the next digest.