How an Insurance Company Built an ML-Powered Underwriting Engine on Databricks

Underwriting decisions for commercial policies took 5-7 business days. That's not because the underwriters were slow - they were drowning. Each commercial policy required reviewing 40+ data points spread across 6 different systems: the Guidewire policy admin system, claims history database, third-party property data feeds, weather and catastrophe models, credit scoring APIs, and inspection reports stored as PDFs in a SharePoint folder. Underwriters spent most of their time re-keying information between screens, not actually making risk decisions.

The loss ratio had been trending upward at 2.3% per year because the pricing models hadn't been recalibrated since 2021. The actuarial team had built sophisticated R models that could fix this, but "deploying to production" meant emailing an Excel file to the IT team and waiting 6 weeks for a release cycle. By the time a model went live, the data it was trained on was already stale. The actuaries were frustrated. IT was frustrated. The CEO was watching the combined ratio creep toward unprofitable territory.

We built a Databricks Lakehouse that unified all 6 underwriting data sources into a single Delta Lake layer. Each source got its own ingestion pipeline: Guidewire data via JDBC, claims history via a nightly CDC feed, third-party property data via API, weather models via scheduled batch pulls from NOAA and RMS, credit scores via real-time API calls, and inspection reports via a PDF extraction pipeline using Databricks' document AI capabilities.

The actuarial team's existing R models were versioned and deployed through MLflow - no more emailing Excel files. We set up an MLflow Model Registry where actuaries could push new model versions from their R notebooks, run automated validation against holdout datasets, and promote to production with a single click. Alongside the R models, we built Python gradient boosting models (XGBoost) that score risk in real time using the unified feature set. Both model types are served via MLflow Model Serving behind a REST API.

The underwriting workflow was redesigned around this API. When a new submission comes in through Guidewire, an event trigger calls our Databricks endpoint. It pulls all 40 data points automatically, runs them through both the actuarial R model and the XGBoost model, and returns a risk score with a feature-importance explanation showing which factors drove the score. The underwriter now sees a pre-populated screen with all the data they need and a recommended price range. They review and approve - instead of researching from scratch.

The single biggest unlock wasn't the ML models - it was getting actuaries and data engineers working in the same platform. Before this project, the actuarial team worked in R on their laptops, and the engineering team worked in Python on separate infrastructure. Putting both in Databricks notebooks with a shared MLflow registry meant they could collaborate without anyone having to learn a new language. The actuaries kept writing R. The engineers kept writing Python. MLflow didn't care.

Guidewire integration was harder than expected. Their Cloud API has rate limits that aren't well documented, and the event webhook payload schema changes between Guidewire versions without clear deprecation warnings. We had to build a retry queue with exponential backoff and a dead-letter mechanism for failed API calls. If you're integrating with Guidewire, budget an extra 2-3 weeks for API edge cases.

One thing we'd do differently: we should have started with the data quality layer before building models. The third-party property data feed had a 12% null rate on building square footage - a critical feature for property risk scoring. We didn't catch it until model validation showed unexpectedly high error rates on commercial property policies. Adding data quality checks (using Great Expectations) upfront would have saved us a week of debugging downstream.