3 indicators to ensure data science models achieve impact in commercial insurance
Jason Goodman from Unsplash
Every data science team in the P&C commercial insurance space has seen its fair share of models that have failed to achieve the hoped-for business benefit. If you are a leader in this space, how can you maximize the return on your data and analytics investment?
During the last seven years, my data science team has been tackling this problem. One of the most effective actions that we started taking is monitoring every implemented model for three indicators:
Accuracy,
Execution and
Impact.
Why these 3 indicators?
Ultimately, what we care about is the impact. That is, a measurable business value generated by the model. But impact can take time to materialize, particularly for long-tailed commercial insurance lines. As a result, we want to see that the tool reflects (1) accuracy and (2) effective use (execution), which should ultimately lead to (3) the desired impact.
An example of measuring these 3 indicators for a model that streamlines submissions
Let’s say you build a model that streamlines submissions. The model intakes an emailed submission and automatically populates the underwriting tool, reducing expenses and producing faster quotes. Great! Now you want to measure the three elements.
Accuracy. At a minimum, make sure the model takes the submission details and accurately populates the underwriting tool.
Execution. Let’s say the underwriters currently receive submissions via email. We want underwriters to use the new model, which means that they forward emailed submissions to a central service for the model to process, and they use the output from the model to underwrite and quote. Track the extent that both these things are happening.
Impact. Compare the turnaround time for quotes before and after model implementation. Ask how the results translate either into expense savings or top-line growth, because you’re able to quote more with the same team.
To achieve this good result, let’s take a deeper look at measuring each of these three elements.
1. Measuring accuracy
Monitoring your model for accuracy after you roll it out has several benefits.
It assures everything is running OK, meaning that the inputs are being fed in as intended, and calculations are being performed.
It provides evidence to secure more end user buy-in for the model. For example, you can say - “I know you were skeptical when the model rolled out six months ago, but since then you can see that those accounts flagged as high risk already have ten times more reported claims.”
You will see the accuracy deteriorate over time and you can monitor this to decide when to give the model a refresh.
An example on measuring accuracy for a model that streamlines submissions
Let’s take the example of the streamlined submission intake.
How will you monitor accuracy? Some monitoring processes are automated, and some are manual. In this example, check that the model accurately extracts information from submissions by manually conducting random audits to see if the underwriting tool is being populated correctly.
How will you judge the model’s accuracy? The model needs to be accurate enough to fit the purpose. For example, if the model is streamlining submissions for no-touch underwriting, there should be a much higher bar of accuracy compared to a situation where an underwriter reviews the submission for reasonableness.
2. Measuring execution
Measuring execution in commercial lines is critical. In commercial lines, an internal end-user usually sees the model output and then decides what to do with it. When models fail to have impact, it is usually because the end-user is not acting on the tool’s output.
The biggest and most common problem I see is the inability to measure execution. How does this happen? Let’s look at some examples.
An example of measuring execution for a model that streamlines submissions
When reviewing the streamlining of submissions, for example, seeing if underwriters are forwarding their submissions into the central service and using its output should be straightforward.
As another example, let’s say you implement an artificial intelligence model that uses drone images to provide insight about a property’s roof quality. Underwriters see an account’s roof quality in their underwriting tool, and they are supposed to consider this information in their overall underwriting process. How will you know if any underwriters are considering the new data in their overall underwriting process? You don’t.
This is not a good situation. In my experience, even when we’re able to track execution, we only see a fraction of the execution that's needed. So, if you can’t tell if anyone is using the model, assume that no one is.
Here are two ways to fix this problem:
You can ask the underwriters to hit a button in the tool to see the model results. Tracking this will show how often they are actively seeking the model output.
You can be more prescriptive on the action based on model output. For example, if the roof quality score is poor, and the property is in a high hail zone area, underwriters should cap the limits at $5 million. This action can be tracked.
The bottom line is to put extra effort into tracking execution. It’s usually the weakest link in achieving impact, and you might be surprised at how hard it is to track.
3. Measuring impact
Ultimately, your model's goal is to either increase premium, reduce loss ratios or lower expenses. The goal should never be building a better or more sophisticated model.
It is good practice to pick one of these three basic goals and do a simple calculation to show the model’s impact.
Good criteria for impact is:
you describe the impact in the same way the business does,
the end-users are happy to say that you helped them generate this impact and
it’s a simple calculation.
An example of monitoring impact for a model that streamlines submissions
Let’s come back to our example of the model that streamlines submissions. Ideally, you can say, for example, “This model reduced the time to quote by two weeks, allowing us to quote 10% more new business with the same team size and grow the top line by $10 million gross written premium.”
Conclusion
After rolling out your model, monitoring it for accuracy, execution and impact is a critical part of delivering business impact. But what is most important is what you do with the results. In particular, if you see no execution, what do you do? If you are interested in reading an article on that topic, please let me know in the comments below!