Causes of Machine Learning’s productivity problem: Metrics

Published on LinkedIn 2022-01-20

The last two parts of the series: Metrics and Processes are unquestionably interlinked. I think the source of Process problems often originate from a blind spot left by not defining appropriate metrics, so I start there. 

Please follow me if you want to be notified of new content. I write daily about ML/MLOps with no hype or BS.

“If you can’t measure it, you can’t manage it.” said Peter Drucker

Which area is better suited to this quote than the purely data-driven, data-fuelled, data everything Machine Learning?

Yet, people regularly start projects with a hazy idea of where they should go and what they should do. This is even more prevalent in POC thinking, where the host is not convinced of success. Hence, until there is some “progress”, the organisation refuses to think through how the project will affect users (and the business itself).

Failure modes in Data Science projects 

About a year ago, I created a repository of articles about how DS/ML projects fail and categorised them into five major and many subcategories:

• Organisational: Leadership, Employees, Infrastructure

• Intermediate: Legal/Privacy/Bias/Security, Transparency/Communication

• Product Planning: Business Value, Specification

• Product One-Off: Project Execution, Data, Modelling

• Product Ongoing: Operations

As you can see, these cover all aspects of a project. Unfortunately, a successful project needs to achieve a feasible state in all of these.

Photo by Ian Taylor on Unsplash

Link business objectives to quantitative factors

At the project planning stage, seek out stakeholders and talk about the relevant area. This is a discovery phase, not a technical phase. Find out the consequences if the model works one way or another. What are the risks and costs? 

For example, an F1 score or a precision metric doesn’t mean anything. But they can surely answer how much the company loses if a client is repeatedly exposed to Type 1 or Type 2 errors. 

Is your company work in an area concerned with bias and privacy? Talk to the lawyers about how they greenlight other non-ML features. What QA tests do these features need to pass? Meet these with project metrics of your own.

Better project planning and execution 

ML projects often need to change direction mid-course. A well-defined metrics framework allows you to reason about these changes and make informed decisions. It helps you communicate progress toward leadership who might not be familiar with statistical measures. During project execution, conversations about prioritisation will be data-driven instead of subjective feelings (and how loud someone is at staff meetings). But more on this tomorrow…