What's still wrong with MLOps?
David Hershey's excellent article is making the rounds in the MLOps space. In the article, he advocates for better practices and culture in MLOps rather than more tools.
We have enough tools already.
He is voicing similar thoughts that I wrote in my article "What's wrong with MLOps?", which equally did the rounds almost a year ago.
If this knowledge is commonplace and wildly accepted, why are we still arguing about it?
The answer lies in the market. As someone in the business of selling culture and best practices in ML, I can confirm that it is _incredibly_ difficult to sell anything other than products.
We have a free taster presentation which I also presented at PyData 2022, that I do for prospective ML teams. After one of these sessions at a large UK retailer, I bumped into their Head of DS at a meetup, and he said: "Laszlo, I would love to hire your services because the team was really excited about your content, but we simply can't. Like there is no mechanism to pay for your services."
I find it surprising how little involvement large organisations have in developing best practices and recommendations. ML is a brand new area that will be hugely important in the coming decades. The companies that get it right will have a massive advantage over their competition, who then will play catchup with a runaway train.
If your team struggles with getting this right, how do you think they will get better if you don't invest in best practices? Do you expect them to go and figure it out in their free time? They might not have enough free time. So they will experiment with their work. Do you think this is an efficient way of building a better ML practice? What if they are unlucky and don't step on the right solution? Are you ready to scrap their attempts and restart from scratch? Because that's regular in trial and error. What are the political implications in the organisation of the above? So what happens then? Nothing, that's what happens. ML quagmire.
But why don't people buy advice in ML? First, there is a sense of "not getting anything". You, as an advisor, turn up, teach everyone, write process documents and leave. Is that value for money? From a bunch of guys in jeans? Most managers think their employees should go out and train themselves for free at meetups.
Consulting is a reputation business, and Big4/MBB does it best. But they also don't deal with problems at the ML level. They won't be able to tell you the latest trends in DS code practices or a novel way to build your stack. You want a five-year organisation plan.
So this leaves one engagement: Outsourcing. The company has a fixed problem, can't hire enough talent, and they are looking for a temporary solution. That's not going to help the culture, is it?
The exact opposite happens when they buy a product: They see a slick presentation from a VC-funded growth team with inflated numbers and magic use cases. In exchange for their money, they get a concrete, tangible outcome that they can showcase at their next all-hands. Will it change culture?
But if all they buy are products, then VCs will only fund products. These products also need to be "VC sellable". You need to be able to explain them in a 30-second pitch, or the partners will tune out. How do you explain culture in 30 seconds? Are you surprised that millions of laser-targeted MLOps products pop up everywhere? In a legendarily cross-functional space? How will you integrate all these? That's not the problem of the startup and the VC. That's the problem of the buyer who is at the business end of a slick sales pitch, as I wrote above.
These companies also use their marketing budget to "flood the zone" with their messaging about stacks, stack comparisons, "buy vs build", and other sales-related content. This prohibits buying personas' minds from formulating an alternate strategy (don't forget they have a business to run, and they can't accept help as described above) and leaves them no choice but to accept what's happening and buy some stack and push it down to tech leads.
I honestly think this won't change despite the fact that we already know the answer. We will need to think differently. Watch this space for what comes next.
New tools continue to radically improve workflows. Hugging face is horrible but a whole lot better than what existed before. The structure to a Pytorch-Lightning repo is a light and day improvement over one written in pure torch, and provides a lot of tooling around reproducability that heavily overlaps with MLOps. Dagster made the testing of pipelines far easier in the Data Engineering world.
Good ML packages (and their documentation and community) are the most efficient way to nudge users towards a better engineering and ML culture / set of processes. I find that the quality of the code I write and systems I build most closely correlates to the quality of the core APIs I build on top of, not deadlines or culture. Good systems evolve naturally, bottom up when they're easy and fast to implement, and better tooling buys you the time to spend more time (and budget - hire more business savvy people, less raw tech people) thinking about the business problem <-> technology solution fit. Tooling can help all the way to the business value, systems that link the quality of predictions to financial fundamentals help to evaluate if the project makes any sense in the first place.
The pure volume of overlapping tooling feels silly, but that's just free market capitalism, not the wrong focus. These arguments imply that there is some plateau in fundamental underlying technologies and their designs which couldn't be further from the truth.
I agree with the main thesis, but disagree with the dismissal of tooling.