[Interesting content] Navan uses ChatGPT to help travellers manage their journeys
A case study on how to connect technical solutions to business value
I was watching Harry Stebbings's 20VC podcast with Ariel Cohen of Navan. Navan (previously known as TripActions) is an online travel management, corporate card and expense management company for businesses. Something like Concur, if you have ever travelled on a company account, you know what I am talking about. [link: https://www.youtube.com/watch?v=x4IeCQddA1M&t=2408s I put the marker to the relevant part but the entire podcast is interesting]
So the CEO describes how their travel agent bot uses ChatGPT. You can say to the bot:
"Reserve a room in the Westin in New York."
And the bot, instead of asking "Where?" (because previously it didn't know that Westin is a hotel), asks "When?" (the only extra information it needs).
What's the problem with this?
ChatGPT is not a database. You can't just insert information into it. You can't just crawl the internet for all new hotels with locations and other metadata and update a table.
If a new hotel opens in New York tomorrow, ChatGPT won't know it. How would it know?
The only time ChatGPT's "database/memory/brain" is updated is when OpenAI updates its training set, cleans it, retrains the language model, reruns the alignment process with RLHF and SFT and deploys the new model (and advertises this fact so every new ChatGPT users can rerun their tests that the new model does the same thing as before). Classic MLOps stuff. You just outsourced it to OpenAI.
But what happens if the user asks for a new hotel?
Well, that's anybody's guess. You hope the chatbot doesn't pretend that it is reserving a hotel it doesn't know about. Or Navan's software detects that the chatbot's response is not the usual one and responds something like
"This must be a new hotel I don't know about. Let me connect you to an agent who will help you".
This would turn this system into a Human-In-The-Loop system.
The problem with HITL systems is in its name. It needs humans. Your margins will be immediately impacted. You are not infinitely scalable any more. With the growth of your business, you will need to linearly scale your employees. You can predict this by analysing the business problem, like what's the portion of new hotel openings to old ones and old ones closing, and test this in production with real KPIs (essentially, what portion of queries trigger the need for human interaction).
The feasibility of the business case is the profitability of the use case and the above back-of-the-envelope financial modelling.
And if the answer is true, you still need to orchestrate a HITL process which currently most likely requires custom SWE so better prepare your wallet.
(We are working on a platform to achieve just this, get in touch if you are interested, we are looking for angel investors)
Overall, this is an excellent example of a real-life use case of ChatGPT, its drawbacks, how to remedy them, and how to analyse them.
When you see the latest announcement of yet-another-chatgpt product, try walking through the above thought process to understand its viability.