Getting Business Value from AI: Avoiding a (Gen) AI Winter

In some cases, these stories relate to fundamental misunderstandings of how AI/generative AI (Gen AI) works, expectations of what it is capable of, or some of the antisocial applications that it has been used for. 

However, what does appear to have happened is that as people have become more familiar with Gen AI technologies in particular, there has been a sort of standing back and evaluation of what has been going on while we were overly excited by this new technology. This is a healthy thing to do, and as with previous cycles of AI innovation, this has resulted in some sensible questions about the hard value that AI has achieved. For example, one survey found that since 2021 there has been a reduction in the number of AI projects deployed and also a reduction in return on investment (ROI). Another survey found that, along with Gen AI being the primary type of AI solution deployed in organizations, one of the top barriers to implementing AI techniques is ‘estimating and demonstrating value’. What these surveys don’t say is that (Gen) AI does not work, that it cannot do remarkable things, or has no value at all; rather they point to the need for more careful identification of the business problem that needs solving and how that results in value to the business.

This may well be playing out more publicly because Gen AI technology is much more accessible than old style AI or ML and, because of this, many more people and organizations have been empowered to try it out for themselves. How therefore can those of us embarking on (Gen) AI projects build upon and learn from the experiences, successes and failures of AI and ML projects of the past? In this and future articles, I will attempt to bring out some of the key learnings vital for successful AI projects, but first let us be specific about what we mean by ‘AI’ and ‘Gen AI’ and, at a high level, how they work.

In the case of traditional - or ‘old’ - AI approaches, such as machine learning (ML), algorithms work in a specific manner: with appropriate set up and direction, they can output an inference given a set of input data. Building a model in the first place is a task requiring skill, knowledge and creativity. However, perhaps the biggest challenge appears to be how to assemble the elements that can be solved using ML in such a way so that they can provide an overall solution to the business problem. Business problems don’t often fit an off-the-shelf ML solution. They are not set up like a Kaggle competition, whereby the lucky data scientist on the project has to optimize an algorithm given a neat csv of input data, with the output of 1s or 0s (or 0.2, 0.7, 0.9, etc.) generated by the algorithm magically solving the business problem. It is not that such inputs, outputs and intermediate models or algorithms cannot solve a business problem. Instead, it is just that the business use case needs to be correctly identified and structured in the first place, and that the resulting output needs to be wrapped in such a way that it can integrate into an overall solution, which then solves the business problem.

Generative AI is similarly constrained by ensuring that the off-the-shelf solution (foundation model, such as a large language model) can be modified in such a way that the ‘prompt in / text out’ can behave as required for the business problem. Common approaches to adaptation include prompt engineering and retrieval augmented generation (RAG), in addition to the various checks and balances that might be required to promote predictability of behavior and to decrease the chances of undesirable behavior. Therefore, as with ‘old AI’, Generative AI similarly needs to be integrated into an overall solution.

So, what is that overall solution, and how does it solve the business problem? Even before that, we need to understand what the business problem actually is and begin to get a sense of whether it can be usefully addressed using AI or whether other approaches might be more appropriate. All of this is impossible to know ahead of time, but there are things that we can do to get us to that point, with useful steps being to understand the business problem, speak to stakeholders and map the solution to the business logic. I will discuss each of these aspects, along with some final considerations for running AI projects, in more detail over the following series of articles.