If AI Can’t be Trusted, Efficacy and Efficiency Won’t Matter

Well, this turns out to become a column about how to leverage the value that generative AI can deliver while avoiding the perils that loom when either the vendor that we trust or we, ourselves, deviate from the right path.

And, with all the excitement that surrounds the technology, there are plenty of chances to get off the right path. This right path is where customers can and do trust companies that use powerful tools like generative AI, regardless of B2B or B2C business models.

According to the recent Salesforce State of the Connected Customer report, this is not yet the case, especially not for consumers. According to the report, 68 percent of customers say that advances in AI make it more important for companies to be trustworthy. Yet only 47 percent of consumers say that they generally trust companies, while 74 percent of customers (not broken down between business buyers and consumers) are concerned about the ethical use of AI by businesses.

These findings are alarming.

To address these findings, every company executive who plans to invest in and use generative AI needs to get answers to some serious questions.

Importantly, these questions do not only need to be answered by vendors but also within the company.

These questions are not only about technology but also cover legal and ethical aspects. In brief, the use of AI needs to be in line with corporate ESG guidelines. These guidelines, in turn, need to follow high ethical standards and need to be lived.

So, what are these questions?

Some of these questions — and the answers as they should be given — I have addressed in previous articles of this columns. They include more implementation-related questions like defining success, the identification of corresponding KPIs and a risk-aware selection of generative AI models. In addition, I suggest employing a data security architecture. After all, one of the most important questions is the one that asks for the protection of sensitive data.

Terms and conditions?

The need to answer other questions became blatantly obvious, e.g., during the recent ruckus that was caused by Zoom’s terms and conditions. This ruckus was essentially about the realization of how much rights vendors of generative AI and other services might give themselves as part of the contractual and associated agreements. Now, Zoom changed its T&C significantly after a very loud public outcry, but one underlying topic remains: Terms and conditions are regularly made part of a SaaS contract. They are regularly only referenced as a URL; they are lengthy and mostly written in a language that can only be named “legalese”. Another, earlier case, was about the app Lensa, which assumed ownership of content that was generated based on e.g., photos uploaded by its users (NB: With the T&C dated July 10, 2023, this seems to have been changed).

What this demonstrates is that a customer essentially signs a contract that contains a variable part. This variable part can be changed unilaterally and at any given time. And with minimal notice that often does not spell specifics. Add to this the biggest lie on the web: I have read, understood, and agree to the terms and services.

As a consequence of this, you are well advised to insist on contracts that are devoid of parts that can easily change.

Or, paraphrasing the words of Brian Sommer during a conversation with Jon Reed: It needs paper contracts. They don’t have hyperlinks. And they should be shorter than 100 pages.

This is quite amusing in times of digital transformation.

Accuracy: It is all about outcomes!

The importance of accuracy cannot be overstated. Hence, an important question is about the accuracy of answers that are generated by the AI. When publicly demoing its Bard AI for the very first time, Google stock suffered a hit of about $100bn in valuation on that day due to the AI giving a factually wrong answer.

Another case in point is the lawyer who used ChatGPT to prepare a court filing that listed several fake cases. Again, accuracy is non-negotiable if the AI is going to be helpful and perceived as such. According to the Salesforce report mentioned above, the trust level in AI accuracy is still very low.

Important questions to ask — and get answered — include: How relevant is the model’s training for my intended use cases? Get a credible estimate for its answer accuracy for your use cases. How can you improve the model’s relevance for your use cases and last, but not least, can the model “explain” its reasoning to its users in a way that they do not need a data scientist to understand the explanations? This is probably one of the most underdelivered features of AI models yet, although it is one of the most important ones when it comes to acceptance. I have put explainability already onto my AI trajectory in a 2017 article “Why AI, Machine Learning, and Bots? Better Experiences“. That time, I missed out on generative AI, though …

Increasing a model’s accuracy is usually achieved by so-called fine-tuning of the pre-trained delivered model with corporate data and/or by continuous (reinforcement) training that is based on ongoing operations. How this works while keeping the corporate data safe, needs to be answered.

Another possible way to achieve and maintain high accuracy is the deployment of different specialized models for different purposes.

This leads to the questions of which models the vendor uses, what their contractual relations to these models’ vendors are, and especially back to how the security architecture looks like that ensures that no sensitive corporate data is used by these models for other purposes than generating the desired output. How will model accuracy in this scenario be improved without endangering corporate data? You’ll want convincing answers to these questions.

How about the training data?

The question about accuracy has a couple of related follow-up questions. How has the model been trained, i.e., what data has been used? Or related to it: How do the vendors ensure that the trained model that they deliver, suffers minimal hallucinations?

The first of these questions needs both a technical and an ethical answer. The technical answer is usually one of “data that has been scraped from the web” or “available training data sets”, which usually are also scraped from the web. These data sets have the inherent risk of training the model with conflicting data, hence the famous hallucinations (or expensive demo failures). The ethically correct answer is related to this. In how far did the vendor make sure that the model’s training itself did not violate foreign IP, e.g., in the case of using books? This question is far from trivial. Apart from a now unclear judicial situation, it also covers ESG aspects. The knowledge that was used to generally train the AI, may very well be protected by copyright law with the holders being remunerated accordingly. “Fair use” is not always the correct answer here. Ask yourself how the use of systems that do not have a clear answer to copyright adherence fits into your corporate claims that the complete supply chain requires fair treatment and payment.

Related to this are toxicity and bias. How does the vendor’s training — and your finetuning and re-training — make sure that the former is avoided, and the latter minimized? Both topics are highly related to the training- and test data that are used.

What guardrails are in place and how are they enforced?

There are many instances of AI systems that needed to be switched off because of devolving into a toxic system, Microsoft’s Tay being only one of the more prominent examples. Apparently, it is also not too difficult to bypass the guardrails of contemporary LLMs.

Have vendors clearly explain what their models were trained with, what guardrails are in place, and how they are protected. The last thing you want is to associate your brand with a monster.

Jon Reed puts this topic even up a notch in a recent article by asking vendors how their AI systems can get used to minimize human bias. This is particularly interesting when it comes to using AI in recruitment or performance management and training.

This brings me to the last and probably most important question that needs to be addressed.

What is the impact of using AI on my teams?

This is a concern that has deep ethical implications. Artificial Intelligence is all about automation. Automation is about fulfilling tasks with fewer or even without people, which scares people. For employees, the introduction of AI into their jobs is not just business, as one mafia person tells the other just before he shoots him.

Instead, for employees, it is deeply personal. Their concern is real. AI, at best, changes their job.

The answer to this concern is as obvious as it is simple. Show loyalty to the most loyal stakeholder group that your company has: Your employees. That way, the application of AI technologies has a positive impact, first on your employees, and then on your customers who overwhelmingly think that it is important for a human to validate the output of AI.

Setting up an AI governance and having these questions answered will get you a long way in keeping the trust of your employees and your customers by staying strong in the triple-E game.

Don’t just look at efficacy and efficiency, but also at ethics.