ChatGPT: Insightful, Articulate, Inconsistent, and Wrong. A Game Changer?

0
1140

Share on LinkedIn

Source: iStock

Digital assistants are taking a larger role in digital transformations. They can improve customer service, providing more convenient and efficient ways for customers to interact with the organization. They can also free up human customer service agents by providing quick and accurate responses to customer inquiries and automating routine tasks, which reduces call center volume. They are available 24/7 and can personalize recommendations and content by taking into consideration role, preferences, interests, and behaviors.

All of these contribute to improved productivity and efficiency. Right now, bots are only valuable in very narrow use cases and are unable to handle complex tasks. However, the field is rapidly changing and advances in algorithms are having a very significant impact.

For example, there has been a great deal of hype and attention around ChatGPT, the chatbot based on the GPT3 language model from OpenAI. GPT stands for Generative Pretrained Transformer and is an AI algorithm that can create new content based on ingesting large amounts of text and data and deriving language rules and relationships. The generation of text in response to an input can be incredibly nuanced and creative, giving the impression that one is conversing with a human.

Unlike a search engine, it is not simply retrieving information but is generating information-based rules and relationships derived through enormous amounts of data that the algorithm has processed.

But wait, let me have ChatGPT explain

Seth: What is Generative AI?

ChatGPT: Generative AI refers to a type of machine learning model that is capable of generating new, original content based on a given set of input data. This can be in the form of text, images, or other types of data. Generative AI models are trained on a large dataset of examples, and they use this training data to learn the patterns and relationships that are characteristic of the data. They can then use this learned knowledge to generate new content that is similar to the training data, but is not simply a copy of the training data.

It’s pretty amazing. However, it frequently provides incorrect responses that are plausible. It also sometimes cannot find information that is easily retrievable through a web search and may generate different answers to the same question. For a creative task such as copywriting, that is a good thing. However, in an enterprise context, consistency in retrieving information is critical. The point is that as useful as it may be, generative AI will not be the complete answer to powering enterprise digital assistants and customer service bots

To illustrate the inconsistency and understand how the system uses context, I asked ChatGPT about a person however I anonymized the name. (If you searched for this person, you would get a lot of information).

Seth: Who is Joe Smith?

ChatGPT: I’m sorry, but I don’t have any information about a person named Joe Smith. Can you provide any additional context or details about the person you are looking for?

And his company:

Seth: Who is Smithco ?

ChatGPT: I’m sorry, but I do not have information on a company or individual named “Smithco .” Could you provide more context or clarify your question?

However, when I provided information about the company and asked for an elevator pitch, something interesting happened. The system first returned a good pitch:

Seth: Smithco is a firm that specializes in product and customer data. The website is www.smithco.com. Can you write an elevator pitch for the company’s services?

ChatGPT: Smithco is a leading provider of product and customer data services. With a team of experienced professionals and a comprehensive suite of solutions, Smithco helps organizations optimize their product and customer experiences by improving the quality, consistency, and accessibility of their data. Whether you need to streamline your product content creation process, enhance the accuracy of your customer data, or build a scalable knowledge management system, Smithco has the expertise and tools to help you achieve your goals. With a focus on innovation and collaboration, Smithco is committed to delivering value and driving business success for its clients.

A well-crafted response. When I then asked who Smithco was, the response included Smith’s role in the company. (Which had not been provided)

Seth: Who is Smithco ?

ChatGPT: Smithco is a consulting firm that provides information management and technology services to organizations. The company was founded by Joe Smith, who is a thought leader and expert in the field of information management. Smithco helps organizations to optimize their information management, etc.

The system appeared to have retained context from the prior query, but it produced much more detail than what I had given it.

When I repeated the query about the name, this time the system appears to have used information in prior chats to inform its response.

Seth: Who is Joe Smith?

ChatGPT: Joe Smith is the founder and CEO of Smithco, a company that specializes in… etc.

Interesting indeed.

In a new chat session, asking the same question produced another version that had significant inaccuracies. (Italics added for emphasis)

Seth: Who is Joe Smith?

ChatGPT: Joe Smith is a recognized expert in the field of information management. He is the founder and CEO of Smithco, a consulting firm that specializes in helping organizations create, manage, and leverage their information assets.
 
Joe Smith is a well-known speaker and author in the field and has written several books on the subject.

ChatGPT went on to list several books that Joe had not written or contributed to.

During this session, I did not provide any information about Joe or his firm. This time the system was able to recognize Joe without more context.

But not so fast–in another chat session, the same query once again stumped the system.

Logging out and back in produced this result:

Seth: Who is Joe Smith?

ChatGPT: I’m sorry, but I don’t have information on a person by that name. It’s possible that you have the wrong name, or that the person you are asking about is not a well-known figure. Without more context, it is difficult for me to provide any additional information.

But now asking about the company produced this response:

Seth: Who is Smithco ?

ChatGPT: Smithco is a company that provides consulting and technology services in the field of information management, etc. … but then it added: It is not clear from the information available online who the founder or owners of Smithco are.

I had not asked about the founder, but OK. But then when I asked about Joe Smith again, this time it knew he was the founder.

Seth: Who is Joe Smith?

ChatGPT: Based on my research, Joe Smith is the founder and CEO of Smithco, a company that provides consulting and technology services in the field of information management etc.
 
…He has written several books on these subjects, including “The ABCs of KM” and “The Digital Workplace.”…

These details are not true. Joe did not write “The ABCs of KM” and “The Digital Workplace.” He had written one book and coauthored another but not those titles.

When I logged out and back in, the behavior changed again.

Seth: Who is Smithco?

ChatGPT: I’m sorry, but I am not able to find any information about a company or person named “Smithco.” …. Etc.

This was the same response that I had gotten in the prior session before I had provided some company information. When I asked about Joe Smith again, the system recognized me and provided a bio. A further inquiry about Smithco produced a different description and a statement that it was not clear if the company was still in operation (they are).

One last experiment produced the oddest set of incorrect details, listing books Joe had not written and a teaching position he did not hold.

Some people believe that the power of ChatGPT is going to eliminate the need for knowledge management approaches and principles. When I put that question to the system, it responded with a logical, meaningful answer:

ChatGPT: Knowledge management, knowledge architecture, and knowledge engineering play important roles in the design of chatbots.
 
Knowledge management involves the processes and technologies used to capture, store, and share knowledge within an organization. In the context of chatbot design, knowledge management can help to ensure that the chatbot has access to a wide range of relevant information and can provide accurate and useful responses to user queries.
 
Knowledge architecture refers to the structure and organization of knowledge within an organization. In the context of chatbot design, knowledge architecture can help to ensure that the chatbot’s knowledge base is well organized and easy to navigate, which can improve the user experience and the effectiveness of the chatbot.
 
Knowledge engineering involves the process of designing and building knowledge-based systems, such as chatbots. It involves identifying the knowledge and expertise needed for the chatbot to function effectively and designing the chatbot’s knowledge base and information architecture to support these needs. 
Overall, knowledge management, knowledge architecture, and knowledge engineering are important considerations in the design of chatbots because they help to ensure that the chatbot has access to the knowledge and expertise it needs to provide accurate and useful responses to user queries.

To summarize, sometimes the answers from ChatGPT were insightful and articulate, while other times they were inconsistent or outright wrong.

How can ChatGPT improve knowledge management?

In the rest of this article, I will delve into more detail about the value of generalized language models, whether ChatGPT can replace knowledge management, and how ChatGPT relates to knowledge management tools such as ontologies and knowledge graphs.

Seth: When building a chatbot, will a generalized language model work for highly specialized technical content and IP or will a custom knowledge model be needed?

ChatGPT: In general, a generalized language model may not be sufficient for building a chatbot that handles highly specialized technical content and intellectual property (IP). This is because specialized technical content and IP often require a deep understanding of specific domain-specific concepts and terminology, which may not be adequately captured by a generalized language model.
 
In such cases, it may be necessary to use a custom knowledge model that has been specifically tailored to the needs of the chatbot. A custom knowledge model can be created by training a machine learning model on a dataset of domain-specific content and examples of how the chatbot should respond to different queries. This can help to ensure that the chatbot has a deep understanding of the specialized technical content and IP it needs to handle and can provide accurate and useful responses to user queries.
 
Overall, the choice between a generalized language model and a custom knowledge model will depend on the specific needs and requirements of the chatbot, as well as the resources available for building and maintaining the model. In general, a custom knowledge model may be more suitable for handling highly specialized technical content and IP, while a generalized language model may be sufficient for more general-purpose chatbots.

I could not agree more.

Seth: Will ChatGPT solve enterprise knowledge challenges?

ChatGPT: ChatGPT is a variant of the GPT (Generative Pre-trained Transformer) language model developed by OpenAI. It is a machine-learning model that has been trained on a large dataset of human-generated text and can generate human-like text in response to prompts. ChatGPT is not specifically designed to address enterprise knowledge challenges.
 
That being said, ChatGPT or other language models like it could potentially be used as part of a larger solution for addressing knowledge management challenges within an enterprise. For example, a chatbot powered by ChatGPT or a similar language model could be used to provide quick and convenient access to information and knowledge within an organization. However, it would likely be necessary to integrate additional tools and technologies, such as a structured knowledge base or information retrieval system, to fully address enterprise knowledge challenges.
 
It is important to note that ChatGPT is a machine learning model, and it is not capable of independently managing or organizing knowledge within an organization. It can only generate text based on the patterns and relationships it has learned from the data it has been trained on.

Well said. Generalized language models will handle broad and common tasks, but specialized models (taxonomy, ontology, information architecture, knowledge base design) will still be needed for deeper corporate knowledge and information.

Seth: Will ChatGPT replace knowledge management?

ChatGPT: It is unlikely that ChatGPT, or any other single tool or technology, will completely replace knowledge management as a discipline. Knowledge management is a broad field that involves a range of activities and practices designed to identify, create, represent, distribute, and enable the adoption and reuse of knowledge within an organization. These activities can include things like creating and maintaining databases, developing training materials and programs, implementing collaborative tools and platforms, and fostering a culture of knowledge sharing within the organization.
 
While ChatGPT is a powerful language generation model that can generate human-like text, it is not designed to perform all of the functions of a knowledge management system. Instead, it could potentially be used as one tool among many in a knowledge management strategy, to help generate content or assist with tasks like summarization or document creation. However, it is important to keep in mind that relying on any single tool or technology to manage knowledge can be risky, and it is important to have a diverse and well-rounded knowledge management strategy that takes into account the needs and capabilities of the organization.

It will be a tool in the toolkit, not something that will replace the KM function

Seth: Why are you unable to retrieve information that is easy to find through internet search?

ChatGPT: As an AI language model, I am not able to browse the internet or access new information beyond what I was trained on. My knowledge is based on the data that was used to train me, and my responses are generated based on that knowledge. I do not have the ability to search the web or access new information as it becomes available.

Seth: What is the role of ontology in ChatGPT?

ChatGPT: ChatGPT is a language generation model that uses machine learning techniques to generate human-like text based on patterns and relationships in the data it was trained on. It does not have a deep understanding of the underlying concepts and relationships in the information it generates and does not attempt to represent or classify information in the same way that an ontology would.
 
Instead, ChatGPT is primarily focused on generating text that is coherent and natural-sounding, based on the input provided to it. While it may be able to generate text that includes references to concepts and relationships between entities, it does not have an understanding of the meaning or significance of these concepts and relationships.
 
In summary, while ontology may be a useful tool for organizing and representing knowledge in some contexts, it is not directly related to the operation of ChatGPT or other language generation models.

This statement is interesting since many knowledge models do include ontologies. This is a case where the answer sounds plausible but may not be correct.

Seth: What is the role of a knowledge graph in ChatGPT?

ChatGPT: A knowledge graph is a representation of real-world concepts and their relationships, typically in the form of a graph data structure. It is designed to provide a structured and organized way of representing and querying information about the world and is often used in artificial intelligence and natural language processing applications.
 
In the context of ChatGPT, a knowledge graph may not play a direct role. ChatGPT is a language generation model that is designed to generate human-like text based on patterns and relationships in the data it was trained on. It does not have a deep understanding of the underlying concepts and relationships in the information it generates and does not use a knowledge graph to represent or classify this information.
 

In summary, while a knowledge graph may be a useful tool for organizing and representing knowledge in some contexts, it is not directly related to the operation of ChatGPT or other language generation models.

Takeaways

The big takeaways are that tools such as ChatGPT are incredibly powerful. They provide a glimpse of what the future will bring in terms of conversational interaction with everything from our homes full of smart devices, to our cars to corporate information systems. We will be living and working in an ecosystem supported by conversational access to everything.

In my book, The AI Powered Enterprise, I discuss a scenario where a character in the book works with conversational agents from the time he gets up (reminding him of schedules, meetings, commitments, travel) throughout his work day (finding parts for equipment, negotiating terms with a supplier) throughout his evening (handling household tasks like paying bills, balancing his investments, buying tickets to a show) until he goes to sleep (setting lights, room temperature entertainment) and more. Organizations need to prepare for this future by building the knowledge and specific functionality needed to best serve their customers and compete in the marketplace.

A generalized language model and tools like ChatGPT will level the playing field and not using them will be a competitive disadvantage. But standardization just provides efficiency. If everyone has something there is little competitive value. Differentiation is what leads to competitive advantage, and specialized functionality tailored to the needs of the organization (language models, taxonomies, ontologies, knowledge bases) will be the lever that distinguishes one organization from another by helping them serve their customers most effectively.

It will not be long before bots and virtual assistants become the primary mechanism for accessing information and serving customers. The foundation is built on knowledge and the organization that best competes with its knowledge of the market, customers, competitors, and solutions will win in the marketplace.

Keep in mind that tools like ChatGPT are impressive and powerful, but they will not solve your information challenges today. It is still important to manage, curate, and govern data and content. General language models like GPT3 will increasingly be under the covers of commercial software offerings. But customers will still need specific answers to their questions in the context of their circumstances. Understanding customer needs in greater detail will enable deeper connections. Deeper connections require human understanding and interpretation. Automation provides efficiency, humans provide connection.

ADD YOUR COMMENT

Please use comments to add value to the discussion. Maximum one link to an educational blog post or article. We will NOT PUBLISH brief comments like "good post," comments that mainly promote links, or comments with links to companies, products, or services.

Please enter your comment!
Please enter your name here