Here at CDP, we’ve delivered a range of Generative AI (GenAI) projects that use Large Language Models (LLMs). Each has been a journey of discovery, and sometimes frustration. But ultimately each has reinforced the potential for GenAI to dramatically accelerate innovation.

In an attempt to provide a useful contribution that cuts through the noise, we’ve distilled our learnings into a four-part series on how businesses, data scientists and product owners can leverage GenAI for success with a final perspective from a GenAI-powered ChatBot.

In this third article, we draw from our experiences implementing GenAI from a data scientist’s perspective. To get a high-level view of LLMs check out Part 1. Or for insight from the inside out, check out Part 4 of our series: Five practical insights to accelerating innovation with GenAI.

1. Accuracy

You can rely on LLMs for human like behaviours and creativity: However, you need to apply one or more of the following techniques to ensure a robust accuracy in your work.

Quality Data Sets: The foundation of accuracy lies in the quality of the data sets. By combining an off-the-shelf LLM with carefully curated, high-quality proprietary data, you create a robust foundation for generating precise and reliable content.

Verification Measures with RAG Integration: Implementing a search engine integrated with a Retrieval Augmented Generation (RAG) system and a dedicated fact-checking service adds layers of verification. This ensures that the output aligns with actual source material, fortifying the trustworthiness of the generated content.

Fine-tuning for Precision: Fine-tuning a pre-trained LLM on specific tasks and datasets can yield highly accurate results tailored to particular domains. This approach allows for a more controlled output and can be especially effective in specialized fields where precision is paramount.

Prompt Engineering for Flexibility: On the other hand, employing prompting techniques provides a more flexible way to guide the LLM’s output. By carefully crafting prompts or queries, you can influence the type and style of the generated content, allowing for adaptability across a range of contexts and requirements.

Zero-shot prompting is a way to guide the LLMs to provide the output in a particular manner. One approach is to prompt it to decompose the output into logical steps, which encourages the model to apply that logic in coming to its final output.

Peer-review: LLMs are very good at checking another’s output for accuracy. By using two independent, but similarly trained, LLMs to collaborate it is possible to filter out errors that may emerge from using just the one on its own.

Lastly, keep in mind that the output from LLMs always has the potential to mislead. Design in appropriate guard-rails from the start; but also set expectations for risk with the product owner and business throughout the project.

2. Resources

LLMs require substantial resources to operate as well as to train. Plan this in from the start and recognise the dependencies you may be creating for the business in terms of performance and budgets.

Optimize Compute Power: Recognize that LLMs require substantial compute power for optimal performance. Restricting resources may lead to a reduction in the quality of generated content. Therefore, investing in sufficient computational capabilities is crucial to maintain high standards of output.

Quantization: Convert the floating-point weights and activations of LLMs to lower precision integer or fixed-point values to allow more efficient execution on hardware with limited resources. While this can result in some loss of accuracy a quantized model can be fine-tuned or retrained to regain the lost accuracy. Quantization enables large language models to be deployed efficiently on resource-constrained edge devices by reducing memory bandwidth and compute requirements, while aiming to maintain minimal accuracy loss compared to the original model.

Augmentation Over Creation: Rather than building from scratch, consider augmenting existing models. Billions have already been invested in training and refining LLMs to get them this far. Techniques such as embeddings are well established to augment these models with additional training data, making it more practical to enhance existing resources rather than create your own from scratch. This approach allows for cost-effective improvements while leveraging the extensive groundwork laid by previous investments.

Focus on the objectives for the project, not the technologies. LLMs may not be the best solution for many of the steps in the tool chain. Other techniques may be better suited and make fewer demands on the resources you have available. Segmenting the logic flow into distinct steps will provide opportunities to reduce and simplify.

3. Modularity of Architecture

Give yourself time to read up, try out the latest advances, iterate, improve and bring into your modular architecture.

In a dynamic landscape, continual learning and research are essential for harnessing evolving technologies. Iterative development ensures adaptability and refinement, keeping your modular architecture at the forefront of progress. Striking a balance between efficiency and thoroughness is key in integrating advancements, ensuring a seamless and effective implementation. The ever-improving technological landscape necessitates staying vigilant and proactive in enhancing your system.

Langchain is a good example. It’s defining strength lies in its emphasis on modularity. It offers a versatile framework for applications driven by LLMs like GPT-3/4, Anthropic or BLOOM. Its components are abstracted for seamless interaction with LLMs and can be employed independently of the Langchain framework. This modular approach extends to advanced use cases, where components can be combined to create sophisticated functions like Generative Questioning (GQA) or summarization. With features like memory persistence and callbacks, Langchain ensures continuity and control across runs, cementing its reputation as a pioneering framework for LLM applications.

4. Independence

Architectural Autonomy: Design with a focus on architectural independence. By creating a modular framework, you establish a system that is not overly reliant on a specific language model. This ensures adaptability to evolving technologies and allows for seamless integration of future advancements.

Consider Prompt over training: As described above, a well-engineered prompt can include training material within the context window. This could allow you to operate with untrained LLMs as they are; avoiding the tie-in that training implies.

Avoid Vendor Lock-In: Strive for independence from specific vendors or providers. This entails selecting technologies and components that are compatible with a range of models and platforms. Avoiding vendor lock-in promotes flexibility and prevents potential constraints associated with proprietary solution.

5. Confidentiality & Provenance

It is important to understand the source of the information being used. Issues of confidentiality, copywrite and provenance are important considerations and bring risks that the business needs to address.

Security: When working with multiple clients, and internal or external teams, it’s crucial to maintain strict confidentiality and prevent any potential conflicts of interest. Where information is being used to fine-tune LLMs, ensure that you have appropriate separation of models to avoid cross-contamination of information.

Provenance: Establish a system for verifying the provenance of data sources. This involves validating the authenticity and reliability of information before integration into the model. By ensuring that data originates from reputable and trustworthy sources, you enhance the overall integrity and credibility of the generated content.

Source: Referencing the original sources is particularly important when applied to knowledge management and news flows where transparency of source is a key step in understanding the validity of the information being crafted.

Interested in exploring how GenAI can accelerate your innovation?

Come and join us in Cambridge, UK, and Raleigh, NC, where we’ll be running a series of in-person workshops to help clients identify the opportunities (and threats) of GenAI and plan a path to accelerate their innovation.

Here at CDP, we’ve delivered a range of Generative AI (GenAI) projects that use Large Language Models (LLMs). Each has been a journey of discovery, and sometimes frustration. But ultimately each has reinforced the potential for GenAI to dramatically accelerate innovation.

In an attempt to provide a useful contribution that cuts through the noise, we’ve distilled our learnings into a four-part series on how businesses, data scientists and product owners can leverage GenAI for success with a final perspective from a GenAI-powered ChatBot.

In this second article, we draw from our experiences implementing GenAI from a product owner’s perspective. To get a high-level view of LLMs check out Part 1 or for a deeper dive in the technology from a data scientist perspective check out Part 3.

1. Start at the end and work backwards

As with all truly transformative innovation, start by understanding what you are offering your users and work back from them. Ignore the undoubted magic of the technology at this stage – you can rely on that coming later.

You will need to set your success criteria, and this is where to start. Delighting your user base and measuring how they will benefit will do more to drive adoption than any shiny AI tech that might be going on behind the scenes.

Choose your project carefully.

• • Choose an area that you already know well or for which you have a good way of measuring success. This will ensure you see beyond the magic of the black-box and can truly judge the performance and value that LLMs bring.

• • Choose an area where LLMs work to their strengths by taking advantage of at least one of the core competencies they have been shown to do well; summary, expansion, inference and analysis.

2. Don’t forget the basics

Make good use of Service Design techniques to define what success looks like. Map the User Journey and spend time defining the touchpoints and modelling the semantic information architecture.

And then strip it back. Cut away absolutely everything that isn’t vital to the successful outcome you plan for. Don’t let the designers loose until this is done. And consider any investigative work with the technology up to this point as exploratory and should almost certainly be archived.

You’ll then have a clear set of priorities, requirements, information flows and use-cases that everyone understands, and everyone can support. The whole team will be clear about what they are aiming for. Keeping their eye focussed on the prize makes the Product Owner’s primary catch-phrases more effective: “No that is not in scope” and “This is lower priority”.

And if this is starting to sound like the start of any solid digital project – good, it should.

3. Experiment

Give your team as much time as possible to try things out. Build the time into the plan and break the experiments down into small and well-defined steps to learn and iterate.

Look to experiment with the following:

• • How the structure of prompts changes the output.

• • How the different LLMs compare when asked to respond to the same prompt.

• • How to extend the LLMs by adding training to embed your own information data.

Aim to build the experimental steps around the core competencies of Generative AI. And later, bring these together to form an overall solution using your favourite AI automation tool chain.

There will be surprises. There will be frustrations. And there will be changes in the way that you approach the use of the LLMs. Don’t be afraid to pivot on how you use the technology; or indeed ‘if’ you use the technology. But remember the basics, keep your eye on what success looks like and don’t let the team get carried away with ‘shiny object syndrome’.

4. Get lots of feedback

While using AI, remember to share your work with real humans as early as possible: People outside your team who can give you useful feedback. Set up demos within the team to share learnings and put on regular show-and-tell sessions with your target audience. And, as soon as possible, let them try it out – on their own, without you there. They will learn to see beyond the magic, and you will quickly find out what works and what doesn’t.

Your priorities will change – but the fundamental definition of success won’t (hopefully). And don’t forget the importance of plain old testing. The outputs from a LLMs can vary widely with only the smallest changes in training data and prompts. Fortunately, LLMs can come to the rescue here – they are great at evaluating the output from other models through peer- review. Use that capability to help you test. This is also useful for building into the architecture of your solution. Where you have the resources; double up the LLMs to interact and increase the quality of output for a production system.

5. Don’t underestimate the time you need

Don’t underestimate the time it will take to gather, prepare and refine your data. When it comes to data, quality and variety are just as important as quantity. With demographic information, a good distribution of variety is vital to represent your users truly and ethically. And don’t forget to set aside at least 10% randomly selected from the training set so that you can properly test the results.

To save time and increase the training and test data available, explore opportunities to synthesise data to add to your original data set. Also, don’t underestimate the time it will take to test and refine the prompts and LLMs settings to achieve the repeatable outcomes you are looking for. Prompt engineering is an art as well as a skill and takes time to learn.

Finally, know when to stop. It will always be possible to make it a bit better. Be clear about what is good enough and recognise when you get there. The impulse for the team to keep tweaking will never end – it’s simply too absorbing.

Interested in exploring how GenAI can accelerate your innovation?

Come and join us in Cambridge, UK, and Raleigh, NC, where we’ll be running a series of in-person workshops to help clients identify the opportunities (and threats) of GenAI and plan a path to accelerate their innovation.