Welcome to the second part of our series on using Temporal to enhance the reliability of applications built around Large Language Models (LLM) like ChatGPT. In the first part, we discussed creating a Temporal Workflow to process documents and make them available to the LLM. In this post, we will focus on developing a Temporal Workflow to identify relevant documents for a user’s query and provide them as context to a prompt sent to the LLM using Context Injection. Additionally, we will explore how Temporal’s abstraction can improve the reliability of your application and simplify the addition of new features.
Context Injection is a technique used in prompt engineering to provide specific information to the LLM for reference in its responses. In our example, we want our LLM to offer information about Hatchify, a Bitovi open source project. By including relevant documentation along with the user’s query, we can enhance the LLM’s responses.
To implement Context Injection, we start a Temporal workflow using a Temporal client. This workflow creates an S3 bucket for the conversation, generates the prompt with relevant documentation, and then invokes the GPT model with the prompt. The generatePrompt activity finds documentation related to the user’s query using similarity search of embeddings vector data and stores it in a JSON file in the S3 bucket. The invokePrompt activity retrieves the file from S3 and invokes the GPT model with the query and relevant documentation.
Using Temporal for your LLM application offers several benefits. The Durable Execution feature ensures that your workflow is resilient to failures and can continue running even if components experience issues. The ability for workflows to run indefinitely allows for handling long-running tasks efficiently. Additionally, Temporal’s independent Activities make it easy to extend the application’s functionality without creating complex code.
In summary, integrating Temporal into your LLM application for context injection and prompt engineering enhances reliability, scalability, and extensibility. With Temporal, you can build robust applications powered by LLMs while easily adapting to evolving techniques and requirements.
Source link
