“`html
The results presented in the Table 1 seem very appealing, at least to me. The simple evolution performs very well. In the case of the reasoning evolution the first part of question is answered perfectly, but the second part is left unanswered. Inspecting the Wikipedia page [3] it is evident that there is no answer to the second part of the question in the actual document, so it can also be interpreted as the restraint from hallucinations, a good thing in itself. The multi-context question-answer pair seems very good. The conditional evolution type is acceptable if we look at the question-answer pair. One way of looking at these results is that there is always space for better prompt engineering that are behind evolutions. Another way is to use better LLMs, especially for the critic role as is the default in the ragas library.
Metrics
The ragas library is able to not only generate the synthetic evaluation sets, but also provides us with built-in metrics for component-wise evaluation as well as end-to-end evaluation of RAGs.
Picture 2: RAG Evaluation Metrics in RAGAS. Image created by the author in draw.io.
As of this writing RAGAS provides out-of-the-box eight metrics for RAG evaluation, see Picture 2, and likely new ones will be added in the future. In general you are about to choose the metrics most suitable for your use case. However, I recommend to select the one most important metric, i.e.:
- Answer Correctness — the end-to-end metric with scores between 0 and 1, the higher the better, measuring the accuracy of the generated answer as compared to the ground truth.
Focusing on the one end-to-end metric helps to start the optimisation of your RAG system as fast as possible. Once you achieve some improvements in quality you can look at component-wise metrics, focusing on the most important one for each RAG component:
- Faithfulness — the generation metric with scores between 0 and 1, the higher the better, measuring the factual consistency of the generated answer relative to the provided context. It is about grounding the generated answer as much as possible in the provided context, and by doing so prevent hallucinations.
- Context Relevance — the retrieval metric with scores between 0 and 1, the higher the better, measuring the relevancy of retrieved context relative to the question.
RAG Factory
OK, so we have a RAG ready for optimisation… not so fast, this is not enough. To optimise RAG we need the factory function to generate RAG chains with given set of RAG hyperparameters. Here we define this factory function in 2 steps:
- Step 1: A function to store documents in the vector database.
<# Defining a function to get document collection from vector db with given hyperparemeters
# The function embeds the documents only if collection is missing
# This development version as for production one would rather implement document level check
def get_vectordb_collection(chroma_client,documents,embedding_model=\"text-embedding-ada-002\",chunk_size=None, overlap_size=0) -> ChromaCollection:
if chunk_size is None:
collection_name = \"full_text\"
docs_pp = documents
else:
collection_name = f\"{embedding_model}_chunk{chunk_size}_overlap{overlap_size}\"
text_splitter = CharacterTextSplitter(separator=\".\",chunk_size=chunk_size,chunk_overlap=overlap_size,length_function=len,is_separator_regex=False,)
docs_pp = text_splitter.transform_documents(documents)
embedding = OpenAIEmbeddings(model=embedding_model)
langchain_chroma = Chroma(client=chroma_client,collection_name=collection_name,embedding_function=embedding,)
existing_collections = [collection.name for collection in chroma_client.list_collections()]
if chroma_client.get_collection(collection_name).count() == 0:
langchain_chroma.from_documents(collection_name=collection_name,documents=docs_pp,embedding=embedding)
return langchain_chroma
```- Step 2: A function to generate RAG in LangChain with document collection, or the proper RAG factory function.
<# Defininig a function to get a simple RAG as Langchain chain with given hyperparemeters
# RAG returns also the context documents retrieved for evaluation purposes in RAGAs
def get_chain(chroma_client,documents,embedding_model=\"text-embedding-ada-002\",llm_model=\"gpt-3.5-turbo\",chunk_size=None,overlap_size=0,top_k=4,lambda_mult=0.25) -> RunnableSequence:
vectordb_collection = get_vectordb_collection(chroma_client=chroma_client,documents=documents,embedding_model=embedding_model,chunk_size=chunk_size,overlap_size=overlap_size)
retriever = vectordb_collection.as_retriever(top_k=top_k, lambda_mult=lambda_mult)
template = \"\"\"Answer the question based only on the following context.
If the context doesn't contain entities present in the question say you don't know.{context}Question: {question}\"\"\"
prompt = ChatPromptTemplate.from_template(template)
llm = ChatOpenAI(model=llm_model)
def format_docs(docs):
return \"\\n\\n\".join([doc.page_content for doc in docs])
chain_from_docs = (RunnablePassthrough.assign(context=(lambda x: format_docs(x[\"context\"])))| prompt| llm| StrOutputParser())
chain_with_context_and_ground_truth = RunnableParallel(context=itemgetter(\"question\") | retriever,question=itemgetter(\"question\"),ground_truth=itemgetter(\"ground_truth\"),).assign(answer=chain_from_docs)
return chain_with_context_and_ground_truth
```The former function get_vectordb_collection is incorporated into the latter function get_chain, which generates our RAG chain for given set of parameters, i.e: embedding_model, llm_model, chunk_size, overlap_size, top_k, lambda_mult. With our factory function we are just scratching the surface of possibilities what hyperparmeters of our RAG system we optimise. Note also that RAG chain will require 2 arguments: question and ground_truth, where the latter is just passed through the RAG chain as it is required for evaluation using RAGAs.
“`
Source link
