QA RAG with Self Evaluation II
For this variation, a change is made to the evaluation procedure. Along with the question-answer pair, the retrieved context is also passed to the evaluator LLM.
To achieve this, an additional itemgetter function is added in the second RunnableParallel to collect the context string and pass it to the new qa_eval_prompt_with_context prompt template.
rag_chain = ( RunnableParallel(context = retriever | format_docs, question = RunnablePassthrough() ) |RunnableParallel(answer= qa_prompt | llm | retrieve_answer, question = itemgetter(“question”), context = itemgetter(“context”) ) |qa_eval_prompt_with_context | llm_selfeval |json_parser)
Implementation Flowchart :
One of the common pain points with using a chain implementation like LCEL is the difficulty in accessing the intermediate variables, which is important for debugging pipelines. We look at few options where we can still access any intermediate variables we are interested using manipulations of the LCEL
Using RunnableParallel to carry forward intermediate outputs
As we saw earlier, RunnableParallel allows us to carry multiple arguments forward to the next step in the chain. So we use this ability of RunnableParallel to carry forward the required intermediate values all the way till the end.
In the below example, the original self eval RAG chain is modified to output the retrieved context text along with the final self evaluation output. The primary change is that a RunnableParallel object is added to every step of the process to carry forward the context variable.
Additionally, the itemgetter function is used to clearly specify the inputs for the subsequent steps. For example, for the last two RunnableParallel objects, itemgetter(‘input’) is used to ensure that only the input argument from the previous step is passed on to the LLM/ Json parser objects.
rag_chain = ( RunnableParallel(context = retriever | format_docs, question = RunnablePassthrough() ) |RunnableParallel(answer= qa_prompt | llm | retrieve_answer, question = itemgetter(“question”), context = itemgetter(“context”) ) |RunnableParallel(input = qa_eval_prompt, context = itemgetter(“context”)) |RunnableParallel(input = itemgetter(“input”) | llm_selfeval , context = itemgetter(“context”) ) | RunnableParallel(input = itemgetter(“input”) | json_parser, context = itemgetter(“context”) ))
The output from this chain looks like the following :
A more concise variation:
rag_chain = ( RunnableParallel(context = retriever | format_docs, question = RunnablePassthrough() ) |RunnableParallel(answer= qa_prompt | llm | retrieve_answer, question = itemgetter(“question”), context = itemgetter(“context”) ) |RunnableParallel(input = qa_eval_prompt | llm_selfeval | json_parser, context = itemgetter(“context”)))
Using Global variables to save intermediate steps
This method essentially uses the principle of a logger. A new function is introduced that saves its input to a global variable, thus allowing access to the intermediate variable through the global variable
global context
def save_context(x):global contextcontext = xreturn x
rag_chain = ( RunnableParallel(context = retriever | format_docs | save_context, question = RunnablePassthrough() ) |RunnableParallel(answer= qa_prompt | llm | retrieve_answer, question = itemgetter(“question”) ) |qa_eval_prompt | llm_selfeval |json_parser)
Here a global variable called context and a function called save_context are defined. The save_context function saves its input value to the global context variable before returning the same input. In the chain, the save_context function is added as the last step of the context retrieval step.
This option allows access to any intermediate steps without making major changes to the chain.
Using callbacks
Attaching callbacks to the chain is another common method used for logging intermediate variable values. More details on callbacks in LangChain will be covered in a separate post.
