Skip to main content
Retrieval Augmented Generation (RAG) is a common way of building Generative AI applications that have access to custom knowledge bases. Evals hero

​
What you’ll learn:

This guide shows you how to:
  • Build a knowledge base
  • Create a RAG application with a retrieval step that finds relevant documents
  • Track retrieval steps with Weave
  • Evaluate RAG applications using an LLM judge to measure context precision
  • Define custom scoring functions

​
Prerequisites

  • A W&B account
  • Python 3.8+ or Node.js 18+
  • Required packages installed:
    • Python: pip install weave openai
    • TypeScript: npm install weave openai
  • An OpenAI API key set as an environment variable

​
Build a knowledge base

First, compute the embeddings for the articles. You would typically do this once with your articles and put the embeddings and metadata in a database, but here it’s done every time the script runs for simplicity.

​
Create a RAG app

Next, wrap the retrieval function get_most_relevant_document with a weave.op() decorator and create a Model class. Call weave.init('<team-name>/rag-quickstart') to begin tracking all the inputs and outputs of your functions for later inspection. If you do not specify a team name, the output is recorded to your W&B default team or entity.

​
Evaluating with an LLM Judge

When there aren’t simple ways to evaluate your application, one approach is to use an LLM to evaluate aspects of it. Here is an example of using an LLM judge to try to measure the context precision by prompting it to verify if the context was useful in arriving at the given answer. This prompt was augmented from the popular RAGAS framework.

​
Defining a scoring function

As in the Build an Evaluation pipeline tutorial, define a set of example rows to test your app against and a scoring function. The scoring function takes one row and evaluates it. The input arguments should match with the corresponding keys in your row, so question here is taken from the row dictionary. output is the output of the model. The input to the model is taken from the example based on its input argument, so question here too. This example uses async functions so they run fast in parallel. If you need a quick introduction to async, you can find one here.

​
Optional: Defining a Scorer class

In some applications you may want to create custom evaluation classes - where for example a standardized LLMJudge class should be created with specific parameters (e.g. chat model, prompt), specific scoring of each row, and specific calculation of an aggregate score. Weave defines a list of ready-to-use Scorer classes and also makes it easy to create a custom Scorer - the following example shows how to create a custom class CorrectnessLLMJudge(Scorer). On a high-level the steps to create custom Scorer are quite simple:
  1. Define a custom class that inherits from weave.flow.scorer.Scorer
  2. Overwrite the score function and add a @weave.op() if you want to track each call of the function
    • this function has to define an output argument where the prediction of the model will be passed to. Define it as type Optional[dict] in case the model might return “None”.
    • the rest of the arguments can either be a general Any or dict or can select specific columns from the dataset that is used to evaluate the model using the weave.Evaluate class - they have to have the exact same names as the column names or keys of a single row after being passed to preprocess_model_input if that is used.
  3. Optional: Overwrite the summarize function to customize the calculation of the aggregate score. By default Weave uses the weave.flow.scorer.auto_summarize function if you don’t define a custom function.
    • this function has to have a @weave.op() decorator.
To use this as a scorer, you would initialize it and pass it to scorers argument in your `Evaluation like this:

​
Pulling it all together

To get the same result for your RAG apps:
  • Wrap LLM calls & retrieval step functions with weave.op()
  • (optional) Create a Model subclass with predict function and app details
  • Collect examples to evaluate
  • Create scoring functions that score one example
  • Use Evaluation class to run evaluations on your examples
NOTE: Sometimes the async execution of Evaluations will trigger a rate limit on the models of OpenAI, Anthropic, etc. To prevent that you can set an environment variable to limit the amount of parallel workers e.g. WEAVE_PARALLELISM=3. Here is the code in its entirety.

​
Conclusion

This tutorial showed how to build observability into different steps of your applications, like the retrieval step in this example. You also learned how to build more complex scoring functions, like an LLM judge, for automatic evaluation of application responses.

​
Next Steps

Check out the RAG++ course for a more advanced dive into practical RAG techniques for engineers, where you’ll learn production-ready solutions from Weights & Biases, Cohere and Weaviate to optimize performance, cut costs, and enhance the accuracy and relevance of your applications.