> ## Documentation Index
> Fetch the complete documentation index at: https://wb-21fd5541-wbdocs-1882.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Leaderboard Quickstart

> Découvrez comment utiliser Leaderboard Quickstart avec W&B Weave

<Note>
  Il s’agit d’un notebook interactif. Vous pouvez l’exécuter en local ou utiliser les liens ci-dessous :

  * [Ouvrir dans Google Colab](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/leaderboard_quickstart.ipynb)
  * [Voir la source sur GitHub](https://github.com/wandb/docs/blob/main/weave/cookbooks/source/leaderboard_quickstart.ipynb)
</Note>

<div id="leaderboard-quickstart">
  # Leaderboard Quickstart
</div>

Dans ce notebook, nous allons apprendre à utiliser le Leaderboard de Weave pour comparer les performances des modèles sur différents jeux de données et fonctions de scoring. Plus précisément, nous allons :

1. Générer un jeu de données de faux codes postaux
2. Rédiger quelques fonctions de scoring et évaluer un modèle de référence.
3. Utiliser ces techniques pour évaluer une matrice modèles/évaluations.
4. Examiner le leaderboard dans Weave UI.

<div id="step-1-generate-a-dataset-of-fake-zip-code-data">
  ## Étape 1 : Générer un jeu de données fictif de codes postaux
</div>

Nous allons d’abord créer une fonction `generate_dataset_rows` qui génère une liste de données fictives de codes postaux.

```python lines theme={null}
import json

from openai import OpenAI
from pydantic import BaseModel

class Row(BaseModel):
    zip_code: str
    city: str
    state: str
    avg_temp_f: float
    population: int
    median_income: int
    known_for: str

class Rows(BaseModel):
    rows: list[Row]

def generate_dataset_rows(
    location: str = "United States", count: int = 5, year: int = 2022
):
    client = OpenAI()

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {
                "role": "user",
                "content": f"Please generate {count} rows of data for random zip codes in {location} for the year {year}.",
            },
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Rows.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)["rows"]
python
import weave

weave.init("leaderboard-demo")
```

<div id="step-2-author-scoring-functions">
  ## Étape 2 : Créez des fonctions de scoring
</div>

Nous allons ensuite créer 3 fonctions de scoring :

1. `check_concrete_fields` : vérifie si la sortie du modèle correspond à la ville et à l’État attendus.
2. `check_value_fields` : vérifie si la sortie du modèle se situe dans une marge de 10 % par rapport à la population et au revenu médian attendus.
3. `check_subjective_fields` : utilise un LLM pour vérifier si la sortie du modèle correspond au champ "known for" attendu.

```python lines theme={null}
@weave.op
def check_concrete_fields(city: str, state: str, output: dict):
    return {
        "city_match": city == output["city"],
        "state_match": state == output["state"],
    }

@weave.op
def check_value_fields(
    avg_temp_f: float, population: int, median_income: int, output: dict
):
    return {
        "avg_temp_f_err": abs(avg_temp_f - output["avg_temp_f"]) / avg_temp_f,
        "population_err": abs(population - output["population"]) / population,
        "median_income_err": abs(median_income - output["median_income"])
        / median_income,
    }

@weave.op
def check_subjective_fields(zip_code: str, known_for: str, output: dict):
    client = OpenAI()

    class Response(BaseModel):
        correct_known_for: bool

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {
                "role": "user",
                "content": f"My student was asked what the zip code {zip_code} is best known best for. The right answer is '{known_for}', and they said '{output['known_for']}'. Is their answer correct?",
            },
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Response.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)
```

<div id="step-3-create-a-simple-evaluation">
  ## Étape 3 : Créer une Évaluation simple
</div>

Nous définissons ensuite une Évaluation simple à l’aide de nos données factices et des scoring functions.

```python lines theme={null}
rows = generate_dataset_rows()
evaluation = weave.Evaluation(
    name="United States - 2022",
    dataset=rows,
    scorers=[
        check_concrete_fields,
        check_value_fields,
        check_subjective_fields,
    ],
)
```

<div id="step-4-evaluate-a-baseline-model">
  ## Étape 4 : Évaluer un modèle de base
</div>

Nous allons maintenant évaluer un modèle de base qui renvoie une réponse statique.

```python lines theme={null}
@weave.op
def baseline_model(zip_code: str):
    return {
        "city": "New York",
        "state": "NY",
        "avg_temp_f": 50.0,
        "population": 1000000,
        "median_income": 100000,
        "known_for": "The Big Apple",
    }

await evaluation.evaluate(baseline_model)
```

<div id="step-5-create-more-models">
  ## Étape 5 : Créer d’autres Models
</div>

Nous allons maintenant créer 2 modèles supplémentaires pour les comparer à la référence.

```python lines theme={null}
@weave.op
def gpt_4o_mini_no_context(zip_code: str):
    client = OpenAI()

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""Zip code {zip_code}"""}],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Row.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)

await evaluation.evaluate(gpt_4o_mini_no_context)
python
@weave.op
def gpt_4o_mini_with_context(zip_code: str):
    client = OpenAI()

    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": f"""Please answer the following questions about the zip code {zip_code}:
                   1. What is the city?
                   2. What is the state?
                   3. What is the average temperature in Fahrenheit?
                   4. What is the population?
                   5. What is the median income?
                   6. What is the most well known thing about this zip code?
                   """,
            }
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "response_format",
                "schema": Row.model_json_schema(),
            },
        },
    )

    return json.loads(completion.choices[0].message.content)

await evaluation.evaluate(gpt_4o_mini_with_context)
```

<div id="step-6-create-more-evaluations">
  ## Étape 6 : Créez davantage d’évaluations
</div>

Nous allons maintenant évaluer une matrice croisant des modèles et des évaluations.

```python lines theme={null}
scorers = [
    check_concrete_fields,
    check_value_fields,
    check_subjective_fields,
]
evaluations = [
    weave.Evaluation(
        name="United States - 2022",
        dataset=weave.Dataset(
            name="United States - 2022",
            rows=generate_dataset_rows("United States", 5, 2022),
        ),
        scorers=scorers,
    ),
    weave.Evaluation(
        name="California - 2022",
        dataset=weave.Dataset(
            name="California - 2022", rows=generate_dataset_rows("California", 5, 2022)
        ),
        scorers=scorers,
    ),
    weave.Evaluation(
        name="United States - 2000",
        dataset=weave.Dataset(
            name="United States - 2000",
            rows=generate_dataset_rows("United States", 5, 2000),
        ),
        scorers=scorers,
    ),
]
models = [
    baseline_model,
    gpt_4o_mini_no_context,
    gpt_4o_mini_with_context,
]

for evaluation in evaluations:
    for model in models:
        await evaluation.evaluate(
            model, __weave={"display_name": evaluation.name + ":" + model.__name__}
        )
```

<div id="step-7-review-the-leaderboard">
  ## Étape 7 : Consulter le leaderboard
</div>

Vous pouvez créer un nouveau leaderboard en accédant à l’onglet du leaderboard dans l’UI et en cliquant sur "Create Leaderboard".

Vous pouvez également générer un leaderboard directement depuis Python :

```python lines theme={null}
from weave.flow import leaderboard
from weave.trace.ref_util import get_ref

spec = leaderboard.Leaderboard(
    name="Zip Code World Knowledge",
    description="""
This leaderboard compares the performance of models in terms of world knowledge about zip codes.

### Columns

1. **State Match against `United States - 2022`**: The fraction of zip codes that the model correctly identified the state for.
2. **Avg Temp F Error against `California - 2022`**: The mean absolute error of the model's average temperature prediction.
3. **Correct Known For against `United States - 2000`**: The fraction of zip codes that the model correctly identified the most well known thing about the zip code.
""",
    columns=[
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluations[0]).uri(),
            scorer_name="check_concrete_fields",
            summary_metric_path="state_match.true_fraction",
        ),
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluations[1]).uri(),
            scorer_name="check_value_fields",
            should_minimize=True,
            summary_metric_path="avg_temp_f_err.mean",
        ),
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluations[2]).uri(),
            scorer_name="check_subjective_fields",
            summary_metric_path="correct_known_for.true_fraction",
        ),
    ],
)

ref = weave.publish(spec)
```