Skip to content

4.5. (Optional) Homework

KEEP LEARNING AT HOME: We covered a lot in this section!! But there's a lot more left to learn. Here are two areas for you to explore on your own, when you revisit this workshop with your personal copy of the repo.

1. Explore: Observability

  • Revisit the contoso_chat/chat_request.py and evaluators/coherence.py files
    • Observe: the PromptyTracer and @trace decoration features
  • Look for the src/api/.runs folder and click on a .tracy file
    • Observe: the traces to understand the telemetry captured for debugging
  • What happens when we remove a @trace annotation from a method?
  • What happens when we remove: Tracer.add("PromptyTracer", json_tracer.tracer)

2. Explore: Custom Evaluators

  • Copy the Coherence.prompty to a new Politeness.prompty file
  • Modify the system segment to define a "Politeness" metric
  • Modify the user segment to define your scoring guidance
  • Define a sample input & refine Prompty to return valid score
  • Create the test dataset, then assess results against your evaluator.
  • Think about how this approach extends to safety evaluations.

CONGRATULATIONS. You completed the Evaluate stage of your workflow!

In this section, you saw how Prompty-based custom evaluators work with AI-Assisted evaluation, to assess the quality of your application using defined metrics like coherence, fluency, relevance, and groundedness. You got a sense for how these custom evaluators are crafted.