---
sourceDocument: Yokohama Enable AI
sourceDocumentLink: https://servicenow-prod.fluidtopics.net/r/yokohama/intelligent-experiences

 Release :

    - yokohama

ft:locale :

    - en-US

ft:publication_title :

    - Yokohama Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Evaluating the prompt

# Evaluating the prompt {#ariaid-title1}

Release version: Yokohama  
Updated July 31, 2025  
![](https://www.servicenow.com/docs/portal-asset/ico-clock) 2 minutes to read
Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Evaluating the prompt

Evaluating prompts is a continuous process that occurs both during and after prompt development to measure their effectiveness.
It involves analyzing batches of test data outside of the AI Skill Kit to ensure reliable results and avoid overfitting, which happens when a prompt becomes too specialized to specific examples used in development.
Show full answer Show less  

## Prompt Development Evaluation

During development, ongoing and improvised evaluation should be performed by reviewing larger batches of model-generated outputs rather than relying on just one or two examples. This approach helps avoid reacting to random noise and supports iterative improvements based on observed performance.

## Final Performance Evaluation

Before deploying a skill, it is critical to test the prompt on a representative and isolated batch of "test" data that was not involved in development. This prevents overestimating performance due to prompt overfitting and ensures a more accurate assessment of how the prompt will perform in real use cases.

## Evaluation Metrics

Selecting appropriate evaluation metrics depends on the use case and output type:

* **Classification-based assessment:** Suitable for short, well-defined outputs with labeled records (e.g., true/false, multiple-choice). Metrics like precision, recall, and F1 scores can be directly applied.
* **Assessment of longer generations:** For generative AI outputs with multiple acceptable answers, human evaluators score the outputs on several dimensions:
  * **Faithfulness:** Ensures the generated text accurately reflects the prompt context, avoiding hallucinations.
  * **Correctness:** Checks if the output aligns correctly with the skill instructions.
  * **Helpfulness:** Measures how well the generated text supports the intended task, based on user needs.
  * **Fluency:** Assesses grammatical quality, coherence, and typographical accuracy.

Using a scaled scoring system (e.g., 1--5) for these qualitative dimensions is recommended over a simple yes/no evaluation to capture nuances in output quality.  
Evaluating the prompt is an ongoing process that occurs during and after prompt development and completion.

## Prompt evaluation overview {#evaluating-the-prompt__section_dss_msh_ccc}

To determine the effectiveness of your prompt, you should evaluate batches of test data. You should copy the model-generated responses and perform evaluations outside of AI Skill Kit.

## During prompt development {#evaluating-the-prompt__section_twb_psh_ccc}

Ongoing, improvised evaluation should take place alongside the development of the prompt. This ongoing evaluation enables you to adapt the prompt based on observed model outputs. It may be tempting to test a change to a prompt
against just one or two examples, however, to avoid reacting to noise, you should look at larger batches, and consider the statistical significance of the performance differences that you observed.

## Final performance evaluation {#evaluating-the-prompt__section_y5h_zsj_ccc}

Before you deploy a skill, you should test the prompt on a representative batch of data that was isolated from the development process, that is, "test" data. You want to use isolated test data because of a phenomenon known as
prompt overfitting. Iteratively editing a prompt based on the model outputs generated on the same data that is used for testing can lead to significant over-estimates of performance. This result is because the prompt can become
overspecialized to the specific examples used in development. Even though the effect is typically less dramatic than what occurs when fitting machine learn model parameters to a test dataset, it's rooted in the same underlying
principles, and should be avoided.

## Evaluation metrics {#evaluating-the-prompt__section_khf_1tj_ccc}

Selecting the right metrics for evaluation is an important consideration. The following list provides a few approaches, each of which may be more or less appropriate depending on the use case.  
* Classification-based assessment of short generationsThis approach requires labeled records, and it works best when the labels are short, well-defined "right answers," for example, true or false, multiple-choice, or category
  selection. In these cases, the model outputs can usually be parsed and formatted, then metrics like precision, recall, F1 scores, and so on can be directly calculated.

* Assessment of longer generationsMany of the most interesting generative AI use cases require longer model generations, and there are many possible "right answers." In these cases, the output can be scored (by human
  evaluators) along several different axes, for example:

  * FaithfulnessIs the generated text faithful to the context provided in the skill prompt? (The opposite of faithfulness is hallucination, which is to say that the model injects out-of-context information.)

  * CorrectnessIs the generated text correct relative to the skill instruction?

  * HelpfulnessIs the generated text helpful relative to the task that the skill wants to accomplish? (Helpfulness is subjective but it's important to try to measure. Doing so properly requires a solid understanding of
    the needs of the people who will ultimately be using the skill.)

  * FluencyIs the generated text grammatically correct? Does it have any typos, issues with coherency, and so on?

  {#evaluating-the-prompt__ul_yx1_c5j_ccc}  
  Note:  
  It's useful to score these properties on a scale, like 1-5, rather than with yes or no.
{#evaluating-the-prompt__ul_dsl_xtj_ccc}