NLU vocabulary

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of NLU vocabulary

    NLU vocabulary in ServiceNow helps the system accurately recognize the various ways users express requests by defining words, phrases, and patterns. This improves intent prediction by covering synonyms, acronyms, and specific formats that users may use in utterances.

    Show full answer Show less

    Vocabulary Usage and Types

    • Regular vocabulary items: Words or phrases, including acronyms and industry-specific terms, that are case insensitive and mapped to synonyms for consistent intent prediction.
    • Pattern vocabulary items: Use regular expressions (regex) to capture specific formats such as email addresses or phone numbers, allowing precise matching and improved accuracy.

    Vocabulary Sources

    Vocabulary sources reference entire lists or tables (e.g., conference rooms, catalog items) from ServiceNow or static lists. These sources are reusable across NLU models and enable efficient handling of multiple related values by mapping them to a single synonym. Use the @ symbol to reference vocabulary sources in training utterances and entity values.

    Practical Application in Intents

    • Adding vocabulary items for acronyms or uncommon terms (e.g., “sfcrm”) raises model confidence for intent prediction.
    • Pattern vocabulary items can disambiguate common words used in specific contexts (e.g., capitalized “Word” as software) to increase prediction accuracy.
    • Synonyms replace vocabulary items during intent recognition to standardize user input understanding.
    • Entities can be annotated to extract specific information (e.g., software names) from utterances in Virtual Agent conversations.

    Regex Usage in Pattern Vocabulary

    Regex patterns enable precise matching, such as using word boundaries (\b) to differentiate acronyms ("IT") from common words ("it"). Patterns are case sensitive by default but can be adjusted. This flexibility ensures accurate recognition of vocabulary items.

    Managing Vocabulary

    • Create Regular Vocabulary Items: Define words or phrases with synonyms to improve recognition.
    • Create Pattern Vocabulary Items: Use regex to match specific formats within utterances.
    • Create List Vocabulary Sources: Define static lists of terms to be recognized collectively.
    • Create Table Vocabulary Sources: Use values from ServiceNow tables as vocabulary sources to cover dynamic or extensive lists.
    • Sync Table Vocabulary Sources: Regularly synchronize to keep vocabulary aligned with the latest table data, ensuring up-to-date intent prediction.

    Additional Resources

    ServiceNow provides pre-built vocabulary sets for common software and hardware terms, facilitating quick deployment and recognition of typical IT terminology.

    Use NLU vocabulary items to help the system recognize the various ways your users express their requests. Use vocabulary sources to help the system recognize objects in tables or lists, such as names of conference rooms or catalog items.

    This image shows a robot serving a platter of linked molecules. The robot and everything else is saying 'NLU Vocabulary' in 6 different languages.

    Vocabulary usage and context

    Vocabulary helps your model with the various words and phrases that it may encounter from your users. Since humans are not computers, they may come up with different ways to say the same thing. For example, if one user inputs an acronym rather than the full phrase, the system might be able to predict the correct intent by using the context in the utterance. However, by defining the acronym with a vocabulary item, you raise the model's confidence level and capability to predict intents correctly.

    The Build and train your model phase with the Vocabulary tab highlighted.

    Vocabulary items cover the varying words or phrases that might appear in utterances. The vocabulary items are mapped to the synonyms that you provide, for intent prediction. This helps the system recognize the various ways your users express their requests, while at the same time ensuring the consistency, confidence, and accuracy of predictions.

    Vocabulary item types

    NLU provides the following vocabulary types that you can use to create and configure a vocabulary item.
    Type Definition
    Regular

    A word or phrase that is not commonly known, such as a business or industry specific term or acronym. Regular vocabulary is case insensitive, so all case variations will be captured by the vocabulary you create.

    See Create a regular vocabulary item.
    Pattern

    A regular expression (regex) that can capture specific formats such as email addresses and phone numbers.

    See Create a pattern vocabulary item.

    Vocabulary sources

    You can also use vocabulary sources to cover various objects that your users might request. Vocabulary items and vocabulary sources differ in their usage:
    • Use a vocabulary item for an individual word, phrase, or pattern that can easily be mapped to a single synonym for the model to use.
    • Use a vocabulary source to reference a ServiceNow table or list so that the values can all be replaced by the single synonym you define. Vocabulary sources can be reused across all your NLU models.

    Use vocabulary sources when your user utterances refer to objects in a set. The sources can be referenced from a list of values from a ServiceNow table or a static list that you define, such as a list of catalog items or conference rooms. Once you create and save these sources, you can use the @ symbol to specify them in training utterances. You can also use these sources as entity values.

    For example, say you already have a list of all the conference rooms for your office. You can create a vocabulary source to reference that list rather than typing each conference room name into your intent's training utterances.

    To create vocabulary sources, refer to Create a table vocabulary source and Create a list vocabulary source.

    Vocabulary usage in relation to an intent

    Here's an example of how an intent can interact with the vocabulary in its utterance examples.
    Note:
    In NLU vocabulary, the synonym replaces the vocabulary that appears in the utterance.
    Table 1. Intent: OrderSoftware
    Utterance Issue and Solution
    "I need to access sfcrm"
    • Issue: the system doesn't recognize the acronym sfcrm, so can't accurately predict the intent.
    • Solution: Add sfcrm as a regular vocabulary item, and provide a synonym such as CRM software.
    "I need to install Word"
    • Issue: The term word is very common and does not necessarily indicate a software product. The term install may provide helpful context, but the confidence score may be too low to predict the OrderSoftware intent.
    • Solution: Create a pattern vocabulary item with a regex for capitalized Word, so that the system can recognize it as a software product. This added constraint makes it more likely that the OrderSoftware intent will be predicted.
      Note:
      To extract the specific software name to use in a Virtual Agent conversation topic, annotate it as an entity in the utterance.

    Use a single word or short phrase as a synonym for best results. You can map multiple vocabulary items to one synonym. Do not map one vocabulary item to multiple synonyms.

    For more context and examples, see Create an NLU intent.

    A Regex example for a pattern vocabulary item

    Let's say that you want to use a vocabulary item to identify the acronym IT, and map it to the synonym information technology.

    Regular vocabulary items are case-insensitive by default. This means that a regular vocabulary item would match both the acronym IT and the common word it. So you decide to use a pattern vocabulary item.

    The regex pattern that you can use is \bIT\b. The \b is a word boundary marker. This ensures that the pattern would not match ITSM or JIT, for example. The default case sensitivity of pattern vocabulary items means that \bIT\b would not match the common word it.

    Regex details: You can turn off case sensitivity in a pattern by using (?i). You can end that mode by using a minus sign as in (?-i). For example, (?i)te(?-i)st should match both test and TEst, but not teST or TEST.

    For more information, see Using regular expressions in entities.