Amazon CloudWatch data input configuration fields

  • Release version: Xanadu
  • Updated August 1, 2024
  • 5 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Amazon CloudWatch data input configuration fields

    This document describes the configuration fields for setting up Amazon CloudWatch data inputs in ServiceNow, specifically for the Health Log Analytics application (version 26.0.17 and later). It guides customers on how to configure data inputs to stream log data from Amazon CloudWatch into their ServiceNow instance, using MID Servers or MID Server clusters with basic authentication.

    Show full answer Show less

    Basic Configuration

    • Name: Required field to name the new data input.
    • Description: Optional description of the data input.
    • Execute on: Option to select a specific MID Server or a MID Server cluster for pulling log data. Clusters support failover, switching automatically if a MID Server fails.
    • MID Server: Select a MID Server that supports basic authentication (mTLS unsupported). Maximum of 10 streaming data inputs per MID Server by default, configurable in MID Server properties. Health Log Analytics enables log ingestion automatically if disabled.
    • MID Server Cluster: Select a failover cluster with MID Servers supporting basic authentication. At least one MID Server in the cluster must have capacity (fewer than 10 data inputs) to pass validation. Log ingestion is automatically enabled if needed.
    • Service Instance: Required field to bind log data to an existing operational service instance. Customers must create and set the status of a new service instance to Operational if none exists.

    Read-only information includes: Status of the data input, transport protocol used, count of log sources created, time when input was disabled or failed, last log time, and any streaming error messages.

    Query Settings

    • From: Required start date/time to read logs. Earlier dates may cause large data reads and congestion.
    • Group Name(s): Required list of Amazon CloudWatch log groups to search, supporting comma-separated lists or wildcard () for all groups.
    • Prefix: Optional prefix for filtering log streams. Only one prefix per data input is supported; multiple prefixes require multiple data inputs.
    • Filter Pattern: Optional case-sensitive pattern to filter log events. Supports single terms, multiple terms, or inclusion/exclusion patterns to precisely select events.

    Transport Configuration

    • AWS Credentials: Reference to AWS access and secret keys stored in ServiceNow.
    • AWS Region: Required field specifying the AWS region of the CloudWatch cluster (e.g., us-west-1).

    Advanced Configuration

    • Connection Timeout: Milliseconds to wait before AWS connection times out (default 1000ms).
    • Batch Size: Maximum logs retrieved per query (default 2500).
    • Socket Timeout: Milliseconds to wait before timing out data transfer (default 10000ms).
    • Default Timezone: Timezone used if log timestamps lack timezone info (default GMT).
    • Sub Sample Drop/Receive Ratios: Settings to reduce or limit log volume by discarding logs in batches (default -1, meaning disabled).
    • Max Length in Bytes: Maximum size of each log message (default 32766 bytes).
    • Sleep Interval: Seconds to wait before re-querying after no logs are returned (default 60 seconds).
    • Polling Interval: Seconds between polling for new logs (default 0).
    • Drop if Queue is Full: Option to discard logs if MID Server load is high (default false).

    Practical Considerations for ServiceNow Customers

    • Ensure MID Servers or clusters selected support basic authentication; mTLS is not supported for log ingestion.
    • Monitor MID Server capacity as only up to 10 streaming data inputs per MID Server are supported by default.
    • Create and maintain operational service instances to bind log data properly.
    • Use query settings carefully to avoid large data transfers that may congest your system.
    • Adjust advanced settings to optimize log ingestion performance according to your network and processing capabilities.

    Description of the fields on the Amazon CloudWatch data input configuration form.

    Basic configuration

    Field Description
    Name Name of the new data input. This field is required.
    Description Description of the data input.
    Execute on Option to determine whether to use a specific MID Server or a MID Server cluster.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.
    MID

    (Only when the Execute on field is set to Specific MID Server)

    MID Server to which log data from Amazon CloudWatch is pulled.
    Note:
    • You can select only MID Servers that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    • If log ingestion is not enabled for the selected MID Server, Health Log Analytics enables it automatically.
    This field is required.
    MID Server Cluster

    (Only when the Execute on field is set to Specific MID Server Cluster)

    The MID Server cluster to which the log data is pulled.

    The data input runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the data input tasks to the next available MID Server in the cluster according to the configured order.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

    Note:
    • Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the data input form, the MID Server Clusters list displays only failover clusters.
    • The MID Server cluster must include only MID Servers that support basic authentication. mTLS is not supported for log ingestion.
    • Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 data inputs running on it, even when that MID Server is down.
    For more information about MID Server clusters, see Configure a MID Server cluster.

    This field is required.
    Service instance The service instance to which to bind the log data.
    Note:
    If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance to Operational.
    This field is required.
    The following fields show read-only information:
    Field Description
    Status Status of the data input.
    Transport Protocol used to stream the log data.

    This data input uses Amazon CloudWatch to stream log data to your instance.
    Sources count The number of log sources this data input has created.
    Disabled since The time when the data input stopped or failed.
    Last log time The time when the last log streamed in the data input.
    Error message The streaming error.

    This field is populated automatically. It displays only when a streaming error has occurred.
    Table 1. Query settings tab
    Field Description Example
    From The date and time to start reading the data. Data older than this date and time is not read.
    Note:
    Setting this value to a past date might require the system to read large amounts of data, causing congestion.

    This field is required.

    		 Now -1 week
    Group Name(s) The log groups to search. If multiple log groups must be searched, specify the groups in a comma-separated list. To fetch log data from all groups, use an asterisk (*) as a wildcard character.

    This field is required.

    		 hla-cw-loggroup1,hla-cw-loggroup2
    Prefix Name prefix for the Amazon CloudWatch log streams to read from. The data input reads only from log streams with this name prefix.
    Note:
    Only a single log stream prefix per data item is supported. For multiple prefixes, create multiple data inputs.
    		 hla-cw
    Filter pattern Pattern by which to filter incoming events.
    Various types of filter patterns are supported. For example:
    • A pattern for fetching log events that contain a single term.
    • A pattern for fetching log events that contain multiple terms.
    • A pattern for fetching log events that include a term and exclude another.
    Note:
    Filter patterns are case sensitive.
    For more information, see https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html
    • Pattern for fetching log events that contain a single term, "STATUS_CODE":

      "STATUS_CODE"

    • Pattern for fetching log events that contain multiple terms, “STATUS_CODE” and “200”:

      “STATUS_CODE 200”

    • Pattern for fetching log events that include a term, “STATUS_CODE,” and exclude another term, “200”:

      “STATUS_CODE” – “200”
    Table 2. Transport tab
    Field Description
    AWS credentials Field that refers to the AWS Credentials list (aws_credentials.list). The list contains the AWS access and secret access keys.
    AWS region The AWS region where the Amazon CloudWatch cluster runs, for example, us-west-1.

    For a list of AWS regions, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions

    This field is required.

    Advanced configuration

    Table 3. Advanced configuration form
    Field Description Default value
    Connection timeout The number of milliseconds to wait before timing out the AWS connection attempt. 1000
    Batch size The maximum number of logs retrieved per query. 2500
    Socket timeout The number of milliseconds to wait before timing out a data transfer over an established connection. 10000
    Default timezone The default timezone if the log date and time doesn't include timezone information. GMT
    Sub sample drop ratio The number of logs to batch together, out of which one will be discarded. This setting is used to reduce the number of fetched logs. -1
    Sub sample receive ratio The number of logs to batch together, out of which all but one will be discarded. This setting is used to decrease the number of received logs. -1
    Max length in bytes The maximum length of log messages, in bytes. 32766
    Sleep interval The interval, in seconds, to wait before querying again after a query has returned no logs. 60
    Polling interval The interval, in seconds, to wait before polling for new logs. 0
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. False