Microsoft Azure Log Analytics data input configuration fields

  • Release version: Xanadu
  • Updated August 1, 2024
  • 5 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Microsoft Azure Log Analytics data input configuration fields

    This documentation details the configuration fields for setting up Microsoft Azure Log Analytics as a data input source within ServiceNow’s Health Log Analytics application (Version 26.0.17 and later). It guides you through specifying connection parameters, query settings, transport details, and advanced options to stream log data efficiently from Microsoft Azure into your ServiceNow instance.

    Show full answer Show less

    Basic configuration

    • Name and Description: Define a unique name and optional description for the data input.
    • Execute on: Choose whether to run the data input on a specific MID Server or a MID Server cluster. Only MID Servers supporting basic authentication are compatible; mTLS is not supported for log ingestion.
    • MID Server / Cluster: If a specific MID Server or cluster is selected, the configuration enables automatic log ingestion activation if not already enabled. Clusters provide failover support by running on an active MID Server and switching upon failure.
    • Service instance: Bind the log data to an existing operational service instance or create one if none exists.

    Read-only fields

    The form also displays status indicators such as data input status, transport protocol, last log time, number of sources created, and any streaming errors, helping you monitor the health and performance of the data input.

    Query settings

    • From: Set the starting date/time to fetch log data, preventing retrieval of excessively old data which may cause congestion.
    • Data source name: Specify the Microsoft Azure Log Analytics table name to pull logs from (e.g., ContainerLog).
    • Event time property name: Indicate the Azure field representing event time (e.g., TimeGenerated).
    • Max documents per query: Limit rows per query to control data volume (default 500).
    • Columns to select: List columns to retrieve; ignored if a custom query is used.
    • Log query: Optionally provide a JSON-formatted custom query to override other query fields (except the From date). This allows precise filtering and selection of logs.

    Transport configuration

    • Redirect URL: Set the Azure OAuth redirect URI for authentication.
    • Azure service principal credentials: Use Client Secret credentials (Azure EA credentials are not required) to authenticate access.
    • Workspace ID: Provide the Customer ID to call the Azure Log Analytics REST API.

    Advanced configuration

    • Event Processor workers: Number of concurrent workers processing event batches (default 4).
    • Workers Queue Size: Size of the queue for event processing (default 5).
    • Sub sample drop/receive ratio: Settings to reduce event volume by selectively dropping or receiving events in batches.
    • Character encoding: Encoding used for data input (default UTF-8).
    • Sleep interval: Wait time in seconds before re-querying after no events are returned (default 60).
    • Polling interval: Interval in seconds before polling for new events (default 0).
    • Drop if queue is full: Option to discard logs under high MID Server load (default False).

    Practical application for ServiceNow customers

    By correctly configuring these fields, ServiceNow customers can reliably ingest and stream Microsoft Azure log data into their ServiceNow environment. This enables effective health monitoring, log analysis, and incident detection leveraging Azure logs. Understanding each configuration option helps optimize data flow, ensure failover resilience with MID Server clusters, and tailor queries to fetch relevant log events efficiently.

    Description of the fields on the Microsoft Azure Log Analytics data input configuration form.

    Basic configuration

    Field Description
    Name Name of the new data input. This field is required.
    Description Description of the data input.
    Execute on Option to determine whether to use a specific MID Server or a MID Server cluster.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.
    MID

    (Only when the Execute on field is set to Specific MID Server)

    MID Server to which log data from Microsoft Azure Log Analytics is pulled.
    Note:
    • You can select only MID Servers that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    • If log ingestion is not enabled for the selected MID Server, Health Log Analytics enables it automatically.
    This field is required.
    MID Server Cluster

    (Only when the Execute on field is set to Specific MID Server Cluster)

    The MID Server cluster to which the log data is pulled.

    The data input runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the data input tasks to the next available MID Server in the cluster according to the configured order.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

    Note:
    • Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the data input form, the MID Server Clusters list displays only failover clusters.
    • The MID Server cluster must include only MID Servers that support basic authentication. mTLS is not supported for log ingestion.
    • Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 data inputs running on it, even when that MID Server is down.
    For more information about MID Server clusters, see Configure a MID Server cluster.

    This field is required.
    Service instance The service instance to which to bind the log data.
    Note:
    If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance to Operational.
    This field is required.
    The following fields show read-only information:
    Field Description
    Status Status of the data input.
    Transport Protocol used to stream the log data.

    This data input uses Microsoft Azure Log Analytics to stream log data to your instance.
    Disabled since The time when the data input stopped or failed.
    Sources count The number of log sources this data input has created.
    Last log time The time when the last log streamed in the data input.
    Error message The streaming error.

    This field is populated automatically. It displays only when a streaming error has occurred.
    Table 1. Query settings tab

    Health Log Analytics uses the values set in these fields to generate the query for receiving log data from Microsoft Azure Log Analytics. The Log query field enables you to configure a custom query
    Field Description Example
    From Starting date and time for reading the data. Data older than this date and time is not read.
    Note:
    Setting this value to a past date might require the system to read large amounts of data, causing congestion.

    This field is required.

    		 Now -1 week
    Data source name The name of the table in Microsoft Azure Log Analytics where the data input fetches the log data. For more information, see the View table information section in the Microsoft Azure documentation.

    This field is required.

    		 ContainerLog
    Event time property name The Microsoft Azure Log Analytics field in which to detect the event time.

    This field is required.

    		 TimeGenerated
    Max documents per query The highest number of rows retrieved in each query. 500
    Columns to select Comma-separated list of column names to return.
    Note:
    This field is ignored when you provide a custom query.
    		 LogEntry,LogEntrySource
    Log query A custom query for receiving log data from Microsoft Azure Log Analytics.

    The settings in this field override those in all other fields on the Query settings tab except the From field. If the Log query field is empty, Health Log Analytics generates the query using the values set in the other fields.

    For the custom query, use the following JSON format:

    {"query": "query | where TimeGenerated > %s | take 500"}

    		 
    {
"query":"ContainerLog
                    | where LogEntry contains 'cartservice'
                    | where TimeGenerated > %s
                    | take 500",
"workspaces": ["defaultworkspace-3ab145ff-f9cd-433f-8533-d1b1ee24aee6-eus"],
"project": ["TimeGenerated", "LogEntry", "LogEntrySource"]
}
    Table 2. Transport tab
    Field Description
    Redirect Url The redirect URL of the access log application.

    The URL refers to the Microsoft Azure redirect_uri authorization property. For more information, see the Authorization code URL (GET request) section in the Microsoft Azure documentation.

    This field is required.
    Azure service principal credentials The credentials used to access Microsoft Azure resources.

    Choose Client Secret from the drop-down list.

    Note:
    You do not need Azure Enterprise Agreement (EA) credentials.

    This field is required.
    Workspace Id The Customer ID used to call the Microsoft Azure Log Analytics REST API.

    Advanced configuration

    Table 3. Advanced configuration form
    Field Description Default value
    Event Processor workers The number of concurrent event processing workers, where each worker processes a batch of events independently. 4
    Workers Queue Size The queue size of the Event Processor workers. 5
    Sub sample drop ratio The number of events to batch together, out of which one will be discarded. This setting is used to reduce the number of fetched events. -1
    Sub sample receive ratio The number of events to batch together, out of which all but one will be discarded. This setting is used to decrease the number of received events. -1
    Character encoding The character encoding for this data input. UTF-8
    Sleep interval The interval, in seconds, to wait before querying again after a query has returned no events. 60
    Polling interval The interval, in seconds, to wait before polling for new events. 0
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. False