Rsyslog, Filebeat, or Winlogbeat data input configuration fields

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Rsyslog, Filebeat, or Winlogbeat Data Input Configuration Fields

    This guide explains the configuration fields for setting up data inputs using Rsyslog, Filebeat, or Winlogbeat in Health Log Analytics. Understanding these fields is essential for effectively streaming logs to your MID Server and ensuring proper log management within ServiceNow.

    Show full answer Show less

    Key Features

    • Basic Configuration:
      • Data Input Name: Required field for identifying the new data input.
      • Description: Brief description of the data input.
      • MID Server: Select a compatible MID Server for log streaming, limited to those supporting basic authentication.
      • Port: Specify an available port on the MID Server for log streaming; must be approved by your security team.
      • Content Pack: For Linux Filebeat, select a content pack that automates source type and mapping script setup.
    • Tagging and Binding:
      • Path: Full path for log streaming, supporting wildcards.
      • Application Service: Required binding to the appropriate application service.
      • Component: Contextual device type for logs, aiding anomaly detection.
      • Source Type: Defines how logs are parsed and handled within Health Log Analytics.
    • Advanced Configuration:
      • For Rsyslog: Options include SSL/TLS usage, hostname resolution, thread counts, read timeouts, and character encoding.
      • For Beats agents: Configuration similar to Rsyslog, with options for inactivity timeouts and log message length limits.

    Key Outcomes

    By accurately configuring these data input fields, you will enhance your log management capabilities, enabling effective monitoring and analysis of your IT infrastructure through ServiceNow’s Health Log Analytics. This setup ensures that logs are properly streamed, categorized, and utilized for operational insights.

    Description of the fields on the Rsyslog, Filebeat, and Winlogbeat data input configuration forms.

    Basic configuration

    Table 1. Getting Started tab
    Field Description
    Data input name Name of the new data input. This field is required.
    Description Description of the data input.
    MID Server The MID Server to which the logs stream.
    Note:
    • You can select only MID Servers with log ingestion capability that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    This field is required.
    Port

    The port on the MID Server.

    Choose a port within the suggested range from the array. The port must not be occupied by another process. Make sure that your organization’s security team opens the selected port.

    This field is required.
    Content pack (Linux using Filebeat only) The content pack to use.

    Content packs contain default source types and mapping script templates. Health Log Analytics activates the selected pack automatically and uses its mapping script for mapping the data input sources. For more information, see Health Log Analytics content packs for quicker time to value.
    Table 2. Tagging and Binding tab
    Field Description
    Path The full path from which to stream logs. You can use a wildcard. This field is required.
    Application Service The application service to which to bind the log data. This field is required.
    Note:
    If no relevant application service exists, Create an application service and add CIs to it. Set the status of the new application service to Operational.
    Component The device type or stack layer as context for the logs that is used for anomaly detection and correlation. For example: Tomcat.

    Components typically represent CIs in the CMDB. Several components are often clustered together in a single application service.
    Source Type The source type, which defines how Health Log Analytics handles a specific application and parses the log data. For example: Tomcat Catalina.

    Each data input can have multiple source types, based on the diversity of its log formats. Application services and components can have any number of source types.

    Advanced configuration

    For Rsyslog data inputs:

    Table 3. Rsyslog advanced configuration form
    Field Description Default values
    Use SSL/TLS Option for selecting to use SSL/TLS.
    Look up hostnames Option for selecting to perform DNS lookup to resolve IPs to hostnames. false
    Boss thread count The number of threads that manage connections. 1
    Worker thread count The number of threads that handle incoming data. 4
    Read timeout seconds The timeout in seconds since the last read. When the timeout expires, the system closes the channel. 30
    Default timezone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Max length in bytes The maximum length of log messages in bytes. 32766
    Character encoding The character encoding for this data input. UTF-8
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server.

    For data inputs that use Beats agents:

    Table 4. Beats advanced configuration form
    Field Description Default value
    Client inactivity timeout (sec) The timeout, in seconds, to close an inactive channel. 15
    Worker thread count The number of threads that handle incoming data. 4
    Default time zone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Max length in bytes The maximum length of log messages, in bytes. 32766
    Character encoding The character encoding for this data input. UTF-8
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. false