Rsyslog, Filebeat, or Winlogbeat data input configuration fields

  • Release version: Yokohama
  • Updated January 30, 2025
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Rsyslog, Filebeat, or Winlogbeat Data Input Configuration Fields

    This guide explains the configuration fields for data inputs using Rsyslog, Filebeat, or Winlogbeat within ServiceNow’s Health Log Analytics. It helps customers set up log streaming from various sources through a MID Server, enabling efficient log ingestion, parsing, and anomaly detection.

    Show full answer Show less

    Basic Configuration

    • Data input name: Required name for the new data input.
    • Description: Optional description of the data input.
    • MID Server: Required selection of a MID Server with log ingestion capability supporting basic authentication (mTLS servers are excluded). By default, up to 10 data inputs can stream logs to one MID Server, adjustable in MID Server properties.
    • Port: Required port on the MID Server for streaming logs, which must be free and approved by security.
    • Content pack (Linux/Filebeat only): Optional selection of a content pack that provides default source types and mapping scripts to accelerate setup and improve log parsing.

    Tagging and Binding Configuration

    • Path: Required full path or wildcard for log streaming source.
    • Service instance: Required service instance to bind the log data. Customers must create and set to Operational if none exists.
    • Component: Optional device type or stack layer context (e.g., Tomcat) for anomaly detection and correlation, representing CIs in the CMDB.
    • Source Type: Defines how logs are parsed for a specific application; multiple source types can be assigned per data input or service instance.

    Advanced Configuration for Rsyslog Inputs

    Allows customization of connection and data handling parameters:

    • Enable SSL/TLS for secure transmission.
    • DNS lookup for resolving IPs to hostnames.
    • Adjust thread counts for connection management and data processing.
    • Set read timeout, default timezone, message size limits, and character encoding.
    • Control subsampling ratios for event dropping or receiving.
    • Option to discard logs if the MID Server queue is full to prevent overload.

    Advanced Configuration for Beats Inputs (Filebeat, Winlogbeat)

    • Set client inactivity timeout to close idle channels.
    • Configure worker thread count for data handling.
    • Specify default timezone, max message length, and character encoding.
    • Manage event subsampling ratios for dropped or received logs.
    • Option to drop logs if MID Server is under load to maintain stability.

    Practical Benefits for ServiceNow Customers

    By following these configurations, customers can:

    • Efficiently ingest and parse log data from Linux and Windows systems using preferred agents.
    • Bind log inputs to service instances and components to enable detailed anomaly detection and correlation.
    • Customize advanced parameters to optimize performance, security, and resource usage on MID Servers.
    • Ensure reliable log streaming aligned with organizational security and operational policies.

    Description of the fields on the Rsyslog, Filebeat, and Winlogbeat data input configuration forms.

    Basic configuration

    Table 1. Getting Started tab
    Field Description
    Data input name Name of the new data input. This field is required.
    Description Description of the data input.
    MID Server The MID Server to which the logs stream.
    Note:
    • You can select only MID Servers with log ingestion capability that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    This field is required.
    Port

    The port on the MID Server.

    Choose a port within the suggested range from the array. The port must not be occupied by another process. Make sure that your organization’s security team opens the selected port.

    This field is required.
    Content pack (Linux using Filebeat only) The content pack to use.

    Content packs contain default source types and mapping script templates. Health Log Analytics activates the selected pack automatically and uses its mapping script for mapping the data input sources. For more information, see Health Log Analytics content packs for quicker time to value.
    Table 2. Tagging and Binding tab
    Field Description
    Path The full path from which to stream logs. You can use a wildcard. This field is required.
    Service instance The service instance to which to bind the log data. This field is required.
    Note:
    If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance to Operational.
    Component The device type or stack layer as context for the logs that is used for anomaly detection and correlation. For example: Tomcat.

    Components typically represent CIs in the CMDB. Several components are often clustered together in a single service instance.
    Source Type The source type, which defines how Health Log Analytics handles a specific application and parses the log data. For example: Tomcat Catalina.

    Each data input can have multiple source types, based on the diversity of its log formats. Service instances and components can have any number of source types.

    Advanced configuration

    For Rsyslog data inputs:

    Table 3. Rsyslog advanced configuration form
    Field Description Default values
    Use SSL/TLS Option for selecting to use SSL/TLS.
    Look up hostnames Option for selecting to perform DNS lookup to resolve IPs to hostnames. false
    Boss thread count The number of threads that manage connections. 1
    Worker thread count The number of threads that handle incoming data. 4
    Read timeout seconds The timeout in seconds since the last read. When the timeout expires, the system closes the channel. 30
    Default timezone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Max length in bytes The maximum length of log messages in bytes. 32766
    Character encoding The character encoding for this data input. UTF-8
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server.

    For data inputs that use Beats agents:

    Table 4. Beats advanced configuration form
    Field Description Default value
    Client inactivity timeout (sec) The timeout, in seconds, to close an inactive channel. 15
    Worker thread count The number of threads that handle incoming data. 4
    Default time zone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Max length in bytes The maximum length of log messages, in bytes. 32766
    Character encoding The character encoding for this data input. UTF-8
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. false