Configure data inputs (Splunk)

Release version: Washingtondc

Updated February 1, 2024

5 minutes to read

Configure a data input for streaming log messages to your ServiceNow instance using a Splunk heavy forwarder.

Before you begin

Important:

Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.

You must have an installed and configured MID Server with the log ingestion capability enabled.
If the MID Server IP address is exposed by network address translation (NAT), a load balancer, or a similar device, it must have a public IP address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property.
For information about shipping your logs encrypted using SSL TLS, see the Streaming Data With Rsyslog & Filebeat Using SSL [KB0866319] article in the Now Support Knowledge Base.

Configure Splunk to forward logs to your ServiceNow instance using Syslog.
The configuration of this data input assumes the existence of an environment variable named $SPLUNK_HOME. In Unix-like environments, this variable typically points to /opt/splunk.
Note:
The Windows environment uses the same directory structure but with backslashes (\).

Role required: evt_mgmt_admin

About this task

This setup procedure is for streaming logs to your instance using a Splunk heavy forwarder. If you can't use a heavy forwarder, you can use a universal forwarder instead. For more information, see the Splunk Universal Forwarder as a Shipping method [KB0961378] article in the Now Support Knowledge Base.

Note:

A MID Server that is down can cause a blockage in your Splunk pipeline. A full processing queue does not affect the pipeline.

Note:

All Splunk configuration files are located in the $SPLUNK_HOME/etc/system/local/ folder. If a configuration file that you need to modify doesn't exist, create it and save it to this folder.

Procedure

Navigate to All > Health Log Analytics > Data Input > Data Inputs.
On the Data Inputs page, select New.
Choose the Splunk data input.
On the Getting Started tab, fill in the form fields.
For a description of the fields, see Splunk data input configuration fields.
On the Outputs.conf tab, add the following stanzas to the outputs.conf file to make the shipper forward log data over the selected transport protocol on the selected port, and then select Next.

Note:
If you have already configured outputs, merge these lines with your existing configuration.
- Forwarding over TCP:
  Note:
  Use the first stanza only if you have not already configured a tcpout stanza. The second stanza is required for forwarding to Health Log Analytics over TCP.
```
[tcpout]
indexAndForward = 1
defaultGroup = nothing

[tcpout:health_log_analytics]
server = ip-172-31-17-121.us-east-2.compute.internal:5000
sendCookedData = false
compressed = false
```
- Forwarding over UDP:
  
  Note:
  Use the first stanza only if you have not already configured a syslog stanza. The second stanza is required for forwarding to Health Log Analytics over UDP.
```
[syslog]defaultGroup = nothing

[syslog:health_log_analytics]
server = ip-172-31-17-121.us-east-2.compute.internal:5000
```
On the Props.conf tab, edit the props.conf file, and then select Next.
1. Modify existing stanzas or add stanzas to mark source types, application services, and hosts for forwarding to Health Log Analytics.
  Note:
  For best results, mark only source types for forwarding.
  When adding stanzas, use the following name formats:
  
  Source types: [<source type>]. For example: [syslog]
  
  Sources (not recommended): [source::<source>]. For example, [source::myApp]
  
  Hosts (not recommended): [host::<host>]. For example, [host::10.9.8.7]
2. Add the following line to the end of each stanza that you want to forward to Health Log Analytics over either TCP or UDP.
  - Forwarding over TCP:
    TRANSFORMS-cloned_data = clone_for_hla_store_sourcetype, clone_for_hla
  - Forwarding over UDP:
    TRANSFORMS-cloned_data = clone_for_hla_store_sourcetype, clone_for_hla_udp
    This line applies the CLONE_SOURCETYPE transform on the data to prevent manipulation needed for Health Log Analytics processing from affecting your existing data pipeline. For example, to send all logs from the source type "syslog" to Health Log Analytics:
    [syslog] #existing configuration goes here TRANSFORMS-cloned_data = clone_for_hla_store_sourcetype, clone_for_hla_udp
3. Add the following stanza to apply all the relevant transforms needed for Health Log Analytics processing.
  Note:
  Splunk enables you to anonymize sensitive data on the cloned source type for the selected protocol. For more information, see the "Anonymize data" section in the Splunk documentation.
  - Forwarding over TCP:
    [send_to_hla_tcp] TRANSFORMS-health_log_analytics = health_log_analytics_add_sourcetype, health_log_analytics_add_host, health_log_analytics_add_source, health_log_analytics_add_syslog5424, health_log_analytics_add_index, health_log_analytics_add_syslogHost, health_log_analytics_add_time
  - Forwarding over UDP:
    [send_to_hla_udp] TRANSFORMS-health_log_analytics = health_log_analytics_add_sourcetype, health_log_analytics_add_host, health_log_analytics_add_source, health_log_analytics_add_syslog5424, health_log_analytics_add_index, health_log_analytics_add_syslogHost, health_log_analytics_add_time

On the Transforms.conf tab, add the following stanzas to the transforms.conf file, and then select Next.

The third stanza clones the logs for further manipulation without affecting your existing indexing. The remaining stanzas add the information needed to enable correct Health Log Analytics processing.

Note:

You can obfuscate sensitive data by adding a transform here and then modifying the stanza of the cloned source type in the props.conf file.

[accepted_keys]
#Custom field for preserving sourcetype
hla_sourcetype_preservation=_hla_sourcetype

#Store sourcetype in a custom field, since CLONE_SOURCETYPE overwrites it
[clone_for_hla_store_sourcetype]
SOURCE_KEY = MetaData:Sourcetype
REGEX = ^sourcetype::(.+)$
FORMAT = hla_sourcetype::$1
DEST_KEY = _hla_sourcetype

[clone_for_hla]
REGEX=.
DEST_KEY = _TCP_ROUTING
FORMAT = health_log_analytics
CLONE_SOURCETYPE=send_to_hla_tcp

#Only used in case of UDP forwarding
[clone_for_hla_udp]
REGEX=.
DEST_KEY = _SYSLOG_ROUTING
FORMAT = health_log_analytics
CLONE_SOURCETYPE=send_to_hla_udp

#Add metadata to the log message, since metadata is lost when forwarding externally
[health_log_analytics_add_sourcetype]
SOURCE_KEY = _hla_sourcetype
REGEX = ^hla_sourcetype::(.+)$
FORMAT = sourcetype="$1"] $0
DEST_KEY = _raw

[health_log_analytics_add_host]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(.+)$
FORMAT = host="$1" $0
DEST_KEY = _raw

[health_log_analytics_add_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(.+)$
FORMAT = source="$1" $0
DEST_KEY = _raw

[health_log_analytics_add_syslog5424]
REGEX=.
FORMAT = - - [sdid@1234 $0
DEST_KEY = _raw

[health_log_analytics_add_index]
SOURCE_KEY = _MetaData:Index
REGEX = ^(.+)$
FORMAT = $1 $0
DEST_KEY = _raw

[health_log_analytics_add_syslogHost]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(.+)$
FORMAT = $1 $0
DEST_KEY = _raw

[health_log_analytics_add_time]
SOURCE_KEY = _time
REGEX = ^(.+)$
FORMAT = <182>1 $1 $0
DEST_KEY = _raw

On the Finish.conf tab, restart Splunk by running the $SPLUNK_HOME/bin/splunk restart splunkd command.
Select Save.
Health Log Analytics adds the data input record to the Data Inputs table.
Ensure that the data input is configured correctly by selecting Test connection.

Health Log Analytics tries to connect the MID Server to the data repository.
- If the connection was established, the Test connection button is turned off and the Publish button is enabled.
- If the connection failed, the reason for the failure displays in the Error message field. This field displays only when a streaming error has occurred.
  Resolve the issue, select Save if you modified the configuration, and then select Test connection to test the connection again.
  
  Note:
  You can only publish the data input configuration when the connection is created successfully.
Note:
You can revert to the last published configuration by selecting Revert Changes. This option is available only when you're modifying a configuration that has been published previously.
Select Publish to publish the data input to the MID Server.

Result

The data input starts streaming log messages to your instance using a Splunk shipper.

Note:

If the Health Log Analytics AI engine is down and data has stopped streaming, a notification appears at the top of the data input configuration page. When this happens, contact ServiceNow support.

What to do next

Make sure that the data input is streaming data.