Data collection and discovery using Netflow
Summarize
Summary of Data collection and discovery using Netflow
Service Mapping in ServiceNow supports discovery of Configuration Items (CIs) and their connections using the Netflow protocol, enhancing traffic-based discovery methods. Netflow collects network traffic data alongside traditional methods like netstat and lsof commands, and VPC Flow Logs. This approach enables more comprehensive mapping of network relationships between CIs.
Show less
While base systems rely on TCP-related data from netstat, ss, and lsof commands by default, incorporating Netflow requires additional configuration.
Key Features
- Netflow Collector: A dedicated component that receives Netflow data. Its deployment varies based on the operational mode:
- Testing mode: The collector is installed on a separate server within the organization’s network, distinct from the MID Server. Data collection is semi-automated and requires manual copying of the data file to the MID Server.
- Standard operation: The collector runs on the same server as the MID Server, enabling fully automated data collection and processing.
- Data flow: Netflow daemons gather data from switches, which the Netflow Collector writes into an nfdump output file summarizing network traffic.
- Data processing: In testing setups, the nfdump file may need conversion to gzip format before being manually transferred to the MID Server. The MID Server processes this raw data, placing processed information on the ECC queue.
- Integration with Service Mapping: Sensors pull processed flow data from the ECC queue into the saflowconnection table. Service Mapping uses this table, along with the cmdbtcp table, to enrich connection details for discovered CIs.
Benefits for ServiceNow Customers
- Enhanced discovery accuracy by leveraging Netflow data to identify CI connections that traditional network commands might miss.
- Flexible deployment options to support both testing and fully automated production environments.
- Improved network topology mapping, enabling better understanding and management of CI relationships and dependencies.
Implementation Notes
Customers must configure the Netflow Collector appropriately based on whether they are testing or deploying in standard operation. Manual data handling is required in test mode, whereas standard mode supports full automation. Proper placement of the Netflow Collector relative to the MID Server is critical for successful data ingestion and processing.
Service Mapping can perform discovery based on data collected using the Netflow protocol. Netflow is a protocol that Service Mapping can use to collect data about CIs and their connections along with Netstat and lsof commands.
Using the Netflow protocol for collecting data is one of the traffic-based discovery methods. Other methods deployed by Service Mapping are using netstat and lsof commands and the VPC Flow Logs. For more information, refer to Traffic-based discovery in Service Mapping.
In base systems, which are the default or standard configurations, traffic-based discovery relies solely on TCP-related data collected using the netstat, ss, and lsof commands. Discovery based on Netflow and VPC logs requires additional configuration. You can enrich your traffic-based discovery by configuring Service Mapping to use the Netflow protocol.
- For the test purposes
- This setup results in half automated data collection flow, where Service Mapping imports data only if you manually copy it from the Netflow Collector. You place the Netflow Collector on a server inside your organization network. This must be a server different from the server hosting the MID Server. You configure and test this setup as described in Configure onetime data import using Netflow for testing purposes.
- For standard operation
- This setup results in fully automated data collection flow, where all involved components send, collect and analyze data automatically. You place the Netflow Collector on the same server as the MID Server inside your organization network. For instructions, see Configure data collection using Netflow.
- The Netflow daemon runs and receives data from switches communicating with servers in the organization. The Netflow Collector writes received data from the Netflow daemon.
- The server, hosting the Netflow collector, uses the Netflow nfdump utility to write the data into the nfdump output file. This file summarizes the raw data on all switches used for server communication.
Figure 1. Collecting data and writing it into the nfdump output file
- In testing setups, where the Netflow Collector is located not on the same server as the MID Server, you may need to convert the nfdump into the gzip format. Then you must manually copy the raw data in the nfdump output file onto the MID Server.
Figure 2. Copying the nfdump output file onto the MID Server
- The MID Server processes the raw data in the nfdump output file and places the processed information onto the ECC queue.
Figure 3. Analyzing the raw data and placing it at the ECC Queue
- A sensor retrieves the processes data from the ECC queue and writes it into the Flow Connection [sa_flow_connection] table.
Whenever Service Mapping checks the ECC queue and receives information on a discovered CI, it checks these tables for any data on outbound connections related to the CI: the cmdb_tcp and sa_flow_connection tables. If these two tables contain unique data that patterns did not discover, Service Mapping enriches the information about the CI connections and adds them to the map.
Figure 4. Service Mapping retrieves data from the sa_flow_connection table