Resolving MID Server issues

Release version: Xanadu

Updated August 1, 2024

3 minutes to read

Summarize

Summarized using AI

Summary of Resolving MID Server Issues

This guide helps ServiceNow customers troubleshoot and resolve issues related to the MID Server, ensuring reliable communication and operation between the MID Server and the ServiceNow instance. It includes monitoring techniques, use of diagnostic tools like Java Flight Recorder, and procedures for addressing common problems.

Show full answer Show less

Monitoring the MID Server

Verify MID Server activity by checking log files such as agent0.log.0 and wrapper.txt for warnings and errors.
On Windows hosts, confirm the MID Server service status in the Windows Services console, correlating it with the wrapper-override.conf configuration.
Monitor Java process memory usage to ensure it does not exceed configured limits.
Test network connectivity between the MID Server host and the ServiceNow instance, resolving firewall or routing issues.
Review MID Server status on the instance's MID Servers page and enable email, SMS, or push notifications for alerts; the “MID Server Down” notification is enabled by default.

Java Flight Recorder (JFR) Integration

Use JFR on hosts with Java 14+ to analyze JVM memory and thread activity for performance troubleshooting.
Start, stop, and write JFR recordings from the MID Server instance page to capture JVM event data on demand.
JFR files are automatically cleaned after 24 hours by default, with configurable retention via mid.jfr.data.clean.interval.
Custom JFR event writing can be disabled using the mid.jfr.record.aprobe parameter.

Troubleshooting Procedures and Tools

Follow documented procedures and consult Knowledge Base articles for specific MID Server problems, such as unexpected shutdowns or duplicate CIs during discovery.
Use the ServiceNow instance’s dedicated tables to monitor active MID Server issues, including resource thresholds, credential problems post-instance cloning, and user connectivity errors.
Set up alerts and notifications for resource threshold breaches (CPU, JVM memory) and user login or connectivity failures.
Enable and interpret detailed MID Server user debugging output from system logs for deeper diagnostics.
Utilize the MID Server Upgrade History module to track upgrade progress and troubleshoot upgrade-related errors.

Common MID Server Issues

MID Server fails to start: Often caused by improper configuration; reviewing logs and configuration files is essential.
Post-cloning credential issues: Automatically detected and notified by the system to prevent authentication failures.
Resource threshold alerts: Warnings generated when CPU or JVM memory usage crosses configured limits, enabling proactive management.
User connectivity problems: Logged in a dedicated table with events to facilitate troubleshooting and notification.

Related Concepts and Tasks

Customers can also refer to related tools and configurations such as MID Server upgrades, dashboard, File Cleaner, privileged commands, and methods for managing MID Server lifecycle (start, stop, restart, pause) and memory sizing.

Understanding system requirements, properties, parameters, configuration priorities, and reserved characters further supports effective MID Server management and troubleshooting.

Troubleshoot problems with the MID Server to find solutions. Monitor the MID Server to receive alerts about issues as they occur. Troubleshooting procedures exist to resolve specific problems with the MID Server. The Knowledge Base on Hi contains several articles to help you troubleshoot MID Server issues.

Monitor the MID Server

Monitoring a MID Server involves verifying entries in log files, confirming network connectivity, and checking MID server status.

If the MID Server is on a Windows host, navigate to the Windows Services console, locate the service name that matches the name that appears from the wrapper-override.conf file. If the MID Server process is the only Java process running on the host, monitor the memory used by java.exe and alert on less than the maximum configured memory defined in the ~\agent\conf\wrapper-override.conf folder.
Ensure that the agent0.log.0.lck file appears in the ~\agent\logs folder to confirm that the MID Server running and logging system activity in the agent0.log.0 file.
Review the following logs for warning, critical, and severe errors: ~\agent\logs\agent0.log.0 ~\agent\logs\wrapper.txt See Manage ECC Queue content for a MID Server to see how to open log entries from the instance.
Test MID Server connectivity to confirm that the MID Server host is able to communicate with the instance. Resolve any issues with the network connectivity such as conflicts with firewalls, access control lists, and routing errors.
From the MID Server instance, navigate to the MID Servers page, and review the status of the MID Server. For additional information, click a Name.
Set up email, SMS, and push notifications to alert you when issues occur with MID servers. The MID Server Down notification is enabled by default. See Notifications for details.

Java Flight Recorder

Java Flight Recorder (JFR) uses an articulate data model to analyze Java Virtual Machine (JVM) internal and application specific memory usage and thread related details. On hosts with Java 14 or later, Event Streaming enables analysis of specific JVM or application events.

The JFR file actions Start JFR Recording, Stop JFR Recording, or Write JFR File, are accessed on the instance in the MID Server's Related Links. By starting and stopping the JFR recording, you can dump JFR data to the disk on demand. While JFR recording is on, you can write to the JFR file from the MID Server instance page.

The JFR file is deleted by a scheduler to clean the data after 24 hours. This default behavior can be overridden by adding the following configuration parameter in the config.xml: <parameter name="mid.jfr.data.clean.interval" value="<time in milisec>"/>

Custom JFR events can be written to JFR file. To disable event writing, add the following configuration parameter to the config.xml: <parameter name="mid.jfr.record.aprobe" value="false”/>

Troubleshooting procedures

The following procedures provide information on resolving specific problems with the MID Server. Knowledge Base articles also cover a wide range of issues. If you are encountering problems which are not covered in these procedures, search the Knowledge Base for related articles. For instance, if you are experiencing symptoms such as the MID Server going down or not responding, or CIs being duplicated during discovery, see KB0597571 for information and recommended solutions.

MID Server active issues: The ServiceNow® instance has a dedicated table that publishes active issues with MID Servers and alerts administrators when a MID Server is in danger of exceeding its resources.
Active MID Server post-cloning credential issues: The system provides automatic processes to detect and notify you of possible MID Server credential issues after instance cloning.
MID Server resource threshold alerts: The instance displays warnings when a MID Server breaches its resource thresholds for CPU and JVM memory usage, enabling users to create email notifications or custom scripts when a breach occurs.
MID Server user connectivity issues: The instance writes issues involving MID Server user logins and network connectivity to the MID Server Issue [ecc_agent_issue] table and creates events you can use in custom scripts or to send email notifications.
Interpreting MID Server user debugging output: Debugging output from the system log is available in either a summary or detailed view for MID Server user issues, but must be enabled manually.
MID Server Upgrade History: Use this module to troubleshoot errors that occur during the MID Server upgrade process. The MID Server Upgrade Histories table contains a record of each instance upgrade. The MID Server Upgrade Stages table shows the status of each MID Server and its upgrade progress, including any errors encountered.