Resolving MID Server issues

Release version: Yokohama

Updated January 30, 2025

3 minutes to read

Summarize

Summarized using AI

Summary of Resolving MID Server issues

This guide helps ServiceNow customers troubleshoot and resolve common issues with the MID Server. It covers monitoring techniques, identifying problems, and using built-in alerts and logs to quickly detect and address MID Server performance and connectivity issues. Understanding these procedures enables efficient MID Server management and minimizes downtime.

Show full answer Show less

Monitoring the MID Server

Check MID Server logs (agent0.log.0, wrapper.txt) for warning and error messages to identify problems.
Verify MID Server status from the MID Servers page in the ServiceNow instance and confirm connectivity between the MID Server host and the instance.
On Windows hosts, use the Services console to manage the MID Server service and monitor the Java process memory against configured limits.
Alerts such as MID Server Down notifications are enabled by default and can be configured to send email, SMS, or push notifications for real-time issue awareness.

Java Flight Recorder (JFR)

JFR allows detailed analysis of the Java Virtual Machine (JVM) and application memory usage and thread activity on MID Server hosts running Java 14 or later.

From the MID Server instance, users can start, stop, or save JFR recordings to gather diagnostic data on demand.
JFR files are automatically cleaned after 24 hours by default, but this interval can be customized via configuration parameters.
Custom JFR events can be disabled or enabled through specific configuration settings.

Troubleshooting Procedures and Tools

Use Knowledge Base articles for detailed resolutions to common issues such as MID Server downtime or discovery problems.
The ServiceNow instance maintains a dedicated table for active MID Server issues, alerting administrators when resource limits are approached or breached.
Automatic detection and notification of credential issues after instance cloning help prevent connectivity failures.
Resource threshold alerts notify users of high CPU or JVM memory usage, enabling proactive management through notifications or scripts.
User connectivity issues and login problems are logged in the eccagentissue table, with events generated for custom handling.
Debugging output for MID Server user issues can be enabled manually, providing either summary or detailed logs for investigation.
The MID Server Upgrade History module tracks upgrade attempts and errors to assist with troubleshooting upgrade failures.

Common Issues and Resolutions

MID Server fails to start: Often due to improper configuration; verify settings and logs to identify root causes.
Post-cloning credential problems: Automatically detected with alerts after instance cloning.
Resource threshold breaches: Alerts help prevent MID Server overload by notifying administrators when CPU or memory usage is excessive.

Related Concepts and Tasks

Customers can reference additional resources and perform related tasks such as manually starting, stopping, or restarting MID Servers, adjusting JVM memory size, pausing MID Servers, and reviewing system requirements and configuration parameters. These capabilities support comprehensive MID Server management and issue resolution.

Troubleshoot problems with the MID Server to find solutions. Monitor the MID Server to receive alerts about issues as they occur. Troubleshooting procedures exist to resolve specific problems with the MID Server. The Knowledge Base on Hi contains several articles to help you troubleshoot MID Server issues.

Monitor the MID Server

Monitoring a MID Server involves verifying entries in log files, confirming network connectivity, and checking MID server status.

If the MID Server is on a Windows host, navigate to the Windows Services console, locate the service name that matches the name that appears from the wrapper-override.conf file. If the MID Server process is the only Java process running on the host, monitor the memory used by java.exe and alert on less than the maximum configured memory defined in the ~\agent\conf\wrapper-override.conf folder.
Ensure that the agent0.log.0.lck file appears in the ~\agent\logs folder to confirm that the MID Server running and logging system activity in the agent0.log.0 file.
Review the following logs for warning, critical, and severe errors: ~\agent\logs\agent0.log.0 ~\agent\logs\wrapper.txt See Manage ECC Queue content for a MID Server to see how to open log entries from the instance.
Test MID Server connectivity to confirm that the MID Server host is able to communicate with the instance. Resolve any issues with the network connectivity such as conflicts with firewalls, access control lists, and routing errors.
From the MID Server instance, navigate to the MID Servers page, and review the status of the MID Server. For additional information, click a Name.
Set up email, SMS, and push notifications to alert you when issues occur with MID servers. The MID Server Down notification is enabled by default. See Notifications for details.

Java Flight Recorder

Java Flight Recorder (JFR) uses an articulate data model to analyze Java Virtual Machine (JVM) internal and application specific memory usage and thread related details. On hosts with Java 14 or later, Event Streaming enables analysis of specific JVM or application events.

The JFR file actions Start JFR Recording, Stop JFR Recording, or Write JFR File, are accessed on the instance in the MID Server's Related Links. By starting and stopping the JFR recording, you can dump JFR data to the disk on demand. While JFR recording is on, you can write to the JFR file from the MID Server instance page.

The JFR file is deleted by a scheduler to clean the data after 24 hours. This default behavior can be overridden by adding the following configuration parameter in the config.xml: <parameter name="mid.jfr.data.clean.interval" value="<time in milisec>"/>

Custom JFR events can be written to JFR file. To disable event writing, add the following configuration parameter to the config.xml: <parameter name="mid.jfr.record.aprobe" value="false”/>

Troubleshooting procedures

The following procedures provide information on resolving specific problems with the MID Server. Knowledge Base articles also cover a wide range of issues. If you are encountering problems which are not covered in these procedures, search the Knowledge Base for related articles. For instance, if you are experiencing symptoms such as the MID Server going down or not responding, or CIs being duplicated during discovery, see KB0597571 for information and recommended solutions.

MID Server active issues: The ServiceNow® instance has a dedicated table that publishes active issues with MID Servers and alerts administrators when a MID Server is in danger of exceeding its resources.
Active MID Server post-cloning credential issues: The system provides automatic processes to detect and notify you of possible MID Server credential issues after instance cloning.
MID Server resource threshold alerts: The instance displays warnings when a MID Server breaches its resource thresholds for CPU and JVM memory usage, enabling users to create email notifications or custom scripts when a breach occurs.
MID Server user connectivity issues: The instance writes issues involving MID Server user logins and network connectivity to the MID Server Issue [ecc_agent_issue] table and creates events you can use in custom scripts or to send email notifications.
Interpreting MID Server user debugging output: Debugging output from the system log is available in either a summary or detailed view for MID Server user issues, but must be enabled manually.
MID Server Upgrade History: Use this module to troubleshoot errors that occur during the MID Server upgrade process. The MID Server Upgrade Histories table contains a record of each instance upgrade. The MID Server Upgrade Stages table shows the status of each MID Server and its upgrade progress, including any errors encountered.