MID Server pre-upgrade check
Summarize
Summary of MID Server pre-upgrade check
The MID Server pre-upgrade check is an automated validation process that runs before upgrading a MID Server in ServiceNow. Its purpose is to detect and block potential issues that could cause upgrade failures or MID Server outages. The check is triggered both automatically when the instance detects an outdated MID Server version and manually during upgrades.
Show less
An AutoUpgrade monitor inside each MID Server compares the MID Server version with the instance version and initiates the pre-upgrade validation tests if an upgrade is needed. If any blocking issues are found, the upgrade is halted and logged in the MID Server Issue [eccagentissue] table until resolved. The monitor runs hourly until all checks pass, then proceeds to download and install the upgrade.
Pre-upgrade validation tests
The pre-upgrade check runs two types of tests:
- Mandatory tests: All must pass to proceed with the upgrade. Failures block the upgrade and create active issues that must be fixed.
- Alert tests: Failures generate warnings but do not stop the upgrade process.
Mandatory tests verify:
- At least 2 GB free disk space on the MID Server host
- Access permission to download packages from install.service-now.com
- Digital signature verification of a sample download file
- File permission checks including extraction, copying, reading, and deleting upgrade files
- Host system is not 32-bit
- Linux MID Servers have glibC version 2.17 or higher
Alert tests (Windows MID Servers) include:
- Presence of any services blocking upgrade
- Status of Application Experience service on the host
- Checks on the MID Server running user account, recommending a non-administrator user for security
Common blocking errors and warnings
Blocking errors that stop upgrades and require resolution include:
- Insufficient disk space (< 2 GB)
- Failure to download update packages due to network or permission issues
- Digital signature verification failures
- File permission issues preventing extraction or folder creation
- Host machine running 32-bit OS (unsupported)
- Linux glibC version below required minimum
Non-blocking warnings that appear in logs but do not prevent upgrade include:
- Windows service Log On As user format or group lookup issues
- PowerShell-related warnings if PowerShell is not installed or usable (not mandatory for MID Server operation)
- Running the MID Server Windows service as LocalSystem or Administrator user (not recommended)
- Specific service conflicts like CiscoAMP or disabled Application Experience service
Configuring the pre-upgrade check
The pre-upgrade check is enabled by default through the mid.upgrade.runprecheck parameter set to true. Customers can disable the pre-upgrade check:
- For an individual MID Server by setting
mid.upgrade.runprecheck=falsein that MID Server’sconfig.xmlfile. - For all MID Servers by adding a global mid.upgrade.runprecheck property with value false in the MID Server Property [eccagentproperty] table.
Practical implications for ServiceNow customers
This pre-upgrade check helps ensure that MID Server upgrades proceed smoothly without unexpected failures or outages by proactively validating system requirements, permissions, and configuration. Customers should monitor the eccagentissue table for any logged issues blocking upgrades and address them promptly.
Adhering to best practices such as ensuring sufficient disk space, proper network access, running the MID Server under a non-administrator user, and meeting OS and library prerequisites will facilitate successful upgrades. Disabling the pre-upgrade checks is possible but not recommended unless troubleshooting specific scenarios.
Prior to an upgrade, the MID Server runs tests to identify issues that could cause the upgrade process to fail or result in a MID Server outage.
Each MID Server contains an AutoUpgrade monitor that compares the MID Server version with that of the instance to determine if the MID Server needs to upgrade. If the AutoUpgrade monitor discovers that the MID Server version is out of date, the monitor runs pre-upgrade validation tests for that MID Server. If an issue is detected, a message is logged to the MID Server Issue [ecc_agent_issue] table, and the upgrade is blocked. The AutoUpgrade monitor continues to run every hour, until all the tests pass. If there are no blocking issues, the MID Server downloads the appropriate upgrade package and begins the upgrade process. The MID Server pre-upgrade check also runs for the upgrades triggered by the instance when the instance was upgraded, as well as for manual upgrades.
Pre-upgrade tests
The pre-upgrade runs two sets of tests: Mandatory testsand Alert tests. Every mandatory test is required to pass to start MID Server upgrade. If a mandatory test was not passed, the test is recorded in the agent log and a MID Server active issue is created in the MID Server issue [ecc_agent_issue] table. These errors are published before the actual MID Server upgrade occurs and must be resolved before the upgrade can continue. MID Server stops the Upgrade process and changes the status based on MID Server states. If any alert test was not passed, a message is added in the agent log and MID Server issue table, but the pre-upgrade check continues and MID Upgrade doesn’t stop.
- At least 2 GB of free disk space
- Access to the download site at install.service-now.com
- Verify the digital signature of a sample downloaded file
- Permission to execute, which involves the following:
- Extract a Zip archiver to the upgrade temp folder: agent/work/upgrade_temp
- Copy files from the upgrade temp folder to the agent folder
- Read a text file and validate the content
- Delete the pre-upgrade check contents
- MID Server host machine is not a 32 bit system
- On Linux MID Servers, the glibC version is at least 2.17
- Any MID Server Upgrade blocker services is running on the host machine
- Status of Application Experience on the host machine
- For security best practice, run the MID Server as a non-administrator user. If the current Log On As user account for the MID Server is LocalSystem or is a user that is part of the Administrators group, the upgrade does not fail, but this configuration is not recommended.
Errors that block the upgrade
- Not enough free disk space. The system reports <n> bytes free
- This message is displayed when less than 2 GB of free disk space is detected on the MID Server host. This error is also written to the MID Server agent log.
- Unable to download updates from the install server
- This message indicates that either the MID Server host does not have permission to download the installation package from install.service-now.com, or network problems prevent connection. This error is also written to the MID Server agent log.
- Signature verification failed:<message>
- This message indicates that a Verification Exception has occurred when pre-upgrade check tries to verify the digital signature of the downloaded check file.
- Unable to verify digital signature:<message>
- This message indicates that the verification signature process was interrupted by an exception.
- Unable to extract contents of pre upgrade check zip
- This message indicates that the service account on the MID Server host does not have permission to extract the pre-upgrade ZIP archive to the temporary folder.
- Unable to create folder <upgrade check file path>
- This message indicates that the MID Server service account does not have permission to create the upgradeCheck folder for the pre-upgrade checking files in the agent/package path.
- Unable to verify file permissions: <message>
- This message indicates an exception has occurred when checking file permissions, such as a file that does not exist or access failure.
- MID Server Windows Service is not running as LocalSystem or a local Administrator
- This message warns that the Windows service is not running with the desired permissions.
- Upgrade Failure: The host machine does not meet the minimum system requirements to upgrade the MID server.
- MID upgrade is not supported on 32-bit host. Please refer to KB0863694 for more information.
- Upgrade Failure: Host does not meet the minimum system requirements to upgrade the MID server. Please refer to MID Server product documentation on GlibC library for more information
- This error can only occur for Linux MID Servers. Refer to Java version support for more information.
Non-blocking warnings
- Unable to parse $logOnAsUser
- This message warns that the Log On As User value for the Windows service is not in
either of these expected formats:
- user@domain.company.com
- domain\user
- Unable to look up Log On As user's groups
When the instance attempts to look up the logged on user's group memberships, it executes the net user <username> command. The instance expects a certain output structure by the Windows service from this command and issues this warning if the expected output does not match the actual output.
These PowerShell warnings are written only to the MID Server agent log. Because PowerShell is not required to use a MID Server, these configuration issues do not prevent a Windows MID Server from upgrading. However, these warnings might indicate issues in your environment that require attention.
- Skipping PowerShell upgrade checks since PowerShell is not usable
- PowerShell 3.0 (at a minimum) is not installed or powershell.exe is not available to the MID Server service user.
- Skipping Powershell test: Target folder has not been initialized
- Skipping Powershell test: Unable to detect the service name
- Service CiscoAMP running on the MID Server host machine is identified as a MID Server upgrade blocker
- For more information, refer to KB0870329.
- The Application Experience is disabled on the MID Server host, which can cause the auto-upgrade process to fail and the MID Server to go down. Enable the service on the host to ensure successful auto-upgrade
- For more information, refer to KB0597552.
- Continuing with upgrade, but the following issue was encountered during upgradeCheck: <exception message>
- This message indicates that there was an issue running the PowerShell portion of the pre-upgrade tests.
Disabling the pre-upgrade check
A MID Server configuration parameter called mid.upgrade.run_precheck is set to true by default, which allows the automatic pre-upgrade test to run. To disable these tests for a single MID Server, add this parameter to that MID Server's config.xml file and set it to false. To disable these tests for all MID Servers, add a new record to the MID Server Property [ecc_agent_property] table called mid.upgrade.run_precheck. Set the value of this property to false and leave the MID Server field blank.