Types of defined desktop actions

AI agents use these desktop actions to interact with desktop applications, perform UI-based tasks, and automate end-to-end workflows. There are two types of desktop actions that can be added as tools to AI agents:

On-screen task

These desktop actions help you simulate humans interacting with UI elements on your thick client applications, legacy systems, or SaaS applications without APIs. These actions include clicking buttons, typing into text boxes, selecting from dropdown menus, and more. These desktop actions make it easier to automate repetitive tasks. They encapsulate repeatable UI interactions, such as screens, anchors, and steps. You can create, manage, and test your desktop actions using the no-code Design workspace.

Example: Filling in fields on a payroll application, updating inventory in a point-of-sale system, or submitting a request through a desktop insurance claims app.

You can access and view the existing desktop actions in the ServiceNow instance by navigating to All > AI Desktop Actions > Desktop Actions. You can see the details of related desktop action applications, AI agents, and desktop action execution records for on-screen task, and AI agents and desktop action execution records for background desktop actions.

Background task

These desktop actions include prebuilt connectors that enable your AI agents to interact with various applications and system components in the background. These connectors streamline automation by offering actions for common tasks, reducing the need for complex scripting. Each connector focuses on a specific application or system area, providing a collection of related methods. You can't create background task desktop actions.

Example: Reading data from Microsoft Excel, extracting emails from Microsoft Outlook, generating a PDF, compressing files into a ZIP, fetching records from a database, or sending a system notification.

Background task desktop actions are supported for the following applications:

Note:

For the AI agents to perform Background task automations, the following applications must already be installed on the end-user's system.

Application	Description
Microsoft Excel	Enables AI agents to perform standard actions on the Microsoft Excel documents. For example, data manipulation, content modification, and information retrieval from spreadsheets. The following methods are supported: ReadData WriteData FindAndReplace Note: CSV and password-protected files aren't supported.
Microsoft Outlook	Enables AI agents to perform standard actions on the Microsoft Outlook application. For example, you can automate sending or replying to emails. The following methods are supported: DeleteMail ForwardMail GetMailsFromFolder MoveMail Reply SaveAllAttachments SendMail GetMail MarkAsRead MarkAsUnread Note: Only the classic view is supported. Shared inboxes aren’t supported.
Microsoft Word	Enables AI agents to perform standard actions on the Microsoft Word documents. For example, you can replace text in a document. The following methods are supported: GetText InsertText ReplaceText Note: Password-protected files aren't supported.
PDF	Enables AI agents to perform standard actions on the PDF documents. For example, extracting text, converting to Excel or Images, and merging and spiting files. The following methods are supported: GetText ConvertToExcel ConvertToImages Merge Split Note: Password-protected files aren't supported. You can't manipulate e-signed PDFs such as performing merge and split. This action corrupts the PDF files.
PowerShell	Enables AI agents to execute PowerShell commands and scripts and returns the results. The following methods are supported: InvokeCommand InvokeScript Note: Each PowerShell step runs in a new session. Output from one step isn’t carried over to the next step.
SQL	Enables AI agents to execute custom SQL queries against supported databases (MSSQL, ORACLE, MYSQL, OLEDB, ODBC) and returns results as JSON. The following method is supported: ExecuteQuery
Secure Shell (SSH)	Enables AI agents to securely connect to remote servers to run SSH commands and scripts; and returns the execution results. This connector supports non-interactive commands. The following method is supported: RunCommand Note: Each SSH step runs in a new session. Output from one step isn’t carried over to the next step.
System Actions	Enables AI agents to perform standard Windows system operations. For example, starting an app, creating a ZIP file, or deleting any file or folder. The following methods are supported: StartApp Terminate DateTimeNow SetEnvironmentVariable GetEnvironmentVariable CopyFileOrFolder DeleteFileOrFolder WriteToFile ReadFromFile GetFilesFromFolder ExtractFile CreateZip

Ways to create desktop actions

You can create on-screen desktop actions in the following ways:

Auto-capture screens and steps

With Action recorder, you can record your interactions with desktop applications to create automated workflows. It records every step you take, including clicks, keystrokes, and data entry, along with visual and contextual information. By recording steps, you can automate tasks that replicate your interactions. You can save the recorded screens and steps as a reusable desktop action. For more information, see Action recorder in AI Desktop Actions.

Manually capture screens and steps

You can manually take a screen capture, add anchors and steps to automate a series of steps you perform on your computer, such as clicking buttons, typing text, or interacting with different applications, and then save this sequence as a reusable desktop action.

Benefits of desktop actions

• Eliminate manual effort by automating repetitive tasks for frequent desktop activities.
• Execute complex sequences of steps quickly and consistently.
• Connect desktop application processes with your broader ServiceNow automations.
• Standardize task execution to minimize human error.
• You can record steps directly on your desktop applications.
• Automate tasks that span across multiple desktop applications.
• Save recorded sequences as desktop actions to use them in various automations.
• Review, rearrange, edit, or delete individual steps within a recorded action.

How the Design workspace works

Record steps

Capture your interactions with desktop applications, including clicks, text input, and screen navigation.

Define steps

Specify individual steps like Click, Set Text, or Get Text to interact with elements on your screen.

Application details

Select and store the desktop applications involved in your automations in the Details tab.

Organize screens

Structure your recorded steps by application screens, making it easy to review and edit.

Save and publish

Store your desktop actions in your ServiceNow instance for reuse and deployment.

What you can automate

Desktop actions support a wide range of desktop application interactions, including:

• Entering data into forms.
• Extracting information from applications.
• Navigating through application menus and screens.
• Performing steps like saving files or generating reports.
• Transferring data between different desktop applications.