Managing CMDB data deletion
Summarize
Summary of Managing CMDB data deletion
The Integration Commons for CMDB (sncmdbintutil) store app provides methods to manage deletion of Configuration Management Database (CMDB) records that have been removed from third-party source applications. This ensures that the CMDB stays synchronized with source data by removing outdated or deleted configuration items (CIs).
Show less
Post-script method
This method checks the Source [sysobjectsource] table for CIs not updated in the latest run of a specific source feed and removes these CIs synchronously from the target CMDB tables, processing one record at a time. It is commonly used by most Service Graph Connectors for record removal.
The post-script method utilizes the RecordRemoval script to support different deletion strategies:
- Hard deletion: Permanently removes records from the CMDB, making them unrecoverable, suitable for data no longer needed.
- Soft deletion: Marks records as deleted without physical removal, preserving data for potential recovery and historical tracking.
- Optimized soft deletion: Managed by the Identification and Reconciliation Engine (IRE) using specialized extension attribute tables and reconciliation rules to efficiently handle deleted records.
Administrators can control the number of records handled in optimized soft deletion by creating and configuring the sncmdbintutil.irepayloaditemssize system property (integer type) within the Integration Commons for CMDB app. This property defaults to 100 records per batch but can be adjusted to tune performance.
Transform map-based method
This method also checks the Source [sysobjectsource] table for CIs not updated in the last run but performs deletions concurrently using a multi-threaded approach, improving processing time especially for large data sets.
The method uses the Integration Commons Remove Record [sncmdbintutilremoverecord] staging table to track non-updated CIs by storing details such as discovery source, source feed, target table sysid, and target table name. Multiple import sets store the target record data, enabling parallel deletion of all records in an import set.
The on-start script in the Remove Record Transform Map executes the deletion of records from the target tables.
This transform map-based deletion is adopted by Service Graph Connectors focused on efficiency, such as the SG-Tanium Remove Software data source within the Service Graph Connector for Tanium.
The Integration Commons for CMDB (sn_cmdb_int_util) store app uses post-script and transform map methods to remove imported Configuration Management Database (CMDB) records that were deleted later from the source third-party application.
Post-script method
The post-script method checks the Source [sys_object_source] table for configuration items (CIs) not updated in the last run for a specific source feed. It then removes any non-updated CIs from the associated target CMDB tables synchronously, handling the removal of target records one at a time.
Most Service Graph Connectors use the post-script method to manage the removal of any CMDB records.
The post-script method uses the RecordRemoval script to handle various record deletion strategies. Hard deletion involves permanently and completely removing records from the CMDB, making them unrecoverable and suitable for data that is no longer needed. Soft deletion marks records as deleted without actually removing them from the CMDB, allowing for potential recovery and preserving historical data. Optimized soft deletion, managed by the Identification and Reconciliation Engine (IRE), enhances this soft deletion approach by using a specialized table, such as the extension attributes table, and applying reconciliation rules to manage deleted records more efficiently. By default, the table can hold up to 100 records at a time, but as a user with the admin role, you can adjust this limit by setting the sn_cmdb_int_util.ire_payload_items_size system property. This property is not available by default and you must create this property of type integer within the Integration Commons for CMDB (sn_cmdb_int_util) store app. For more information, see Add a system property.
Transform map-based method
The transform map-based method checks the Source [sys_object_source] table for configuration items (CIs) not updated in the last run for a specific source feed. It then utilizes the multi-threaded deletion approach to remove any non-updated CIs from the associated target CMDB tables concurrently, handling the removal of all target records within an import set at once. Multiple import sets are created to store target record data within the Integration Commons Remove Record [sn_cmdb_int_util_remove_record] staging table enabling multi-threaded deletion of records. Therefore, reducing the data processing time.
The Integration Commons Remove Record [sn_cmdb_int_util_remove_record] staging table tracks non-updated CIs by storing target record details including the discovery source, source feed, target table sys_id, and target table name of these CIs. The on-start script defined within the Remove Record Transform Map then deletes the records from the associated target tables.
The transform map-based method is used by a Service Graph Connector to improve the data processing time. For example, the SG-Tanium Remove Software data source within the Service Graph Connector for Tanium removes software data using the transform map-based method. See Service Graph Connector for Tanium.