--- sourceDocument: Yokohama IT Operations Management sourceDocumentLink: https://servicenow-prod.fluidtopics.net/r/yokohama/it-operations-management Release : - yokohama ft:locale : - en-US ft:publication_title : - Yokohama IT Operations Management ft:clusterId : - itom bundleId : - itom workflow : - Technology --- # Map raw log data # Map raw log data {#ariaid-title1} * Release version: Yokohama * * Updated January 30, 2025 * * ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 6 minutes to read Mapping raw log data that streams into your instance determines how the data is handled. Health Log Analytics automatically structures logs, creates metrics for anomaly detection, and presents alerts based on how your data is tagged. ## Before you begin Role required: evt_mgmt_admin ## About this task By default, Health Log Analytics tries to [auto-map](https://servicenow-prod.fluidtopics.net/mf0bznHQxZJFB0M_~kSg~A "By default, the Health Log Analytics AI engine tries to auto-map every incoming log line to the correct tags. You can change automatic mapping results manually by defining a JavaScript function.") every incoming log line to the correct tag. If properties aren't discovered automatically, map the data input sources manually by defining a JavaScript function. In the JavaScript function, you are required to map only the service instance (here called application service). Mapping the component and the source type is optional: Health Log Analytics tries to extract their values from the log data automatically. If the attempt fails, it assigns the default values. If you map the component but not the source type or vice versa, the system tries to extract the missing value from the log data. If it fails, it assigns the component value to the source type or vice versa, depending on which one you have mapped. This feature is supported in the Health Log Analytics application, Version 20.0.11 - July 2021, available from the [ServiceNow Store](https://store.servicenow.com/sn_appstore_store.do#!/store/home). Note: (ACC data inputs only) When the Agent Client Collector switches to a different MID Server to provide failover protection, it must switch to a different ACC data input. Therefore, all ACC data inputs must have the same JavaScript function. Health Log Analytics provides the latest published JavaScript function to all existing and future ACC data inputs, replacing the previous script. This feature is supported in the Health Log Analytics application, Version 22.0.12 - December 2021 and later, available from the [ServiceNow Store](https://store.servicenow.com/sn_appstore_store.do#!/store/home). For information about failover protection in Agent Client Collector Log Analytics (ACC-L), see [Agent Client Collector Log Analytics](https://servicenow-prod.fluidtopics.net/SuAKpX6~2tbaSzdG4PW6ng "Agent Client Collector Log Analytics (ACC-L) enables you to stream log data from Linux and Windows hosts to a ServiceNow instance, using the Agent Client Collector."). Note: You can manipulate raw log data before Health Log Analytics maps and structures it. For more information, see [Edit raw log data before processing](https://servicenow-prod.fluidtopics.net/R8YQ8bBaQwl6UzzSdCocKA "You can modify raw log data and drop or break up log messages before they are processed in the MID Server, and therefore before Health Log Analytics maps and structures it. For example, you could prevent sensitive data from reaching the system by replacing user names and passwords with an asterisk (*)."). ## Procedure 1. Navigate to AllHealth Log AnalyticsMappingData Input Mapping. 2. Open a record. Note: The first time the Data Input Mapping form is displayed, Health Log Analytics fetches log samples automatically. In subsequent sessions, fetch fresh samples by selecting Refresh Samples. Note: If the HLA engine is down and data has stopped streaming, a notification appears at the top of the Data Input Mapping page. When this happens, contact ServiceNow support. 3. **Optional:** To forward the complete raw log message, disable automatic header properties detection for this data input by selecting Disable header detection. For more information, see [Header properties detection](https://servicenow-prod.fluidtopics.net/LzAM_2YKrYJTqSsR3WZD4w "In Health Log Analytics, automatic header properties detection separates the transport header from the inner log message and forwards only the inner log message to the source type structure. The inner message contains the actual log data without including shipping information."). 4. **Optional:** Activate Test mode by setting the Test mode value to ON. In Test mode, Health Log Analytics doesn't create the source types, sources, or any other objects it creates in the standard flow to avoid blowing up Elasticsearch storage with sample data. For more information, see [Log data auto-mapping and mapping](https://servicenow-prod.fluidtopics.net/mf0bznHQxZJFB0M_~kSg~A "By default, the Health Log Analytics AI engine tries to auto-map every incoming log line to the correct tags. You can change automatic mapping results manually by defining a JavaScript function."). 5. **Optional:** View how the current JavaScript function affects log lines. 1. Add a sample message in the Test manual sample field. 2. Select Go. 3. Note how the JavaScript function affects the log lines. {#hla-data-input-mapping__substeps_lpl_xbt_f4b} 6. In the Raw input sample field, choose a log sample that will show the effect of your new JavaScript function on log lines when you test it. 7. **Optional:** Enable HLA to identify service instances and components from log fields and automatically map log data to them by selecting Enable auto map from log field. Note: * Automatic log context mapping can only be configured in the Service Operations Workspace. For more information about this feature, see [Map logs to service instances, components, source types](https://servicenow-prod.fluidtopics.net/l2KweYyeD6_33LgGJUQMFQ "Map your logs to service instances, components, and source types so that Health Log Analytics (HLA) can generate contextual alerts."). * If a manually-defined mapping script exists, automatic log context mapping overwrites it. * To be able to define a JavaScript function manually in the current form, the Enable auto map from log field check box must be clear. {#hla-data-input-mapping__ul_uyq_g33_hhc} 8. Define a JavaScript function that maps your data input sources to the correct service instance (here called application service), component, and source type. Note: (ACC data inputs only) Make sure that your JavaScript function can be used to handle data streamed by all ACC data inputs. 1. In the JavaScript console, either modify the provided default JavaScript function or a custom JavaScript function selected from the JS functions templates drop-down list, or define a new JavaScript function. The provided templates can act as a starting point for your custom script code. Available templates are: * Default Mapping Script * Glide Node Log Mapping Script * Glide Sys Log Mapping Script * Linux OS logs streamed with Syslog Note: Logs must have a syslog wrapper for this JavaScript function to work properly. * Mid Server Log Mapping Script * Windows - Winlogbeat logs streamed with Winlogbeat {#hla-data-input-mapping__ul_elx_lq3_hhc} The JavaScript function for mapping raw log data uses the following objects: * Signature: function map(sample, metadata) {#hla-data-input-mapping__table_k1s_br1_24b__entry__2} | Object | Description | |-|-| | sample | The current log sample after preprocessing. | | metadata | Object that contains: * Event Stream: Access via: \. For example: metadata.eventStream.origin * Transform header 1: Access via: \. For example: metadata.headers.i1.type * Transform header 2: Access via: \. For example: metadata.headers.i2.type * type Transform header 3: Access via: \. For example: metadata.headers.i3.type * Fallback Assignment: Access via: \. For example: metadata.fallBacks.host {#hla-data-input-mapping__ul_ufg_d2r_vnb} | [ ] {#hla-data-input-mapping__table_k1s_br1_24b} Note: You can create and associate multiple services per source type. * Return type and structure Note: The JavaScript function returns a map of two entries. Do not change this return structure. {#hla-data-input-mapping__table_z3g_fs1_24b__entry__2} | Object | Description | |-|-| | applicationService | The existing application service (that is, the service instance) to which this sample will be assigned. Note: The service instance must be bound to anomaly alerts related to the log. | | component | The component to which this sample will be assigned. | | sourceType | The source type to which this sample will be assigned. | [ ] {#hla-data-input-mapping__table_z3g_fs1_24b} * To discard a log message, call `return drop()`. {#hla-data-input-mapping__ul_c4p_1fl_vnb} 2. Test the JavaScript function by selecting Test. Testing the JavaScript function enables you to view the outcome of the script on the log sample. For a description of the displayed fields, see [JavaScript function test outcome fields](https://servicenow-prod.fluidtopics.net/~ts4zubl86YEdXM~rP4oHQ "Description of the JavaScript function test outcome fields."). Note: If your new JavaScript function is not behaving as expected, you can revert to the last published one by selecting the Revert JS Function related link. 3. **Optional:** Make any necessary adjustments and then test the JavaScript function again. 4. **Optional:** Compare the outcome of multiple tests. Comparing the test results of several versions of the JavaScript function can help you refine the script until it achieves the desired outcome. {#hla-data-input-mapping__substeps_yyw_nxs_mmb} 9. When you have finalized the JavaScript function, select Save template to save it. You can either save the JavaScript function as a new template or overwrite the currently selected template. * To save the JavaScript function as a new template, enter a new name in the Template name field. * To overwrite the template currently selected in the JS function templates field, leave the Template name field empty. {#hla-data-input-mapping__ul_zr2_3jt_f4b} 10. Select Publish to save the JavaScript function to the database. ## Result When the JavaScript function is published, Health Log Analytics uses it to map data input sources. The new script is automatically added to the list of JS function templates from which you can choose. This feature is supported in the Health Log Analytics application, Version 20.0.11 - July 2021, available from the [ServiceNow Store](https://store.servicenow.com/sn_appstore_store.do#!/store/home). (ACC data inputs only) Health Log Analytics provides the published JavaScript function to all existing and future ACC data inputs. The new JavaScript function replaces the previous script. ## What to do next * (Optional) [Edit your raw log data](https://servicenow-prod.fluidtopics.net/R8YQ8bBaQwl6UzzSdCocKA "You can modify raw log data and drop or break up log messages before they are processed in the MID Server, and therefore before Health Log Analytics maps and structures it. For example, you could prevent sensitive data from reaching the system by replacing user names and passwords with an asterisk (*).") before Health Log Analytics maps and structures it. If you want to perform this task immediately, click the Go to Preprocessor related link to proceed to the Data Input Preprocessor page. * (Optional) Fine-tune how the system reads your log data by [refining the Source Type structure](https://servicenow-prod.fluidtopics.net/7xffVJXJtf_JlE08nlOH4w "Fine-tune how Health Log Analytics reads your inner log messages and detects anomalies by customizing the extracted properties in the source type structure."). This step enables you to reclassify auto-classified properties and change auto-mapped labels. {#hla-data-input-mapping__ul_p2k_bds_fnb} **Related concepts** * [Log data auto-mapping and mapping](https://servicenow-prod.fluidtopics.net/mf0bznHQxZJFB0M_~kSg~A "By default, the Health Log Analytics AI engine tries to auto-map every incoming log line to the correct tags. You can change automatic mapping results manually by defining a JavaScript function.") * [Header properties detection](https://servicenow-prod.fluidtopics.net/LzAM_2YKrYJTqSsR3WZD4w "In Health Log Analytics, automatic header properties detection separates the transport header from the inner log message and forwards only the inner log message to the source type structure. The inner message contains the actual log data without including shipping information.") **Related tasks** * [Source type and log source relationships](https://servicenow-prod.fluidtopics.net/TXzUuwJ2Sgk1DBg6~jT8QA "Explore the many-to-many relationships between source types and log sources to help you optimize data input mapping in Health Log Analytics.") * [Stop extraction of unneeded log data](https://servicenow-prod.fluidtopics.net/N_rY673APoTEWMVW68Toqg "If an extracted string of data is not descriptive enough or contains redundant text or information, you can set Health Log Analytics to stop extracting such data from your logs.") * [Ensure extraction of specific log data](https://servicenow-prod.fluidtopics.net/kbW5ugyUh7Lu_7W9ed7uow "Set Health Log Analytics to extract specified terms from fields and map them to specific components.") *[\>]: and then