--- sourceDocument: Australia ServiceNow AI Platform Administration sourceDocumentLink: https://servicenow-prod.fluidtopics.net/r/platform-administration Release : - australia ft:locale : - en-US ft:publication_title : - Australia ServiceNow AI Platform Administration ft:clusterId : - platadm bundleId : - platadm workflow : - Platform --- # Configure crawl settings # Configure crawl settings for an Amazon S3 external content connector {#ariaid-title1} Release version: Australia Updated May 15, 2026 ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 4 minutes to read Define inclusion and exclusion filters to specify the buckets and file types you want your Amazon S3 external content connector to retrieve when running content crawls. ## Before you begin A connector administrator must have already created the Amazon S3 external content connector that you want to configure crawl settings for. To learn about this procedure, see [Create an Amazon S3 external content connector](https://servicenow-prod.fluidtopics.net/Y2yL_3DRKj9RTJAFEpBFWA "Create an external content connector to retrieve searchable content from your Amazon S3 source system."). Role required: sn_ext_conn.xcc_admin ## About this task This task is optional. By default, the Amazon S3 external content connector crawls all buckets and file types from its specified source system and sends files with all supported extensions to AI Search for indexing. Only perform this task if you want the connector to use any of the following non-default settings: * Inclusion or exclusion filters for the buckets to crawl when running content crawls * Inclusion or exclusion filters for the file extensions to retrieve when running content crawls {#configure-crawl-settings-amazon-s3-external-content-connector__ul_onp_x1f_tdc} Content is only retrieved from the source system if it passes all of your configured crawl setting filters. If any crawl setting filter excludes a content item, the external content connector doesn't retrieve it. Important: By default, each external content connector can index up to one million (1,000,000) content items from its source system. When a connector exceeds this limit, it continues to crawl the source system, but only sends content item deletions and updates to AI Search for indexing, ignoring new content items. The connector logs an error message for every 10,000 content items it crawls beyond the indexing limit.{#configure-crawl-settings-amazon-s3-external-content-connector__store-app-external-content-connectors-indexing-limit-numeric-ph} When a connector's indexed content item count exceeds 800,000, a warning message appears in the connector's UI to indicate that it's approaching the indexing limit. If the connector reaches the indexing limit, an error message appears in its UI. External content connectors that support user permissions crawls can handle permissions for up to five hundred thousand (500,000) users and their groups. If a connector retrieves users in excess of this limit, user and group permissions may not be correctly applied to the connector's retrieved content. As a result, the content may not be searchable. If one of your connectors reaches the content indexing limit, you can update its crawl settings and file inclusion/exclusion filters to reduce the number of content items it retrieves. Alternatively, if you need a connector to index more than 1,000,000 content items, you can create a Customer Service and Support case at to request a limit increase for the connector. ## Procedure 1. Navigate to AllExternal Content ConnectorsExternal Content Admin Home. {#configure-crawl-settings-amazon-s3-external-content-connector__navigate-ext-content-connectors-admin-home-step} {#configure-crawl-settings-amazon-s3-external-content-connector__navigate-ext-content-connectors-admin-home-step} 2. In the Connectors list, select the record for the Amazon S3 external content connector whose settings you want to modify. 3. In the connector editor's Settings tab, select Crawl settings. {#configure-crawl-settings-amazon-s3-external-content-connector__select-crawl-settings-step} {#configure-crawl-settings-amazon-s3-external-content-connector__select-crawl-settings-step} 4. Select one of the following Content filtering options: * To crawl all buckets from the source system, select Crawl all content. * To crawl only a specified set of buckets from the source system, select Include only these buckets, then use the Add S3 bucket URLs to include field and Add button to enter URLs for buckets you want the connector to include when crawling. As an example, you might enter https://s3.amazonaws.com/published to only include searchable content from a bucket with name published. * To crawl all but a specified set of buckets from the source system, select Exclude only these user boxes, then use the Add S3 bucket URLs to exclude field and Add button to enter URLs for buckets you want the connector to exclude when crawling. As an example, you might enter https://s3.amazonaws.com/legacy_bucket to exclude searchable content from a bucket with name legacy_bucket. {#configure-crawl-settings-amazon-s3-external-content-connector__choices_cdr_s4x_sdc} 5. Select one of the following Filetype filtering options: * To retrieve all attachments with supported file extensions from the source system, select Crawl all attachments. * To retrieve only attachments with specified file extensions from the source system, select Include only these file extensions, then use the File extensions to include field to enter attachment file extensions you want the connector to include when crawling. As an example, you might enter .docx to retrieve only attachments with the Microsoft Word file format. * To retrieve all attachments except those with specified file extensions from the source system, select Exclude only these file extensions, then use the File extensions to exclude field to enter attachment file extensions you want the connector to exclude when crawling. As an example, you might enter .csv to exclude attachments with the Comma-Separated Values (CSV) file format. {#configure-crawl-settings-amazon-s3-external-content-connector__choices_p3p_g11_5fc} For details on the supported attachment file extensions, see [Binary file extensions supported in External Content Connectors](https://servicenow-prod.fluidtopics.net/wyAI934z2WbwQ_js9LjRqA "Connector administrators can restrict the binary file types an external content connector retrieves by specifying file extensions in inclusion or exclusion filters."). 6. **Optional:** If you want AI Search to automatically generate captions for content in attachments and files retrieved by the connector, select the Multimodal captions option. When you select this option, the Platform Multimodal Service automatically generates descriptive captions for images, tables, charts, and other visual elements found in retrieved attachments and files. You can discover these retrieved attachments and files by searching for terms from their generated captions. This option is only available when the Platform Multimodal Service plugin is activated on your instance. * For details on activating the plugin, see [Activate the Platform Multimodal Service plugin](https://servicenow-prod.fluidtopics.net/Aps2A3ZsA_gtaZTILcqu2w "Enable automatic generation of searchable descriptive captions for images, tables, charts, and other visual elements found in indexed attachments."). * To learn how to select the VLM (visual learning model) provider and model used for the Platform Multimodal Service, see [Configure multimodal captioning for AI Search](https://servicenow-prod.fluidtopics.net/nE2_exg8skwGj6J7f2cuwQ "Use the AI Search Admin console to select the visual language model (VLM) provider and model for multimodal captioning."). 7. Select Save and validate. {#configure-crawl-settings-amazon-s3-external-content-connector__steps_yqk_jjw_22c} ## Result The Amazon S3 external content connector is updated with your modified crawl settings. Important: All content the connector retrieves from your Amazon S3 buckets is treated as public content, searchable by everyone who has access to your configured AI Search experience. ## What to do next To retrieve content from your Amazon S3 source system using your modified crawl settings, create and run a one-time content crawl for your Amazon S3 external content connector. To learn about creating and running one-time content crawls, see [Create a content crawl for an external content connector](https://servicenow-prod.fluidtopics.net/fFJ92ZYif96fkc2JA4HtyQ "Retrieve searchable content and metadata from your source system with a content crawl. Run the crawl as a one-time task or schedule it to run on a recurring basis."). *[\>]: and then