urlscan.io Enrichment

Overview

The urlscan.io built-in enrichment in ThreatConnect® allows you to use the extensive URL scanning capabilities of urlscan.io directly within ThreatConnect, providing you with contextual information about a URL, its IP address, and the associated ASN that can assist in assessing the reputation, trustworthiness, and potential security implications of a URL and the hosting network to which it belongs. 

This article describes how to enable and configure the urlscan.io built-in enrichment, view data from the enrichment on the Enrichment tab of a URL Indicator’s Details screen, and import Indicators from urlscan.io into ThreatConnect.

User Roles

• To enable and configure the urlscan.io enrichment, your user account must have a System role of Administrator.
• To view urlscan.io data on the Enrichment tab of an Indicator’s Details screen, your user account can have any Organization role.
• To retrieve data manually on the URLScan card on the Enrichment tab of an Indicator’s Details screen, your user account can have any Organization role.

Enabling the urlscan.io Enrichment

Follow these steps to enable and configure the urlscan.io enrichment on your ThreatConnect instance:

1. Hover over Settingson the top navigation bar and select System Settings.
2. Select the Indicators tab on the System Settings screen and then click Enrichment Tools in the menu on the left side of the screen.
3. Click Editin the Options column for URLScan and fill out the fields on the Edit Vendor window (Figure 1) as follows:
   

    

   • Enable Vendor: Select this checkbox to enable urlscan.io.
   • Enable Automatic Retrieval: Select this checkbox to enable automatic data retrieval for urlscan.io. If automatic data retrieval is enabled, urlscan.io data will automatically populate when a user clicks on a URL Indicator’s Enrichment tab for the first time. This checkbox is selected by default.
   • API Key: Enter the API key that will be used to retrieve data from urlscan.io.
     Hint

     You can create a free urlscan.io API user account at https://urlscan.io/user/signup/.
   • VALIDATE: After entering the urlscan.io API key, click this button to validate it. If the API key is accepted, the VALIDATE button’s label will change to VALID.
   • Query Visibility: Select the visibility level for queries sent to urlscan.io. Available choices include Private, Public, and Unlisted.
   • Tag Settings: Select the Automatically import URLScan tags checkbox to import tags applied to a URL in urlscan.io into ThreatConnect as Tag objects and apply them to the corresponding URL Indicator being imported into ThreatConnect.
     Note

     If the Automatically import URLScan tags checkbox is selected and a tag applied to a URL in urlscan.io exactly matches an ATT&CK^® Tag in ThreatConnect, then that ATT&CK Tag will be applied to the URL Indicator in ThreatConnect; otherwise, a new standard Tag will be created and applied to the URL Indicator in ThreatConnect.
   • Lookup/Retrieve: Select the Indicator type(s) for which to retrieve data from urlscan.io. The only available Indicator type is URL.
   • Click the SAVE button to save the configuration for urlscan.io.

When urlscan.io is enabled, a value of true will be displayed in the Enabled column for its entry on the Enrichment Tools screen.

Data Overview

The Overview section of the URLScan card (Figure 2) provides a summary of data retrieved from urlscan.io for a URL Indicator and the date and time the data were last retrieved.

• Detection: This field specifies whether the scanned URL is malicious.
• Domain: The primary domain or subdomain contacted by the URL.
• IP: The primary IP address contacted by the URL.
• Submitted URL: The original URL that was submitted to urlscan.io.
• Effective URL: The URL of the primary webpage after redirection.
• Country: The country associated with the primary IP address contacted by the URL.
• ASN name: The primary autonomous system (AS) number and name contacted by the URL.
• Website Contacted: The number of unique IP addresses, countries, and domains or subdomains the URL contacted in order to perform the specified number of HTTP transactions.
• Tags: The tag(s) applied to the URL in urlscan.io.
• Targeted Brands/Vertical: The name and industry vertical of the brand(s) targeted by the URL.

Viewing Results on urlscan.io

To view the results for the URL Indicator on the urlscan.io website, click the View Results on URLScan.io link at the upper right of the URLScan card (Figure 2).

Viewing Screenshots of URLs

To view a PNG screenshot of the webpage associated with the URL Indicator, click the View Screenshot URL link at the bottom of the URLScan card (Figure 2).

URLScan Detailed View

Click the Open Detailed View link on the URLScan card to display the URLScan Detailed View drawer (Figure 3). This drawer displays cards with additional data retrieved from urlscan.io. The cards are collapsed by default. Figure 3 shows the URLScan Detailed View drawer with all available cards expanded.

See Table 1 for a list of cards that may be displayed on the URLScan Detailed View drawer for a URL Indicator.

Importing Indicators From urlscan.io Into ThreatConnect

You may import some or all of the Indicators displayed on the IP and ASN Details and Links to Domains cards into ThreatConnect and associate them to a new or existing Group. You may also import Indicators displayed on these cards into ThreatConnect and associate them directly to the enriched Indicator (i.e., the URL Indicator whose Details screen you are viewing) via a custom association.

Follow these steps to import Indicators from urlscan.io into ThreatConnect:

1. Expand one of the following cards on the URLScan Detailed Viewdrawer (Figure 3) to display a table containing Indicators retrieved from urlscan.io that are related to the enriched URL Indicator:
   • IP and ASN Details: The Indicators on this card will be imported as Address Indicators.
   • Links to Domains: The Indicators on this card will be imported as Host Indicators.
2. Select the checkbox for each Indicator to import into ThreatConnect, or select the checkbox in the table’s header to import all Indicators displayed on the current page in the table.
   Important

   If a selected Indicator already exists in the ThreatConnect owner into which you are importing data, that copy of the Indicator will be updated based on the information entered and options configured during the import.
3. Expand the Import dropdown at the top left of the card and select one of the following import options (Figure 4).

   

    

   • To New Group: Select this option to import the selected Indicators and associate them to a new Group created during the import.
   • To Existing Group: Select this option to import the selected Indicators and associate them to an existing Group.
   • As an Indicator: Select this option to import the selected Indicators and associate them directly to the enriched Indicator via a custom association.

Importing Indicators Into a New Group

Follow these steps to import Indicators from urlscan.io and associate them to a new Group created during the import:

1. Follow Steps 1–3 in the “Importing Indicators From urlscan.io Into ThreatConnect” section and select To New Group from the Import dropdown.
2. Proceed through the steps on the Create screen to create the Group. There are three steps in the Group creation process: Details (required), Associations (optional), and Attachments (optional).

Step 1: Enter Details About the Group

The Details step of the Create screen (Figure 5) is a required step where you enter basic information about the Group you are creating. 

Follow these steps to fill out the fields on the Details step:

1. Provide the following details for the Group:
   • Type: By default, Event is selected. If desired, select another Group type from the dropdown.
   • Fill out all remaining fields on the Details section.
     Note

     By default, a UrlScan Enrichment Tag will be applied in the Tags field. You can remove this Tag if desired.
     Important

     If you select the Apply Tags to Associations checkbox, it is recommended that you remove the UrlScan Enrichment Tag so that this Tag does not get applied to the enriched Indicator (that is, the Indicator whose Enrichment tab from which you are importing urlscan.io data), as the enriched Indicator will be added as an association to the new Group. Alternatively, if you want the UrlScan Enrichment Tag to be applied to all associations except the enriched Indicator, you can select the Apply Tags to Associations checkbox, retain the UrlScan Enrichment Tag, and then, after completing the import, navigate to the Overview tab of the Details screen for the enriched Indicator and remove the Tag manually.
2. Click Next to proceed to the Associations step.
   Note

   The Save button is available only on the Associations and Attachments steps.

Step 2: Configure Associations for the Group (Optional)

The Associations step of the Create screen (Figure 6) is an optional step where you configure the Indicators from urlscan.io that are being created and associated to the new Group.

Follow these steps to configure the fields on the Associations step:

1. Associations: Review the table containing the Indicators that will be created and associated to the Group. This table includes all selected Indicators and the enriched URL Indicator and its corresponding Host Indicator. To remove an Indicator from the table, click Deletein the Actions column.
   Note

   The table in the Associations card will include a Private column if your System Administrator turned on private Indicators for your ThreatConnect instance. To mark an Indicator as private, select the corresponding checkbox in the Private column.
   Note

   A checkmark in the Known column indicates that the corresponding Indicator exists in the owner in which you are creating the Group and Indicators.
2. Association Details: On the Associations Details card, provide the following details for all Indicators that will be created and associated to the Group:
   Important

   All information added in this section will be applied to the enriched Indicator (that is, the Indicator whose Enrichment tab from which you are importing urlscan.io data), because the enriched Indicator is always added as an association to the new Group, along with the Indicators selected on the card on the URLScan Detailed View (Figure 4). If the enriched Indicator has a default Description, a Threat Rating, or a Confidence Rating and you enter a value for one of these fields, then that value will replace the existing value for the enriched Indicator. Tags entered in this section will be applied in addition to the enriched Indicator’s existing Tags.
   • Description: Enter a default Description for the Indicators. If you entered a Description for the Group on the Details step and selected Apply Description to Associations, the text box will contain that Description.
   • Tags: Enter one or more Tags to apply to the Indicators. If you entered Tags for the Group on the Details step and selected Apply Tags to Associations, the text box will contain those Tags.
   • Threat Rating: Set the Threat Rating for the Indicators.
   • Confidence Rating: Set the Confidence Rating for the Indicators.
3. Click Next to proceed to the optional Attachments step, or click Save to create the Group and Indicators.

Step 3: Upload File Attachments to the Group (Optional)

The Attachments step of the Create screen (Figure 7) is an optional step where you can upload and attach related files to the Group.

Follow these steps to configure the fields on the Attachments step:

1. Upload files for which Document Groups will be created and associated to the Group being created.
2. After each file is uploaded, the filename will be displayed below the upload area, along with a checkbox labeled Add to Malware Vault. Leave this checkbox cleared unless you are uploading a malware file.
3. Click Save to create the Group and associated Indicators.

The Indicators will be imported into ThreatConnect and associated to the newly created Group, and the Group’s Details screen will be displayed. You can view the Indicators on the Indicator Associations card of the Group’s Associations tab.

Importing Indicators Into an Existing Group

Follow these steps to import Indicators from urlscan.io and associate them to an existing Group:

1. Follow Steps 1–3 in the “Importing Indicators From urlscan.io Into ThreatConnect” section and select To Existing Group from the Import dropdown. 
2. Proceed through the steps on the Import to Existing Group screen to select a Group and configure its associations. There are two steps in this process: Select Group (required) and Associations (optional).

Step 1: Select Group

The Select Group step of the Import to Existing Group screen (Figure 8) is a required step where you select the Group to which to associate the imported Indicators.

Follow these steps to select a Group to which to associate the imported Indicators:

1. Select a Group to which the selected Indicators, as well as the enriched Indicator and its corresponding Host Indicator, will be associated. To search for a Group, enter its name in the search bar above the table containing all Groups.
2. Click Next to proceed to the Associations step.
   Note

   The Save button is available only on the Associations step.

Step 2: Configure Associations for the Group (Optional)

The Associations step of the Import to Existing Group screen (Figure 9) is an optional step where you configure the Indicators from urlscan.io that are being created and associated to the existing Group.

Follow these steps to configure the fields on the Associations step:

1. Associations: Review the table containing the Indicators that will be created and associated to the Group. This table includes all selected Indicators and the enriched URL Indicator and its corresponding Host Indicator. To remove an Indicator from the table, click Deletein the Actions column.
   Note

   The table in the Associations card will include a Private column if your System Administrator turned on private Indicators for your ThreatConnect instance. To mark an Indicator as private, select the corresponding checkbox in the Private column.
   Note

   A checkmark in the Known column indicates that the corresponding Indicator exists in the owner in which you are creating the Group and Indicators.
2. Association Details: On the Associations Details card, provide the following details for all Indicators that will be created and associated to the Group:
   Important

   All information added in this section will be applied to the enriched Indicator (that is, the Indicator whose Enrichment tab from which you are importing urlscan.io data), because the enriched Indicator is always added as an association to the existing Group, along with the Indicators selected on the card on the URLScan Detailed View (Figure 4). If the enriched Indicator has a default Description, a Threat Rating, or a Confidence Rating and you enter a value for one of these fields, then that value will replace the existing value for the enriched Indicator. Tags entered in this section will be applied in addition to the enriched Indicator’s existing Tags.
   • Description: Enter a default Description for the Indicators.
   • Tags: Enter one or more Tags to apply applied to the Indicators.
   • Threat Rating: Set the Threat Rating for the Indicators.
   • Confidence Rating: Set the Confidence Rating for the Indicators.
3. Click Save to create the Indicators and associate them to the existing Group.

The Indicators will be imported into ThreatConnect and associated to the existing Group, and the Group’s Details screen will be displayed. You can view the Indicators on the Indicator Associations card of the Group’s Associations tab.

Importing Indicators as Indicators

Follow these steps to import Indicators from urlscan.io and associate them directly to the enriched Indicator via a custom association:

1. Follow Steps 1–3 in the “Importing Indicators From urlscan.io Into ThreatConnect” section and select As an Indicator from the Import dropdown. 
2. Fill out the fields on the Import Indicators window (Figure 10) as follows: 

    

   • New Indicators to Be Imported & Associated: This section displays the Indicators that will be imported into ThreatConnect and associated directly to the enriched Indicator. To remove an Indicator from this list, click Delete.
3. Click Import Indicators to import the Indicators and associate them directly to the enriched Indicator via a custom association.

The Indicators will be imported into ThreatConnect and associated to the enriched Indicator, and the Associations tab of the enriched Indicator’s Details screen will be displayed. You can view the Indicators on the Indicator Associations card of this tab.

Retrieving Data Manually

When you click on a URL Indicator’s Enrichment tab for the first time, data will be retrieved from urlscan.io automatically if your System Administrator has enabled automatic data retrieval for urlscan.io. Otherwise, a message stating that “Automatic Data Retrieval has been disabled by the System Administrator” will be displayed on the card, and you will need to click the Retrieve Data button to populate the card with data. Once data have been retrieved, they will be cached for a period of time configured by your System Administrator. Each time you revisit that Indicator’s Enrichment tab, the cached urlscan.io data will be displayed until this period of time has passed.

To retrieve the latest urlscan.io data for the Indicator manually, click the Retrieve Data button on the URLScan card (Figure 2).

Enriching Indicators Using the ThreatConnect API

You can also use the ThreatConnect v3 API to enrich URL Indicators with data from urlscan.io. For instructions on using the ThreatConnect v3 API to enrich Indicators, see Indicator Enrichment Overview.

ThreatConnect^® is a registered trademark of ThreatConnect, Inc.
MITRE ATT&CK^® and ATT&CK^® are registered trademarks of The MITRE Corporation.

20146-05 v.04.A