Recorded Future Intelligence Engine Integration User Guide

Overview

The ThreatConnect® integration with Recorded Future® ingests Risk List entities (Domain, Hash, IP, URL, and Vulnerability), Threat Map entities (Malware and Actor), Alert entities, and Analyst Notes from Recorded Future. It then creates corresponding objects with select Recorded Future metadata in ThreatConnect.

Dependencies

ThreatConnect Dependencies

• Active ThreatConnect Application Programming Interface (API) key
• ThreatConnect instance with version 7.6.2 or newer installed

Recorded Future Dependencies

• Active Recorded Future API token
• Active Recorded Future module subscriptions:
  • Risk List: SecOps Intelligence, Threat Intelligence, and/or Vulnerability Intelligence (see the “Risk List Types” row in Table 1 for more information)
  • Threat Map: Threat Intelligence module
  • Standard Alerts: The required subscription varies by module. Please refer to the Recorded Future documentation (requires a login to view) for more information.

Initial Installation and Configuration

Follow these steps to install and configure version 2.0.x of the Recorded Future Intelligence Engine App on your ThreatConnect instance:

1. Log into ThreatConnect with a System Administrator account.
2. Install version 2.0.x of the Recorded Future Intelligence Engine App via TC Exchange™.
3. Use the ThreatConnect Feed Deployer to set up and configure the Recorded Future Intelligence Engine App, using the parameter definitions in Table 1 for guidance.

Upgrade Installation and Configuration

Follow these steps to upgrade to and configure version 2.0.x of the Recorded Future Intelligence Engine App on your ThreatConnect instance:

1. Log into ThreatConnect with a System Administrator account.
2. On the top navigation bar, hover over Playbooks and select Services.
3. Locate and turn off version 1.0.x of the Recorded Future Intelligence Engine Feed API Service.
4. Clickin the Options column and select Delete to delete version 1.0.x of the Recorded Future Intelligence Engine Feed API Service.
5. Install version 2.0.x of the Recorded Future Intelligence Engine App via TC Exchange™.
6. Use the ThreatConnect Feed Deployer to set up and configure the Recorded Future IntelligenceEngine App, using the parameter definitions in Table 1 for guidance.

Configuration Parameters

Parameter Definitions

The parameters defined in Table 1 apply to the configuration parameters available when using the Feed Deployer to configure the App.

Recorded Future Intelligence Engine UI

After successfully configuring and activating the Feed API Service, you can access the Recorded Future Intelligence Engine user interface (UI). This UI allows you to interact with and manage the Recorded Future integration.

Follow these steps to access the UI:

1. Log into ThreatConnect with a System Administrator account.
2. On the top navigation bar, hover over Playbooks and select Services.
3. Locate and turn on the Recorded Future Intelligence Engine Feed API Service.
4. Click the link in the Service’s API Path field. The Recorded Future Intelligence Engine UI will open in a new browser tab.

The following screens are available in the Recorded Future Intelligence Engine UI:

• Dashboard
• Jobs
• Tasks
• Download
• Batch Errors
• Attachment Status

Dashboard

The Dashboard screen (Figure 1) provides an overview of the total number of Risk List entities (Domain, Hash, IP, URL, and Vulnerability), Threat Map entities (Malware and Intrusion Set), Alert entities (Event and Document) and Analyst Notes (Report, Email Address, Domain, Hash, IP, URL, Vulnerability, Malware, and Intrusion Set) retrieved from Recorded Future. Depending on the available data, cards representing all or a subset of these object types will be displayed on the Dashboard screen.

Jobs

The Jobs screen (Figure 2) breaks down the ingestion of Recorded Future data into manageable Job-like tasks, displaying all processes that are pending, in progress, complete, and failed. The ⋯ menu in a Job’s row provides the following options:

• Details: View details for the Job, such as counts of downloaded and batched Groups and start and end times for Alert monitoring, download, and upload.
• Download Files: Download metadata files for all Jobs and data (convert, download, and upload) files for completed Jobs.
• Batch Errors: View errors that have occurred for the Job on the Batch Errors screen.

You can filter Recorded Future Intelligence Engine App Jobs by the following elements:

• Job ID: Enter text into this box to search for a Job by its Job ID.
• Job Type: Select Job types to display on the Jobs screen. 
• Status: Select Job statuses to display on the Jobs screen.
• Pipeline: Select the pipeline types to display on the Jobsscreen:
  • alerts: Alert entities
  • analyst_note: Analyst Note entities
  • threat_intel: Risk List and Threat Map entities

Add a Job

You can add ad-hoc Jobs on the Jobs screen. Follow these steps to create a request for an ad-hoc Job for the  Recorded Future Intelligence Engine App:

1. Click the Add Job button at the upper right of the Jobs screen (Figure 2).
2. Fill out the fields on the Add Job drawer (Figure 3) as follows: 

    

   • Risklist Types: (Optional) Select the Risk List entity types to include in the ad-hoc Job.
   • Threat Map Types: (Optional) Select the Threat Map entity types to include in the ad-hoc Job.
   • Alert Types: (Optional) Select the Alert entity types to include in the ad-hoc Job.
   • Alert Start Time: (Optional) Enter the time at which monitoring for triggered Alerts should start.
     Note

     Alert Start Time applies only to Alert entities. If no value is specified, the 1,000 most recent Alerts will be downloaded.
   • Alert End Time: (Optional) Enter the time at which monitoring for triggered Alerts should end.
     Note

     Alert End Time applies only to Alert entities. If no value is specified, the 1,000 most recent Alerts will be downloaded.
3. Click Submit to submit the request for the ad-hoc Job.

Tasks

The Tasks screen (Figure 4) displays all Tasks that may be part of a Job, including each step of the download, convert, and upload processes, as well as Tasks for the Recorded Future Intelligence Engine App, such as Monitor, Scheduler, and Cleaner. The current status (Idle, Paused, or Running), name, description, and heartbeat timeout length, in minutes, are displayed for each Task. The ⋯ menu in a Task’s row provides the following options, depending on the Task’s status:

• Run (idle and paused Tasks only)
• Pause (idle and running Tasks only)
• Resume (paused Tasks only)
• Kill (running Tasks only)

Under the table is a dashboard where you can view runtime analytics.

Download

The Download screen (Figure 5) lets you download JavaScript® Object Notation (JSON) data for Recorded Future entities and then upload the data into ThreatConnect.

Follow these steps to download JSON data for a Recorded Future entity on the Download screen and then upload the data into ThreatConnect:

1. Recorded Future Type: Select a Recorded Future entity type from the following options on the Recorded Future Type dropdown: IPAddress (IP), URL, Hash, InternetDomainName (Domain), CyberVulnerability (Vulnerability), StandardAlert (Alert), Malware, and Actor.
2. External ID: Enter the ID for the Recorded Future entity of the selected type. For IP, Hash, Domain, and URL Risk List entities, prepend ip:, hash:, idn:, and url:, respectively, to the entity's ID. For Vulnerability Risk List entities, use the CVE ID or Recorded Future ID. The following examples demonstrate the ID format for each Risk List entity type:
   • IP: ip:124.71.84.65
   • Hash: hash:092910024190a2521f21658be849c4ac9ae6fa4d5f2ecd44c9055cc353a26875
   • Domain: idn:efavengh.com
   • URL: url:https://send.exploit.in/
   • Vulnerability: CVE-2019-0841or ZgFn9x
3. Click Download. The JSON data will be displayed in two columns: Results (raw JSON data) and Converted (JSON data in ThreatConnect batch format) (Figure 6).

    

4. Click Upload to submit the converted threat intelligence data via the ThreatConnect Batch API.

Batch Errors

The Batch Errors screen (Figure 7) displays a table with details on batch errors that have occurred for Job requests. You can filter the table by error type or enter keywords to filter by Job ID or reason for error.

Attachment Status

The Attachment Status screen (Figure 8) displays a table with details on ThreatConnect's attempts to download Report attachments from Recorded Future.  You can enter keywords to filter the table by the Recorded Future Group ID, which can be useful if you do not see a Recorded Future attachment in ThreatConnect as expected, or by status.

Data Mappings

The data mappings in Table 2 through Table 11 illustrate how data are mapped from Recorded Future Intelligence API endpoints into the ThreatConnect data model.

Domain

ThreatConnect object type: Host Indicator

Hash

ThreatConnect object type: File Indicator

IP

ThreatConnect object type: Address Indicator

URL

ThreatConnect object type: URL Indicator

Vulnerability

ThreatConnect object type: Vulnerability Group

Analyst Note

ThreatConnect object type: Report Group

Note Entity

ThreatConnect object type: Report Group

Actor

ThreatConnect object type: Intrusion Set Group

Malware

ThreatConnect object type: Malware Group

Standard Alerts

ThreatConnect object type: Event Group

Risk Score Mappings

ThreatConnect follows the Criticality mapping in Recorded Future when assigning a Threat Rating to data ingested from Recorded Future; however, because the Recorded Future Criticality rating goes only from 0–4, it has been augmented by 1 in ThreatConnect to fit the 0–5 scale for Threat Rating. Table 12 shows how the Recorded Future risk scores are mapped to Threat Rating in ThreatConnect.

Frequently Asked Questions (FAQ)

When configuring the Recorded Future Intelligence Engine App, I do not see Analyst Notes offered as a downloadable data type. How do I ensure my Recorded Future Analyst Notes are imported into ThreatConnect?

Analyst Notes added to the selected Risk List entity type(s), as well as those created in your Recorded Future modules within the last 24 hours, will be imported into ThreatConnect. See the “Analyst Note” section for more information on how Analyst Notes are mapped to the ThreatConnect data model.

Why are there several URL errors in the batch errors report? (e.g., [xyz.com] could not be processed as a valid URL due to missing or invalid data (summary is invalid for the given type))

URL errors occur when URL objects coming from Recorded Future use an invalid URL format. Some examples of this behavior include the following:

• ww3.xyz.com: This URL is missing the protocol, such as http://.
• http:ww2.xyz.com/page#: This URL is terminated with a special character.

URL objects with an invalid URL format will not be imported into ThreatConnect. Note that this issue occurs rarely.

Why are Indicators with risk scores that are less than the minimum risk score being ingested into ThreatConnect?

Indicators with risk scores that are less than the minimum risk score are ingested because they exist as links from other Risk List entities. To prevent Indicators with a risk score less than the minimum risk score (i.e., the value for the App's Minimum Risk Score parameter) from being ingested, clear the Collect Indicators Linked in Recorded Future Below the Minimum Risk Score checkbox in the Feed Deployer when configuring and deploying the App.

How does the Recorded Future Intelligence Engine Feed API Service App differ from the Recorded Future Risk List Job App?

The Recorded Future Risk List Job App allows users to do the following:

• collect data from Risk List entities
• create Indicators with evidence details
• map evidence details to a Description Attribute and risk rules to Tags

The Recorded Future Intelligence Engine Feed API Service App collects data in the following ways:

• ingests Risk List entities with several of their Attributes
• ingests all of the Analyst Notes and attached PDFs associated with Risk List entities
• ingests associated Risk List entities and allows users to view the first-level associations created between the entities and Analyst Notes (i.e., the actual link)
• obtains a link to each Risk List entity's Recorded Future Intelligence Card
• ingests Analyst Notes that have been published in the last 24 hours daily

For more information on how Feed API Service Apps function in ThreatConnect, see Feed API Services.

How long does the Recorded Future Intelligence Engine Feed API Service App take to ingest a complete set of data on its initial run?

In most cases, the Recorded Future Intelligence Engine App takes 2–4 days to complete the initial data ingestion, depending on how you configure risk score and which Risk List entity types you select to ingest in the Feed Deployer.

In which order are Risk List entities ingested on the initial run?

On the initial run of the Recorded Future Intelligence Engine App, Risk List entities will be ingested in the following order:

1. Domain
2. Hash
3. IP
4. URL
5. Vulnerability

You will likely see Risk List entities ingested in chunks along with the associated or linked entities. Note that there may be delays between the creation of the Risk List source entities and the Risk List link entities in ThreatConnect.

How often does the Recorded Future Intelligence Engine App ingest each Risk List entity type after the initial run?

See the following table for each Risk List entity type's download frequency. Analyst Notes are downloaded daily.

How can I identify where the Recorded Future Intelligence Engine App collected Indicators and Groups from in Recorded Future?

The following Source: Tags, which are applied to Indicators and Groups in ThreatConnect that were ingested from Recorded Future, indicate where the integration collected them from in Recorded Future:

• Source: Risk List: The entity was ingested from a Risk List.
• Source: Risk List Link: The entity was ingested as a linked entity from a Risk List entity.
• Source: Analyst Note: The entity was ingested from an Analyst Note.
• Source: Analyst Note Link: The entity was ingested as a linked entity from an Analyst Note.
• Source: RF Alert: The entity was ingested from an Alert.
• Source: RF Alert Link: The entity was ingested as a linked entity from an Alert.
• Source: RF Threat Map: The entity was ingested from a Threat Map entity.
• Source: RF Threat Map Link: The entity was ingested as a linked entity from a Threat Map entity.

Note that an Indicator or Group may have more than one of these Tags applied to them, as the corresponding entities could be associated to one another.

ThreatConnect^® is a registered trademark, and TC Exchange™ is a trademark, of ThreatConnect, Inc.
JavaScript^® is a registered trademark of Oracle Corporation.
Recorded Future^® is a registered trademark of Recorded Future, Inc.
MITRE ATT&CK^® and ATT&CK^® are registered trademarks of The MITRE Corporation.

30083-02 EN Rev. A