Dataminr Pulse Alerts Engine Integration User Guide

Overview

The Dataminr Pulse Alerts Engine Feed API Service App in ThreatConnect® unlocks real-time alerting by seamlessly ingesting Dataminr Pulse Cyber Alerts and converting them into actionable intelligence within ThreatConnect. The integration ingests Cyber Alerts from Dataminr Pulse Alert Lists and creates corresponding objects in ThreatConnect with select Dataminr Pulse metadata and AI summary information:

• Alerts are created as Event Groups in ThreatConnect. Intel Agent and Live Brief AI summaries from Dataminr Pulse are included as AI insights for the Event Group in ThreatConnect, with Intel Agent and Live Brief Tags added to Event Groups that have those respective AI summary types.
• Discovered Entities for Alerts are created as Groups (Intrusion Set, Malware, or Vulnerability) associated to the Event Group corresponding to the ingested Alert.
• Key Points for Alerts are created as Indicators (Address, ASN, File, Host, or URL) or Groups (Intrusion Set or Vulnerability) associated to the Event Group corresponding to the ingested Alert.

The Dataminr Pulse Alerts Engine App ingests new Alerts into ThreatConnect every five minutes starting from the time at which the integration is deployed.

Dependencies

ThreatConnect Dependencies

• Active ThreatConnect Application Programming Interface (API) key
• ThreatConnect instance with version 7.11.2-M1218R or newer installed

Dataminr Dependencies

• Dataminr Pulse API public key
• Dataminr Pulse API secret

Application Setup and Configuration

The Dataminr Pulse Alerts Engine App leverages the Feed Deployer to create a Source for data ingestion from Dataminr Pulse in an Organization and to configure the corresponding Service’s ingestion and authentication parameters. After you install the Dataminr Pulse Alerts Engine App on your ThreatConnect instance, you can deploy it to any Organization. It must be deployed separately for each Organization in which you want to create a Source for data ingestion and a corresponding Service.

Install the Dataminr Pulse Alerts Engine App

Follow these steps to install the Dataminr Pulse Alerts Engine App on your ThreatConnect instance:

1. Log into ThreatConnect with a System Administrator account.
2. From the Settingsmenu on the top navigation bar, select TC Exchange Settings.
3. Select the Catalog tab on the TC Exchange™ Settings screen.
4. Locate the Dataminr Pulse Alerts Engine App on the Catalog tab.
5. Click Installin the Options column for the App.
6. Click INSTALL in the App’s Release Notes window.
7. After you install the Dataminr Pulse Alerts Engine App, the Feed Deployer opens automatically. Follow the procedure in the “Deploy the Dataminr Pulse Alerts Engine App to an Organization” section to deploy the Dataminr Pulse Alerts Engine App to a Source in an Organization and configure the corresponding Service.

Deploy the Dataminr Pulse Alerts Engine App to an Organization

Follow these steps to deploy the Dataminr Pulse Alerts Engine App to an Organization:

1. Log into ThreatConnect with a System Administrator account.
2. From the Settingsmenu on the top navigation bar, select TC Exchange Settings.
3. Locate the Dataminr Pulse Alerts Engine App on the Installed tab. Then select Deploy from the Options ⋮ dropdown.
4. Follow the instructions in Table 1 to fill out the fields in the Feed Deployer window for a deployment of the Dataminr Pulse Alerts Engine App.

    

   Name	Description	Required?
   Source Tab
   Sources to Create	Enter the name of the Source for the feed. Note Unless you are redeploying the feed to an existing Source in an Organization, the name of the Source must be unique on your ThreatConnect instance. It is recommended to add the Organization’s name to the end of the default Source name (e.g., Dataminr Pulse Alerts Engine – Demo Organization) for easy identification of the Source’s owner.	Required
   Owner	Select the Organization in which the Source will be created.	Required
   Activate Deprecation	Select this checkbox to allow confidence deprecation rules to be created and applied to Indicators in the Source.	Optional
   Create Attributes	Select this checkbox to allow custom Attribute Types for the Dataminr Pulse Alerts Engine App to be created on the System level of your ThreatConnect instance. Important It is recommended that you keep this checkbox selected. If you deselect it, data from the Dataminr Pulse Alerts Engine App mapped to those Attribute Types will not be ingested.	Optional
   Parameters Tab
   Launch Server	Select tc-job as the launch server for the Feed API Service.	Required
   Dataminr List IDs	Enter the IDs for the Dataminr Pulse Cyber Alert Lists to ingest into ThreatConnect. If you leave this field blank, the App will ingest all Dataminr Pulse Cyber Alerts to which your Dataminr Pulse API account has access. Note If a List whose ID is entered in this field is deleted from Dataminr, the Dataminr Pulse Alerts Engine App may stop working. Please cross-check the available Alert Lists in Dataminr Pulse against the List IDs entered in this field to ensure uninterrupted ingestion from the App.	Optional
   Variables Tab
   Dataminr API Public	Enter the Dataminr Pulse API public key.	Required
   Dataminr API Secret	Enter the Dataminr Pulse API secret.	Required
   Confirm Tab
   Run Feeds after deployment	Select this checkbox to run the Dataminr Pulse Alerts Engine Service immediately after you click DEPLOY on the Feed Deployer window.	Optional
   Confirm Deployment Over Existing Source	This checkbox and a warning message are displayed on the Confirm tab if the Source name entered on the Source tab is already used by a Source owned by the selected Organization. To confirm redeploying the App to the existing Source, select the checkbox. This will activate the DEPLOY button. Otherwise, you must return to the Source tab and either change the Source name or select a different Organization. Warning When you redeploy a Feed API Service to a Source, existing data in the Source may be overwritten. Redeployment will also create a new Service for the Feed API Service App. It is recommended that you delete the previous Service for the Feed API Service App after the new one is created.	Optional

5. Click DEPLOY on the Confirm tab of the Feed Deployer window to deploy the Dataminr Pulse Alerts Engine App in the Organization, which will create a Source for the feed in the Organization and a corresponding Feed API Service.

Dataminr Pulse Alerts Engine UI

After installing the Dataminr Pulse Alerts Engine App and deploying it to an Organization, you can access the Dataminr Pulse Alerts Engine user interface (UI), where you can manage data ingestion from Dataminr Pulse into the Source created in the Organization.

Follow these steps to access the Dataminr Pulse Alerts Engine UI:

1. Log into ThreatConnect with a System Administrator account or a user account in the Organization with an Organization role of Organization Administrator.
2. From the Automation & Feeds dropdown on the top navigation bar, select Services.
3. Locate the row for the Dataminr Pulse Alerts Engine Feed Service.
   Hint

   You can select Feed Service from the Service Type dropdown at the upper right to filter the screen to show only Feed API Services.
4. Turn on the slider in the Enable column if the Service is not already enabled.
5. Click the link in the Service’s API Path field to open the Dataminr Pulse Alerts Engine UI.

The following screens are available in the Dataminr Pulse Alerts Engine Service UI:

• Dashboard
• Jobs
• Tasks
• Download
• Batch Errors

Dashboard

The Dashboard screen (Figure 1) provides an overview of the number of Cyber Alerts, categorized by Alert subtype, ingested from Dataminr Pulse.

Jobs

The Jobs screen (Figure 2) breaks down the ingestion of Dataminr Pulse data into manageable Job-like tasks, displaying all processes that are pending, in progress, complete, and failed. The ⋯ menu in a Job’s row provides the following options:

• Details: View details for the Job, such as download, convert, and upload start and complete times and counts of downloaded and batched Groups and Indicators.
• Download Files: Download metadata files for all Jobs and data (convert, download, and upload) files for completed Jobs.
• Batch Errors: View errors that have occurred for the Job on the Batch Errors screen.

You can filter Dataminr Pulse Alerts Engine Service Jobs by the following elements:

• Job ID: Enter text into this box to search for a Job by its Job ID.
• Job Type: Select Job types to display on the Jobs screen.
• Status: Select Job statuses to display on the Jobs screen.

Tasks

The Tasks screen (Figure 3) displays all Tasks that may be part of a Job, including each step of the download, convert, and upload processes, as well as Tasks for the Dataminr Pulse Alerts Engine Service, such as Monitor, Scheduler, and Cleaner. The current status (Idle, Paused, or Running), name, description, and heartbeat timeout length, in minutes, are displayed for each Task. The ⋯ menu in a Task’s row provides the following options, depending on the Task’s status:

• Run (idle and paused Tasks only)
• Pause (idle and running Tasks only)
• Resume (paused Tasks only)
• Kill (running Tasks only)

Under the table is a dashboard where you can view runtime analytics.

Download

The Download screen is not functional for Services for version 2.0.0 of the Dataminr Pulse Alerts Engine App.

Batch Errors

The Batch Errors screen (Figure 4) displays an overview of the batch error types that have occurred for Job requests. You can enter keywords to filter by Job ID.

Select an error type to open a drawer containing a table with details on all batch errors of that type (Figure 5). You can enter keywords to filter by reason for error.

Data Mappings

The data mappings in Table 2 through Table 12 illustrate how data are mapped from Dataminr Pulse API endpoints into the ThreatConnect data model.

Alert (Cyber)

ThreatConnect object type: Event Group

Address

ThreatConnect object type: Address Indicator

ASN

ThreatConnect object type: ASN Indicator

Malware From Intel Agent

ThreatConnect object type: Malware Group

Malware From Metadata

ThreatConnect object type: Malware Group

Threat Actor From Intel Agent

ThreatConnect object type: Intrusion Set Group

Threat Actor From Metadata

ThreatConnect object type: Intrusion Set Group

URL

ThreatConnect object type: Host or URL Indicator

Vulnerability From Intel Agent

ThreatConnect object type: Vulnerability Group

Vulnerability From Metadata

ThreatConnect object type: Vulnerability Group

Frequently Asked Questions (FAQ)

When does ThreatConnect start ingesting data from Dataminr Pulse? 

If you select the Run Feeds After Deployment checkbox in the Feed Deployer when deploying the Dataminr Pulse Alerts Engine App to an Organization, ThreatConnect will ingest Dataminr Pulse data once you click DEPLOY, with the time of deployment being the starting time from which to retrieve Alerts. This means that ThreatConnect will ingest only Alerts created at the time of deployment or later; it will not ingest Alerts created before the time of deployment. If you do not select the Run Feeds After Deployment checkbox, ThreatConnect will ingest Dataminr Pulse data once you turn on the corresponding Service for the App on the Services screen.

Does ThreatConnect ingest Cyber Alerts from all Dataminr Pulse Alert Lists? 

If you leave the Dataminr List IDs field blank in the Feed Deployer when deploying the Dataminr Pulse Alerts Engine App to an Organization, then ThreatConnect will ingest Cyber Alerts from all Dataminr Pulse Alert Lists you have provisioned. If you enter List IDs, then ThreatConnect will ingest Cyber Alerts only from the Dataminr Pulse Alert Lists with those IDs. Please check your Dataminr API rate limits to inform your decision on the number of Alerts you can ingest into ThreatConnect.

How can I filter my ThreatConnect data to show only Alerts from select Alert Lists from Dataminr Pulse?

Use a ThreatConnect Query Language (TQL) query to filter ingested Dataminr Pulse Alerts by Alert List. For example, the following query returns only Alerts that belong to the Vulnerability Flash Alert List (that is, the query returns only Event Groups that have an Alert List Attribute with a value of Vulnerability Flash):

typeName in ("Event") and attributeAlert_List in ("Vulnerability Flash")

How can I filter my ThreatConnect data to show only Alerts with a particular priority from Dataminr Pulse?

Use a TQL query to filter ingested Dataminr Pulse Alerts by priority. For example, the following query returns only Alerts whose priority is either Urgent or Flash (that is, the query returns only Event Groups that have a Priority Attribute with a value of Urgent or Flash):

typeName in ("Event") and attributePriority in ("Urgent", "Flash")

How can I filter my ThreatConnect data to show only Alerts with AI summaries from Dataminr Pulse?

Use a TQL query to filter ingested Dataminr Pulse Alerts for Alerts that have AI summary data. For example, the following query returns only Alerts that have Intel Agent or Live Brief AI summaries (that is, the query returns only Event Groups that have a Tag with a value of Intel Agent or Live Brief):

typeName in ("Event") and tag in ("Intel Agent", "Live Brief")

Similarly, the following query returns only Alerts with AI summary data provided by Dataminr and containing the text Microsoft Server Message Block (that is, the query returns only Event Groups that have AI insights that are provided by Dataminr and that contain the text Microsoft Server Message Block):

typeName in ("Event") and aiProvider="Dataminr" and insights contains "Microsoft Server Message Block"

ThreatConnect® is a registered trademark, and TC Exchange™ is a trademark, of ThreatConnect, Inc.
MITRE ATT&CK® and ATT&CK® are registered trademarks of The MITRE Corporation.
JavaScript® is a registered trademark of Oracle Corporation.

30087-02 EN Rev. A