Dataminr Pulse Alerts Engine Integration User Guide

Overview

The Dataminr Pulse Alerts Engine feed API service app unlocks the power of real-time alerting in ThreatConnect® by ingesting Dataminr Pulse Cyber Alerts and converting them into actionable intelligence in ThreatConnect. The app ingests Cyber Alerts from Dataminr Pulse Alert Lists every 10 minutes and creates corresponding objects in ThreatConnect with select Dataminr Pulse metadata and AI-powered context:

• Alerts are created as Event Groups in ThreatConnect. Intel Agent and Live Brief AI content from Dataminr Pulse are included as AI insights for the Event Group in ThreatConnect, with Intel Agent and Live Brief Tags added to Event Groups that have those respective AI content types.
• Discovered Entities for Alert Intel Agents are created as Indicators (Address, Host, or URL) or Groups (Intrusion Set, Malware, or Vulnerability) associated to the Event Group corresponding to the ingested Alert.
• Key Points for Alerts are created as Indicators (Address, ASN, File, Host, or URL) or Groups (Intrusion Set, Malware, or Vulnerability) associated to the Event Group corresponding to the ingested Alert.

Dependencies

ThreatConnect Dependencies

• Active ThreatConnect API key
• ThreatConnect instance with version 7.11.2-M1218R or newer installed
  Note

  Certain context-enriching elements captured in attributes will be limited or unavailable on ThreatConnect instances running on versions earlier than 7.12.3.

Dataminr Dependencies

• Dataminr Pulse API public key
• Dataminr Pulse API secret
• (Optional) Dataminr List IDs
• (Optional) Dataminr Alert IDs
• (Optional) Dataminr hydration endpoint configured to receive Vulnerability hydration data
  Note

  The hydration endpoint provides Common Platform Enumeration (CPE) data for Vulnerabilities ingested from Dataminr. It requires a subscription. Please contact your Dataminr Customer Success representative for more information.

Dataminr List IDs

When configuring a deployment of the Dataminr Pulse Alerts Engine app to an Organization, you can specify Dataminr Pulse List IDs from which to ingest Alerts. Follow these steps to obtain a List ID in Dataminr Pulse:

1. Log into Dataminr Pulse.
2. Select an Alert from the List whose List ID you want to identify.
3. From the Actions dropdown, select Email.
4. In the EMAIL CONTENT section of the Email alert window, locate the Link to Alert in Dataminr Pulse: section. In the link, the List ID is the number between #alertDetailWL/ and /alertDetail.
   Example

   In https://app.dataminr.com/#alertDetailWL/9999999/alertDetail/6/123456789012345678901234567890-1234567890-1, the List ID is 9999999.

Dataminr Alert IDs

When downloading data for a single Alert from Dataminr Pulse to upload into ThreatConnect, you need to specify the ID of the Alert to download. Follow these steps to obtain an Alert ID in Dataminr Pulse:

1. Log into Dataminr Pulse.
2. Select an Alert from the List whose List ID you want to identify.
3. From the Actions dropdown, select Email.
4. In the EMAIL CONTENT section of the Email alert window, locate the Link to Alert in Dataminr Pulse: section. In the link, the Alert ID is the number after the last /character.
   Example

   In https://app.dataminr.com/#alertDetailWL/9999999/alertDetail/6/123456789012345678901234567890-1234567890-1, the Alert ID is 123456789012345678901234567890-1234567890-1.

Application Setup and Configuration

The Dataminr Pulse Alerts Engine app leverages the Feed Deployer to create a Source for data ingestion from Dataminr Pulse in an Organization and to configure the corresponding service’s ingestion and authentication parameters. After you install the Dataminr Pulse Alerts Engine app on your ThreatConnect instance, you can deploy it to any Organization. It must be deployed separately for each Organization in which you want to create a Source for data ingestion and a corresponding service.

Install the Dataminr Pulse Alerts Engine App

Follow these steps to install the Dataminr Pulse Alerts Engine app on your ThreatConnect instance:

1. Log into ThreatConnect with a System Administrator account.
2. From the Settingsmenu on the top navigation bar, select TC Exchange Settings.
3. Select the Catalog tab on the TC Exchange™ Settings screen.
4. Locate the Dataminr Pulse Alerts Engine app on the Catalog tab.
5. Click Installin the Options column for the app.
6. Click INSTALL in the app’s Release Notes window.
7. After you install the Dataminr Pulse Alerts Engine app, the Feed Deployer opens automatically. Follow the procedure in the “Deploy the Dataminr Pulse Alerts Engine App to an Organization” section to deploy the Dataminr Pulse Alerts Engine app to a Source in an Organization and configure the corresponding service.

Deploy the Dataminr Pulse Alerts Engine App to an Organization

Follow these steps to deploy the Dataminr Pulse Alerts Engine app to an Organization:

1. Log into ThreatConnect with a System Administrator account.
2. From the Settingsmenu on the top navigation bar, select TC Exchange Settings.
3. Locate the Dataminr Pulse Alerts Engine app on the Installed tab. Then select Deploy from the Options ⋮ dropdown.
4. Follow the instructions in Table 1 to fill out the fields in the Feed Deployer window for a deployment of the Dataminr Pulse Alerts Engine app.

    

   Name	Description	Required?
   Source Tab
   Sources to Create	Enter the name of the Source for the feed. Note Unless you are redeploying the feed to an existing Source in an Organization, the name of the Source must be unique on your ThreatConnect instance. It is recommended to add the Organization’s name to the end of the default Source name (e.g., Dataminr Pulse Alerts Engine – Demo Organization) for easy identification of the Source’s owner.	Required
   Owner	Select the Organization in which the Source will be created.	Required
   Activate Deprecation	Select this checkbox to allow confidence deprecation rules to be created and applied to Indicators in the Source.	Optional
   Create Attributes	Select this checkbox to allow custom attribute types for the Dataminr Pulse Alerts Engine app to be created on the System level of your ThreatConnect instance. Important It is recommended that you keep this checkbox selected. If you deselect it, data from the Dataminr Pulse Alerts Engine app mapped to those attribute types will not be ingested.	Optional
   Parameters Tab
   Launch Server	Select tc-job as the launch server for the feed API service.	Required
   Dataminr List IDs	Enter the IDs for the Dataminr Pulse Cyber Alert Lists to ingest into ThreatConnect. If you leave this field blank, the app will ingest all Dataminr Pulse Cyber Alerts to which your Dataminr Pulse API account has access. Note If a List whose ID is entered in this field is deleted from Dataminr, the Dataminr Pulse Alerts Engine app may stop working. Please cross-check the available Alert Lists in Dataminr Pulse against the List IDs entered in this field to ensure uninterrupted ingestion from the app.	Optional
   Notification Digest Interval	Select the interval at which the Dataminr Pulse Alerts Engine app should send notifications about job failure outcomes to the Notifications Center for users who are members of the Source for the Dataminr Pulse Alerts Engine service. The dissemination of the notifications is determined by each user’s Notifications Center settings.	Required
   Notification Types	Select the notification types to include in the notification digest. Note The App Startup notification is sent only once, when the app is started for the first time. Selection of this option determines whether the initial digest sent to the Notifications Center will include a notification about successful app startup. Note All notifications about app functionality (startup, startup failure, and shutdown) and job failure outcomes (failure, retry, and recovery) are provided on the Notifications screen in the service UI.	Optional
   Variables Tab
   Dataminr API Public	Enter the Dataminr Pulse API public key.	Required
   Dataminr API Secret	Enter the Dataminr Pulse API secret.	Required
   Confirm Tab
   Run Feeds after deployment	Select this checkbox to run the Dataminr Pulse Alerts Engine service immediately after you click DEPLOY on the Feed Deployer window.	Optional
   Confirm Deployment Over Existing Source	This checkbox and a warning message are displayed on the Confirm tab if the Source name entered on the Source tab is already used by a Source owned by the selected Organization. To confirm redeploying the app to the existing Source, select the checkbox. This will activate the DEPLOY button. Otherwise, you must return to the Source tab and either change the Source name or select a different Organization. Warning When you redeploy a feed API service to a Source, existing data in the Source may be overwritten. Redeployment will also create a new service for the feed API service app. It is recommended that you delete the previous service for the feed API service app after the new one is created.	Optional

5. Click DEPLOY on the Confirm tab of the Feed Deployer window to deploy the Dataminr Pulse Alerts Engine app in the Organization, which will create a Source for the feed in the Organization and a corresponding feed API service.

Dataminr Pulse Alerts Engine UI

After installing the Dataminr Pulse Alerts Engine app and deploying it to an Organization, you can access the Dataminr Pulse Alerts Engine UI, where you can manage data ingestion from Dataminr Pulse into the Source created in the Organization.

Follow these steps to access the Dataminr Pulse Alerts Engine UI:

1. Log into ThreatConnect with a System Administrator account or a user account in the Organization with an Organization role of Organization Administrator.
2. From the Automation & Feeds dropdown on the top navigation bar, select Services.
3. Locate the row for the Dataminr Pulse Alerts Engine feed API service.
   Hint

   Select Feed Service from the Service Type dropdown at the upper right to filter the screen to show only feed API services. If there are multiple services for the Dataminr Pulse Alerts Engine app, you can identify the one configured for your Organization by clicking the row for a service to view its Details drawer, which includes an Organization field showing the Organization that owns the Source for that service.
4. Turn on the toggle in the Enable column if the service is not already enabled.
5. Click the link in the service’s API Path field to open the Dataminr Pulse Alerts Engine UI.

The following screens are available in the Dataminr Pulse Alerts Engine service UI:

• Dashboard
• Jobs
• Tasks
• Download
• Batch Errors
• Notifications
• Attachment Status

Dashboard

The Dashboard screen provides an overview of the number of Cyber Alerts, categorized by Alert subtype, ingested from Dataminr Pulse.

Jobs

The Jobs screen breaks down the ingestion of Dataminr Pulse data into manageable job-like tasks, displaying all processes that are pending, in progress, complete, and failed. The ⋯ menu in a job’s row provides the following options:

• Details: View details for the job, such as download, convert, and upload start and complete times and counts of downloaded and batched Groups and Indicators.
• Download Files: Download metadata files for all jobs and data (convert, download, and upload) files for completed jobs.
• Batch Errors: View errors that have occurred for the job on the Batch Errors screen.

You can filter Dataminr Pulse Alerts Engine service jobs by the following elements:

• Job ID: Enter text into this box to search for a job by its job ID.
• Job Type: Select job types to display on the Jobs screen.
• Status: Select job statuses to display on the Jobs screen.

Tasks

The Tasks screen displays all tasks that may be part of a job, including each step of the download, convert, and upload processes, as well as tasks for the Dataminr Pulse Alerts Engine service, such as monitor, scheduler, and cleaner. The current status (Idle, Paused, or Running), name, description, and heartbeat timeout length, in minutes, are displayed for each task. The ⋯ menu in a task’s row provides the following options, depending on the task’s status:

• Run (idle and paused tasks only)
• Pause (idle and running tasks only)
• Resume (paused tasks only)
• Kill (running tasks only)

Under the table is a dashboard where you can view runtime analytics.

Download

The Download screen lets you download JavaScript® Object Notation (JSON) data for an Alert in Dataminr Pulse and then upload the data into ThreatConnect. Follow these steps to download JSON data for a Dataminr Pulse Alert on the Download screen and then upload the data into ThreatConnect:

1. Alert ID: Enter the Alert ID for the Alert to download.
2. Click Download. The JSON data will be displayed in two columns: Results (raw JSON data) and Converted (JSON data in ThreatConnect batch format).
3. Click Upload to submit the converted threat intelligence data via the ThreatConnect Batch API.

Batch Errors

The Batch Errors screen displays an overview of the batch error types that have occurred for job requests. You can enter keywords to filter by job ID. Select an error type to open a drawer containing a table with details on all batch errors of that type. You can enter keywords to filter by reason for error.

Notifications

The Notifications screen displays a table with details on notifications regarding app functionality and job failure outcomes for the Dataminr Pulse Alerts Engine service.

Attachment Status

The Attachment Status screen is not functional for services for version 2.0.5 of the Dataminr Pulse Alerts Engine app.

Data Mappings

The data mappings in Table 2 through Table 15 illustrate how data are mapped from Dataminr Pulse API endpoints to the ThreatConnect data model.

Alert (Cyber)

ThreatConnect object type: Event Group

Address From Intel Agent

ThreatConnect object type: Address Indicator

Address From Metadata

ThreatConnect object type: Address Indicator

ASN From Metadata

ThreatConnect object type: ASN Indicator

File From Metadata

ThreatConnect object type: File Indicator

Malware From Intel Agent

ThreatConnect object type: Malware Group

Malware From Metadata

ThreatConnect object type: Malware Group

Threat Actor From Intel Agent

ThreatConnect object type: Intrusion Set Group

Threat Actor From Metadata

ThreatConnect object type: Intrusion Set Group

URL From Intel Agent

ThreatConnect object type: Host or URL Indicator

URL From Metadata

ThreatConnect object type: Host or URL Indicator

Vulnerability From Intel Agent

ThreatConnect object type: Vulnerability Group

Vulnerability From Metadata

ThreatConnect object type: Vulnerability Group

Vulnerability From the Hydration Endpoint

ThreatConnect object type: Vulnerability Group

Frequently Asked Questions (FAQ)

When does ThreatConnect start ingesting data from Dataminr Pulse?

If you select the Run Feeds After Deployment checkbox in the Feed Deployer when deploying the Dataminr Pulse Alerts Engine app to an Organization, ThreatConnect will ingest Dataminr Pulse data once you click DEPLOY, with the time of deployment being the starting time from which to retrieve Alerts. This means that ThreatConnect will ingest only Alerts created at the time of deployment or later; it will not ingest Alerts created before the time of deployment. If you do not select the Run Feeds After Deployment checkbox, ThreatConnect will ingest Dataminr Pulse data once you turn on the corresponding service for the app on the Services screen.

Does ThreatConnect ingest Cyber Alerts from all Dataminr Pulse Alert Lists?

If you leave the Dataminr List IDs field blank in the Feed Deployer when deploying the Dataminr Pulse Alerts Engine app to an Organization, then ThreatConnect will ingest Cyber Alerts from all Dataminr Pulse Alert Lists you have provisioned. If you enter List IDs, then ThreatConnect will ingest Cyber Alerts only from the Dataminr Pulse Alert Lists with those IDs. Please check your Dataminr API rate limits to inform your decision on the number of Alerts you can ingest into ThreatConnect.

How can I filter my ThreatConnect data to show only Alerts from select Alert Lists from Dataminr Pulse?

Use a ThreatConnect Query Language (TQL) query to filter ingested Dataminr Pulse Alerts by Alert List. For example, the following query returns only Alerts that belong to the Vulnerability Flash Alert List (that is, the query returns only Event Groups that have an Alert List attribute with a value of Vulnerability Flash):

typeName in ("Event") and attributeAlert_List in ("Vulnerability Flash")

How can I filter my ThreatConnect data to show only Alerts with a particular priority from Dataminr Pulse?

Use a TQL query to filter ingested Dataminr Pulse Alerts by priority. For example, the following query returns only Alerts whose priority is either Urgent or Flash (that is, the query returns only Event Groups that have a Priority attribute with a value of Urgent or Flash):

typeName in ("Event") and attributePriority in ("Urgent", "Flash")

How can I filter my ThreatConnect data to show only Alerts with AI content from Dataminr Pulse?

Use a TQL query to filter ingested Dataminr Pulse Alerts for Alerts that have AI content. For example, the following query returns only Alerts that have Intel Agent or Live Brief AI content (that is, the query returns only Event Groups that have a Tag with a value of Intel Agent or Live Brief):

typeName in ("Event") and tag in ("Intel Agent", "Live Brief")

Similarly, the following query returns only Alerts with AI content provided by Dataminr and containing the text Microsoft Server Message Block (that is, the query returns only Event Groups which have AI insights that are provided by Dataminr and that contain the text Microsoft Server Message Block):

typeName in ("Event") and aiProvider="Dataminr" and insights contains "Microsoft Server Message Block"

ThreatConnect® is a registered trademark, and TC Exchange™ is a trademark, of ThreatConnect, Inc.
MITRE ATT&CK® and ATT&CK® are registered trademarks of The MITRE Corporation.
JavaScript® is a registered trademark of Oracle Corporation.

30087-03 EN Rev. A