Elastic Security Integration User Guide
  • 23 May 2024
  • 28 Minutes to read
  • Dark
    Light

Elastic Security Integration User Guide

  • Dark
    Light

Article summary

Software Version
This guide applies to the Elastic Security ThreatConnect integration version 1.0.x.

Overview

The ThreatConnect® integration with Elastic® Security brings ThreatConnect Indicators and their Attributes and associations into Elastic Security. To accomplish this, the integration uses the Filebeat® module to pull data from ThreatConnect via the ThreatConnect v3 API and store the data in Elastic Security as log files. The ThreatConnect log files can then be parsed and visualized using Kibana® Discover, a built-in Kibana dashboard, and the Indicators page in Elastic Security. You may also define detection rules that can activate on the threat intelligence data ingested by Elastic Security.

Integration Diagram_Elastic Security Integration User Guide_Software Version 1.0

Dependencies

ThreatConnect Dependencies

  • Active ThreatConnect Application Programming Interface (API) user
  • ThreatConnect instance with version 7.3.1 or newer installed
Note
All ThreatConnect dependencies will be provided by default to subscribing ThreatConnect Cloud customers. Customers on Dedicated Cloud and On-Premises instances can enable these settings on the Account Settings screen within their ThreatConnect instance.

Elastic Security Dependencies

  • Elastic Stack version 8.11 or newer

Installing the Integration

Follow these steps to install the Elastic Security ThreatConnect integration in Kibana:

  1. In Kibana, open the main menu and navigate to Management > Integrations. Alternatively, click Add integrations at the bottom of the main menu or on the Home page.
  2. On the Browse integrations tab, search for “threatconnect” (without quotation marks). Then click the ThreatConnect card in the search results (Figure 1).
    Figure 1_Elastic Security Integration User Guide_Software Version 1.0

     

    Note
    If the ThreatConnect integration does not show up in Kibana, turn on the Display beta integrations toggle under the All Categories section on the left side of the Integrations page, as the integration may still be in technical preview.
  3. On the ThreatConnect integration page, click Add ThreatConnect to install the ThreatConnect integration (Figure 2).
    Figure 2_Elastic Security Integration User Guide_Software Version 1.0

     

After the ThreatConnect integration is installed, you will be directed to the Add ThreatConnect integration page. Here, you can configure the ThreatConnect integration.

Configuring the Integration

The Elastic Security ThreatConnect integration requires configuration in both the ThreatConnect and Elastic instances that will use the integration.

ThreatConnect Configuration

A ThreatConnect API user account is required to use the Elastic Security ThreatConnect integration. If you do not have a ThreatConnect API user account, follow these steps to create one:

  1. Log into ThreatConnect with an Organization Administrator account.
  2. On the top navigation bar, hover over SettingsSettings iconand select Org Settings.
  3. On the Membership tab of the Organization Settings screen, click Create API User to create an API user. When creating the API user, configure it as follows:
    • System Role: Select Api User.
    • Organization Role: Select Standard User.
    • Include in Observations and False Positives: Select this checkbox.
    • Allow User to Exceed API Link Limit: Select this checkbox.
  4. On the API User Administration window, record the Access ID and Secret Key for the API user, as they will be used when configuring the ThreatConnect integration in Elastic. Then click SAVE.

Elastic Configuration

Step 1: Configuring and Deploying the Integration

After you install the Elastic Security ThreatConnect integration, follow these steps to configure the integration on the Add ThreatConnect integration page in Kibana. In addition to these steps, you may also view a complete list of configuration parameters in the “Configuration Parameters” section.

Important
See the “Configuration Performance Recommendations” section for performance-related recommendations to follow when configuring the Elastic Security ThreatConnect integration.
  1. Configure the following parameters in the Integration settings subsection of the Configure integration section of the page (Figure 3):
    Figure 3_Elastic Security Integration User Guide_Software Version 1.0

     

    • Integration name : (Required) Enter the integration’s name. This name will be visible on the Elastic Agent running the integration.
    • Description: (Optional) Enter a description of the integration, if desired.
    • Advanced options: Click Advanced options to display the following parameter:
      • Namespace: (Required) This parameter controls the name of the integration’s data stream. If desired, change the default namespace inherited from the selected agent policy. Note that pressing Enter in this field will convert the entered text into a tag.
  2. Configure the following parameters in the Settings subsection of the Configure integration section (Figure 4):
    Figure 4_Elastic Security Integration User Guide_Software Version 1.0

     

    • Collect ThreatConnect logs via API: (Required) Turn on this toggle to access the ThreatConnect API.
    • URL: (Required) Enter the base URL of the ThreatConnect instance. Do not include the /api portion of the URL or a trailing slash character (/).
    • Access ID: (Required) Enter the Access ID of the API user in ThreatConnect.
    • Secret Key: (Required) Enter the Secret Key of the API user in ThreatConnect.
    • Advanced options: Click Advanced options to display the following parameters:
      • Proxy URL: (Optional) If you are using a proxy, enter the URL to proxy connections in the following format: http[s]://<user>:<password>@<server name/ip>:<port>. Note that the username and password must be in a URL-encoded format.
      • SSL Configuration:(Optional) If you are using SSL, enter the SSL certificate and certificate authorities (CAs).
  3. Configure the following parameters in the last subsection of the Configure integration section (Figure 5):
    Figure 5_Elastic Security Integration User Guide_Software Version 1.0

     

    • Indicator logs: (Required) Turn on this toggle to collect Indicators from ThreatConnect.
    • TQL: (Optional) Enter a query written in ThreatConnect Query Language (TQL) to filter the Indicators that will be sent to Elastic Security. If this field is left blank, Indicators from all owners that the API user has access to in their Organization will be sent. The following examples demonstrate some of the ways that you can filter Indicators with TQL:
      • rating >= 4 and confidence >= 90: This query searches for Indicators by their Threat and Confidence Ratings. In this example, only Indicators with a Threat Rating greater than or equal to 4 and a Confidence Rating greater than or equal to 90 will be sent to Elastic Security.
      • ownerName in ("CAL Automated Threat Library") and typeName in ("URL", "EmailAddress") and dateAdded > "04-01-2024": This query searches for Indicators by the name of their owner, their Indicator type, and their creation date. In this example, only URL and Email Address Indicators in the CAL Automated Threat Library Source that were created after April 1, 2024, will be sent to Elastic Security.
      • tag in ("phishing", "ransomware"): This query searches for Indicators with one or more of the specified Tags applied to them. In this example, only Indicators with the phishing Tag, the ransomware Tag, or both the phishing and ransomware Tags applied to them will be sent to Elastic Security.
    • Initial Interval: (Required) Specify how far back to retrieve Indicators and their Attributes and associations from ThreatConnect. Indicators that were last modified within the specified amount of time will be retrieved. Acceptable time units for this parameter include the following: (hours), m (minutes), and (seconds).
      Note
      On the integration’s first run, it will retrieve data that were last modified during the specified interval up to the current time. Thereafter, the integration will retrieve data that were last modified starting from the end of the specified interval (e.g., the last 168 hours).
    • IOC Expiration Duration: (Required) Enter the amount of time after which all ingested Indicators will expire in Elastic Security. This setting is required to avoid “orphaned” Indicators that never expire. Acceptable time units for this parameter include the following: (days), (hours), and (minutes).
      Important
      Do not mix time units when entering a value for the IOC Expiration Duration parameter.
    • Interval: (Required) Enter the interval between each request to the ThreatConnect API to retrieve data. Acceptable time units for this parameter include the following: h (hours), m (minutes), and (seconds).
    • Preserve original event: (Optional) If troubleshooting the integration, turn on this toggle to preserve a raw copy of the original event added to the event.original field. Otherwise, leave this toggle turned off.
    • Advanced options: Click Advanced options to display the following parameters:
      • Include Attributes in the response: (Optional) Turn on this toggle to include an Indicator’s Attributes in the ThreatConnect API response. Note that turning on this toggle may result in a larger response size that can cause issues when viewing the data on the Discover page in Kibana.
      • Include Group Associations in the response: (Optional) Turn on this toggle to include an Indicator’s associated Groups in the ThreatConnect API response. Note that turning on this toggle may result in a larger response size that can cause issues when viewing the data on the Discover page in Kibana.
      • Include Indicator Associations in the response: (Optional) Turn on this toggle to include an Indicator’s associated Indicators in the ThreatConnect API response. Note that turning on this toggle may result in a larger response size that can cause issues when viewing the data on the Discover page in Kibana.
      • Batch Size: (Required) Enter the ThreatConnect API batch size. The maximum supported batch size is 10000.
      • HTTP Client Timeout: (Required) Enter the amount of time to wait before declaring that the HTTP client connection has timed out. Acceptable time units for this parameter include the following: ns (nanoseconds), us (microseconds), ms (milliseconds), s (seconds), m (minutes), and h (hours). If working with a large dataset, it is recommended to set this parameter’s value to at least 10m (10 minutes).
      • Enable request tracing: (Optional) The request tracer logs requests and responses to the agent’s local file system for debugging configurations. Enabling this request tracing compromises security and should be used only when troubleshooting the integration.
      • Tags: (Required) Enter the tags to apply to the incoming data in Elastic Security. By default, the following tags are populated in this parameter and should not be removed:
        • forwarded: This tag signifies to Elastic that the Indicators are being forwarded from the ThreatConnect instance’s IP address rather than coming directly from it.
        • threatconnect-indicator: This tag indicates that the data are ThreatConnect Indicators.
      • Processors: (Optional) Define the processors to reduce the number of fields in the exported event or enhance the event with metadata, if desired. The specified processors will execute in the agent before the logs are parsed.
  4. In the Where to add this integration? section, select the agent policy that will host the integration (Figure 6). You can create a new agent policy or select an existing one.
    Figure 6_Elastic Security Integration User Guide_Software Version 1.0

     

  5. (Optional) To preview the API request that will be sent to ThreatConnect, click Preview API request at the bottom of the Add ThreatConnect integration page. Previewing the API request and the corresponding JSON can help reveal any latent issues that were not found with the parameter validations.
  6. Click Save and continue at the bottom of the Add ThreatConnect integration page to save and deploy the integration to the selected agent policy.
Configuration Parameters

Table 1 defines the configuration parameters available when configuring the Elastic Security ThreatConnect integration in Kibana.

Important
See the “Configuration Performance Recommendations” section for performance-related recommendations to follow when configuring the Elastic Security ThreatConnect integration.

 

NameDescriptionRequired?Default Value
Integration nameThe integration’s name. This name will be visible on the Elastic Agent running the integration. Requiredti_threatconnect
DescriptionA description of the integration.Optional-
NamespaceThe name of the integration’s data stream. Note that pressing Enter in this parameter’s field will convert the entered text into a tag.Requireddefault
Collect ThreatConnect logs via API Specifies whether the integration can access the ThreatConnect API. RequiredOn
URLThe base URL of the ThreatConnect instance. Do not include the /api portion of the URL or a trailing slash character (/). Requiredhttps://companyabc.threatconnect.com
Access ID The Access ID of the API user in ThreatConnect.Required-
Secret Key The Secret Key of the API user in ThreatConnect.Required-
Proxy URLThe URL to proxy connections if you are using a proxy. The URL must be entered in the following format, where the username and password are in a URL-encoded format: http[s]://<user>:<password>@<server name/ip>:<port>Optional-
SSL ConfigurationThe SSL certificate and certificate authorities (CAs) to use if you are using SSL.Optional
Indicator logs Specifies whether the integration will collect Indicators from ThreatConnect.RequiredOn
TQLA query written in ThreatConnect Query Language (TQL) that filters the Indicators that will be sent to Elastic Security. If this field is left blank, Indicators from all owners that the API user has access to in their Organization will be sent.Optional-
Initial IntervalSpecifies how far back to retrieve Indicators and their Attributes and associations from ThreatConnect. Indicators that were last modified within the specified amount of time will be retrieved. Acceptable time units for this parameter include the following:
  • (hours)
  • (minutes)
  • (seconds)
Note
On the integration’s first run, it will retrieve data that were last modified during the specified interval up to the current time. Thereafter, the integration will retrieve data that were last modified starting from the end of the specified interval (e.g., the last 168 hours).
Required168h
IOC Expiration Duration The amount of time after which all ingested Indicators will expire in Elastic Security. This setting is required to avoid “orphaned” Indicators that never expire. Acceptable time units for this parameter include the following:
  • (days)
  • h (hours)
  • (minutes)
Important
Do not mix time units when entering a value for the IOC Expiration Duration parameter.
Required 90d
Interval The interval between each request to the ThreatConnect API to retrieve data. Acceptable time units for this parameter include the following:
  • h (hours)
  • (minutes)
  • (seconds)
Required24h
Preserve original event Specifies whether to preserve a raw copy of the original event added to the event.original field. Turn on this toggle only when troubleshooting the integration.OptionalOff
Include Attributes in the responseSpecifies whether to include an Indicator’s Attributes in the ThreatConnect API response. Note that turning on this toggle may result in a larger response size that can cause issues when viewing the data on the Discover page in Kibana.OptionalOff
Include Group Associations in the responseSpecifies whether to include an Indicator’s associated Groups in the ThreatConnect API response. Note that turning on this toggle may result in a larger response size that can cause issues when viewing the data on the Discover page in Kibana.OptionalOff
Include Indicator Associations in the responseSpecifies whether to include an Indicator’s associated Indicators in the ThreatConnect API response. Note that turning on this toggle may result in a larger response size that can cause issues when viewing the data on the Discover page in Kibana.OptionalOff
Batch SizeThe ThreatConnect API batch size. The maximum supported batch size is 10000.Required5000
HTTP Client TimeoutThe amount of time to wait before declaring that the HTTP client connection has timed out. Acceptable time units for this parameter include the following:
  • ns (nanoseconds)
  • us (microseconds)
  • ms (milliseconds)
  • s (seconds)
  • m (minutes)
  • h (hours)
Note
If working with a large dataset, it is recommended to set the HTTP Client Timeout parameter’s value to at least 10m (10 minutes).
Required2m
Enable request tracing The request tracer logs requests and responses to the agent’s local file system for debugging configurations. Enabling this request tracing compromises security and should be used only when troubleshooting the integration.OptionalOff
TagsThe tags to apply to the incoming data in Elastic Security. By default, the following tags are populated in this parameter and should not be removed:
  • forwarded: This tag signifies to Elastic that the Indicators are being forwarded from the ThreatConnect instance’s IP address rather than coming directly from it.
  • threatconnect-indicator: This tag indicates that the data are ThreatConnect Indicators. 
Required
  • forwarded
  • threatconnect-indicator
ProcessorsThe processors to reduce the number of fields in the exported event or enhance the event with metadata. The specified processors will execute in the agent before the logs are parsed.Optional-
Configuration Performance Recommendations

To reduce the risk of performance issues occurring while using the Elastic Security ThreatConnect integration, follow these guidelines when configuring the integration:

  • TQL: If using the TQL parameter to filter Indicators that will be sent to Elastic Security, it is recommended to have your System Administrator increase the TQL query timeout system setting for your ThreatConnect instance via the Settings tab of the System Settings screen. (See the “System Settings” section of ThreatConnect System Administration Guide for more information on viewing and modifying system settings in ThreatConnect.) You may also adjust the TQL query timeout for the ThreatConnect API user account that will be used in the integration by navigating to the Membership tab of the Organization Settings screen in ThreatConnect, editing the API user, selecting the Custom TQL Query checkbox, and entering the maximum amount of time, in milliseconds, that TQL queries made by the user will be allowed to run before timing out.
    Note
    Configuring a custom TQL query timeout for an API user will override the system-level TQL query timeout for that user.
  • Include Attributes in the response: Some Indicators may have over 1,000 Attributes, which can result in a large amount of extra data being sent to Elastic Security when the Include Attributes in the response parameter is turned on. If you experience performance issues, it is recommended to turn off the Include Attributes in the response parameter to prevent Attributes from being sent to Elastic Security.
  • Include Group Associations in the response: Some Indicators may have over 10,000 associated Groups, which can result in a large amount of extra data being sent to Elastic Security when the Include Group Associations in the response parameter is turned on. If you experience performance issues, it is recommended to turn off the Include Group Associations in the response parameter to prevent associated Groups from being sent to Elastic Security.
  • Include Indicator Associations in the response: Some Indicators may have over 10,000 associated Indicators, which can result in a large amount of extra data being sent to Elastic Security when the Include Indicator Associations in the response parameter is turned on. If you experience performance issues, it is recommended to turn off the Include Indicator Associations in the response parameter to prevent associated Indicators from being sent to Elastic Security.
  • Batch Size: If working with small or busy Elastic Agents, or small TQL timeouts for your ThreatConnect instance or ThreatConnect API user account, it is recommended to set the Batch parameter’s value to 5000 and then reduce it by half each time the integration runs until you reach the desired performance.
  • HTTP Client Timeout: If working with small or busy Elastic Agents, a large number of Indicators on the ThreatConnect instance, or small TQL timeouts for your ThreatConnect instance or API user account, it is recommended to double the HTTP Client Timeout parameter’s initial value of 2m and then double it again each time the integration runs until you reach the desired performance.

Step 2: Validating the Data Flow

After you configure and deploy the ThreatConnect integration in Kibana, the Integration policies tab of the ThreatConnect integration page will open (Figure 7). This tab lists all instances of the integration across all agent policies in Kibana. Instances of the integration without Elastic Agents will not run, and the Agents column will display the Add Agent button for those integration instances.

Figure 7_Elastic Security Integration User Guide_Software Version 1.0

 

From the Integrations policies tab, you can also manage integrations by clicking the ⋯ menu in the Actions column and selecting one of the following options: Add Agent, Edit integration, and Delete integration.

Once the ThreatConnect integration is deployed, it will run automatically. There are four areas in Elastic where you can verify whether Elastic Security is ingesting ThreatConnect data:

Fleet
  1. In Kibana, open the main menu and navigate to Management > Fleet.
  2. On the Fleet page, select the Data streams tab to see the activity of the data streams on the Elastic Stack (Figure 8). The ThreatConnect integration dataset will be named ti_threatconnect.indicator, and the data from ThreatConnect will come in as logs. A good sign that Elastic Security is ingesting data is if the log size is in megabytes.
    Figure 8_Elastic Security Integration User Guide_Software Version 1.0

     

  3. You can also observe data on the managed [Logs ThreatConnect] Indicator dashboard in Kibana that is installed with the integration (Figure 9). To view this dashboard, click the ⋯ menu in the Actions column on the Data streams tab for ti_threatconnect.indicator and select View dashboard.

     

  4. By default, the [Logs ThreatConnect] Indicator dashboard’s time range is set to Last 15 minutes. If the count of Indicators and associated Groups on the dashboard is 0, update the time range to match the value entered for the ThreatConnect integration’s Initial Interval parameter. In most cases, updating the time range will result in the dashboard displaying data (Figure 10).
    Note
    To quickly validate whether Elastic Security has ingested data, set the time range to Last 90 days.

     

  5. (Optional) Because the [Logs ThreatConnect] Indicator dashboard is a managed dashboard, its visualizations are set statically and cannot be changed. To clone the dashboard and create a copy of it that you can edit, click Clone at the top right of the dashboard.
Discover
  1. In Kibana, open the main menu and navigate to Analytics > Discover.
  2. Select the logs-*data view from the data view menu at the top left of the Discover page (Figure 11). Note that this data view will be included in the list of available data views by default.

     

  3. Because the logs-* data view is generic, there may be too many data coming from all of the Elastic Stack logging files to easily find the ThreatConnect data. If you encounter this issue, click Create a data view in the Data views section of the data view menu to create the data view for the ThreatConnect data shown in Figure 12.

     

    • Name: Enter ThreatConnect Data in the text box.
    • Index pattern: Enter logs-ti_threatconnect* in the text box. After this value has been entered, matching log files will be displayed on the right side of the Create data view drawer.
      Note
      In Figure 12, the logs-ti_threatconnect.indicator-default source is the normal data stream, and the portion after the “-” character will match the value you entered for the integration’s Namespace parameter. Additionally, the logs-ti_threatconnect.dest_indicator-1 source is where all Indicators will be stored long term.
    • Timestamp field: Keep the default value of @timestamp.
    • Click Save data view to Kibana.
  4. After the ThreatConnect Data data view has been created, it will be selected as the data view on the Discover page automatically. If the ThreatConnect Data data view displays no data, change the time range from the default value of Last 15 minutes to the value entered for the integration’s Initial Interval parameter.
    Note
    To quickly validate whether Elastic Security has ingested data, set the time range to Last 90 days.
  5. When the ThreatConnect Data data view displays data, the Documents table will be populated with a list of documents associated with the integration. To view a document’s full details, click ExpandExpand icon_Elastic Discoverfor the document. If the document contains a threat.indicator.reference field that shows the URL for the Indicator’s Details screen in ThreatConnect (Figure 13), then Elastic Security is ingesting data and the integration has been deployed successfully.
    Important

    If an expanded document contains an error.message field that is populated with data, there may be a configuration error or connection failure between Elastic Security and ThreatConnect. In this case, proceed to the “Integration Troubleshooting” section for further instruction. The following is the most common error message that may be displayed in the error.message field:

    "error.message": [
        "failed eval: internal error: runtime error: invalid memory address or nil pointer dereference",
        "Processor json with tag json_event_original in pipeline logs-ti_threatconnect.indicator-0.2.0 failed with message: field [original] not present as part of path [event.original]"
    ]

     

Index Management
  1. In Kibana, open the main menu and navigate to Management > Stack Management > Index Management.
  2. On the Indices tab of the Index Management page (Figure 14), locate the logs-ti_threatconnect_latest.dest_indicator-1 index and review the values in the Docs count and Storage size columns. If Doc count is greater than 0 and Storage size is in megabytes, then Elastic Security is ingesting data.

     

Intelligence
  1. In Kibana, open the main menu and navigate to Security > Intelligence.
  2. On the Indicators page, Indicators that Elastic Security has ingested will be displayed in the Trend section and the table below it. If the Indicators page displays no data, change the time range from the default value of Last 15 minutes to the value entered for the integration’s Initial Interval parameter.

Integration Troubleshooting

If you experience issues with the Elastic Security ThreatConnect integration (e.g., Elastic Security is not ingesting data), follow these steps to obtain log files that you can use to troubleshoot the integration:

  1. In Kibana, open the main menu and navigate to Management > Integrations > Installed integrations. Then click the ThreatConnect card.
  2. On the ThreatConnect integration page, select the Integration policies tab.
  3. Click the ⋯ menu in the Actions column for the desired integration policy and select Edit integration.
  4. On the Edit ThreatConnect integration page, click Change defaults to the right of the Collect ThreatConnect logs via API toggle. Then click Advanced options below the Preserve original event toggle.
  5. Turn on the Enable request tracing toggle, and then click Save integration to redeploy and rerun the integration (Figure 15).

     

  6. In Kibana, open the main menu and navigate to Management > Fleet.
  7. On the Agents tab of the Fleet page, identify the Elastic Agent host running the ThreatConnect integration and click the host’s name in the Host column (Figure 16). The host running the ThreatConnect integration will have the same agent policy on which the integration is running.

     

  8. On the host page, select the Diagnostics tab. Then click Request diagnostics .zip.
  9. Once the diagnostics ZIP file is generated, it will be displayed in the table below the Request diagnostics .zip button (Figure 17). Click the diagnostics ZIP file's name to download the file.

     

  10. Unzip the diagnostics ZIP file once it is downloaded, and then find the cel folder containing the http-request-trace logs (Figure 18). Note that each trace log is tied to a specific ThreatConnect instance URL. Each time the ThreatConnect instance URL is changed in the URL parameter and the integration runs, there will be a new trace log associated with that URL. The end of the trace log’s filename contains the specific ThreatConnect URL to which the trace log corresponds. Also, if multiple ThreatConnect integrations are running on a single host, the trace logs for each integration will be included in the ZIP file.

     

  11. Open the trace log that corresponds to the URL of the ThreatConnect instance from which the integration should be retrieving data.
    Note
    Because the trace logs are in JSON format, it is recommended to format the logs with a code editor or JSON formatter tool for better viewing.

The following code sample shows an example of a formatted trace log. The first JSON object is the request, and the second JSON object is the response. Note that the entire request is shown in the url.original field and can be used in an API testing tool to verify whether there is an issue with the API request. The second JSON object contains the http.response.status_code and http.response.body.content fields, both of which can be used to debug issues with the API request.

{
    "log.level": "debug",
    "@timestamp": "2024-03-04T15:54:30.102-0500",
    "message": "HTTP request",
    "transaction.id": "CP0QH8NVL6SHE-12",
    "url.original": "https://companyabc.threatconnect.com/api/v3/indicators?fields=associatedGroups&fields=associatedIndicators&fields=attributes&fields=securityLabels&fields=sightings&fields=tags&fields=threatAssess&resultLimit=5000&resultStart=0&sorting=lastModified+asc&tql=lastModified+GEQ+%222024-02-26T20%3A48%3A23Z%22",
    "url.scheme": "https",
    "url.path": "/api/v3/indicators",
    "url.domain": "companyabc.threatconnect.com",
    "url.port": "",
    "url.query": "fields=associatedGroups&fields=associatedIndicators&fields=attributes&fields=securityLabels&fields=sightings&fields=tags&fields=threatAssess&resultLimit=5000&resultStart=0&sorting=lastModified+asc&tql=lastModified+GEQ+%222024-02-26T20%3A48%3A23Z%22",
    "http.request.method": "GET",
    "user_agent.original": "",
    "http.request.body.content": "",
    "http.request.body.bytes": 0,
    "http.request.mime_type": "",
    "event.original": "GET /api/v3/indicators?fields=associatedGroups&fields=associatedIndicators&fields=attributes&fields=securityLabels&fields=sightings&fields=tags&fields=threatAssess&resultLimit=5000&resultStart=0&sorting=lastModified+asc&tql=lastModified+GEQ+%222024-02-26T20%3A48%3A23Z%22 HTTP/1.1\r\nHost: companyabc.threatconnect.com\r\nUser-Agent: Go-http-client/1.1\r\nAuthorization: TC 70295075712214205425:b7RBP1UMhpVOeOGEUeEXuwL5A9/9Lk3sEFbaOBJNWXE=\r\nTimestamp: 1709585303\r\nAccept-Encoding: gzip\r\n\r\n",
    "ecs.version": "1.6.0"
}

{
    "log.level": "debug",
    "@timestamp": "2024-03-04T15:54:30.127-0500",
    "message": "HTTP response",
    "transaction.id": "CP0QH8NVL6SHE-12",
    "http.response.status_code": 401,
    "http.response.body.content": "{\"message\":\"ERROR (Unauthorized): You are not authorized to perform the requested operation.\",\"status\":\"Error\"}",
    "http.response.body.bytes": 111,
    "http.response.mime_type": "application/json",
    "event.original": "HTTP/1.1 401 Unauthorized\r\nConnection: close\r\nTransfer-Encoding: chunked\r\nAccess-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS\r\nAccess-Control-Allow-Origin: *\r\nCache-Control: no-Store,no-Cache\r\nContent-Type: application/json\r\nDate: Mon, 04 Mar 2024 20:54:30 GMT\r\nPragma: no-Cache\r\nServer: nginx\r\nStrict-Transport-Security: max-age=31536000;\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-Xss-Protection: 1; mode=block\r\n\r\n",
    "ecs.version": "1.6.0"
}

See Table 2 for a list of common HTTP status codes and errors that you may encounter when troubleshooting issues with the integration.

 

http.response.status_codehttp.response.body.contentExplanation
400"message\":\"Error processing request. Make sure the URI is encoded. Due to: calculated signature and header signature mismatch."The ThreatConnect instance URL, the API user’s Access ID, or the API user’s Secret Key was entered incorrectly when the integration was configured.
401"ERROR (Unauthorized): You are not authorized to perform the requested operation."There is something wrong with the authentication of the request. Check the values entered for the integration’s Access ID and Secret Key parameters.
404"Resource not found, check the requested path. Please review your request and try again. If the problem persists, contact technical support."The value for the integration’s URL parameter was not entered properly. For example, if the ThreatConnect instance URL is https:/companyabc.threatconnect.com and https://companyabc.threatconnect.com/api is entered into the integration’s URL parameter, an additional /api will be included in the request URL (i.e., https://companyabc.threatconnect.com/api/api).
-"net/http: request canceled".The integration is canceling the HTTP request to the ThreatConnect instance. One way to fix this is to double the value of the integration’s HTTP Client Timeout parameter. Alternatively, decrease the value of the integration’s Batch Size parameter to 1000. Reducing the amount of data per request can increase the transaction rate and prevent requests from reaching the timeout. You may also increase the value of the HTTP Client Timeout parameter and decrease the value of the Batch Size parameter simultaneously to further fine-tune the data retrieval performance.

If the API request was successful and data are flowing from ThreatConnect to Elastic Security, the trace log will look like the following code sample, where there are multiple requests in a row that include a 200 HTTP status code and no response body. The JSON object containing the response body will be shown only when there is an issue with the API request.

{
    "log.level": "debug",
    "@timestamp": "2024-04-30T21:00:18.938Z",
    "message": "HTTP request",
    "transaction.id": "1C01L1K15B5HE-1",
    "url.original": "https://companyabc.threatconnect.com/api/v3/indicators?fields=associatedGroups&fields=associatedIndicators&fields=attributes&fields=securityLabels&fields=sightings&fields=tags&fields=threatAssess&resultLimit=200&resultStart=0&sorting=lastModified+asc&tql=lastModified+GEQ+%222024-04-30T20%3A10%3A17Z%22+AND+ownerName+in+%28%22CAL+Automated+Threat+Library%22%29+and+typeName+in+%28%22URL%22%2C+%22EmailAddress%22%29+and+dateAdded+%3E+%2204-01-2024%22",
    "url.scheme": "https",
    "url.path": "/api/v3/indicators",
    "url.domain": "companyabc.threatconnect.com",
    "url.port": "",
    "url.query": "fields=associatedGroups&fields=associatedIndicators&fields=attributes&fields=securityLabels&fields=sightings&fields=tags&fields=threatAssess&resultLimit=200&resultStart=0&sorting=lastModified+asc&tql=lastModified+GEQ+%222024-04-30T20%3A10%3A17Z%22+AND+ownerName+in+%28%22CAL+Automated+Threat+Library%22%29+and+typeName+in+%28%22URL%22%2C+%22EmailAddress%22%29+and+dateAdded+%3E+%2204-01-2024%22",
    "http.request.method": "GET",
    "user_agent.original": "",
    "http.request.body.content": "",
    "http.request.body.bytes": 0,
    "http.request.mime_type": "",
    "event.original": "GET /api/v3/indicators?fields=associatedGroups&fields=associatedIndicators&fields=attributes&fields=securityLabels&fields=sightings&fields=tags&fields=threatAssess&resultLimit=200&resultStart=0&sorting=lastModified+asc&tql=lastModified+GEQ+%222024-04-30T20%3A10%3A17Z%22+AND+ownerName+in+%28%22CAL+Automated+Threat+Library%22%29+and+typeName+in+%28%22URL%22%2C+%22EmailAddress%22%29+and+dateAdded+%3E+%2204-01-2024%22 HTTP/1.1\r\nHost: companyabc.threatconnect.com\r\nUser-Agent: Go-http-client/1.1\r\nAuthorization: TC 51856757511267639807:8/ZGiYpkF0odcwePmA/yAkZCHJF7mrHaaVe50hQIp5k=\r\nTimestamp: 1714510818\r\nAccept-Encoding: gzip\r\n\r\n",
    "ecs.version": "1.6.0"
}

{
    "log.level": "debug",
    "@timestamp": "2024-04-30T21:00:19.276Z",
    "message": "HTTP response",
    "transaction.id": "1C01L1K15B5HE-1",
    "http.response.status_code": 200,
    "http.response.body.content": "{\"data\":[],\"status\":\"Success\"}",
    "http.response.body.bytes": 30,
    "http.response.mime_type": "application/json",
    "event.original": "HTTP/1.1 200 OK\r\nConnection: close\r\nTransfer-Encoding: chunked\r\nAccess-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS\r\nAccess-Control-Allow-Origin: *\r\nCache-Control: no-Store,no-Cache\r\nContent-Type: application/json\r\nDate: Tue, 30 Apr 2024 21:00:19 GMT\r\nPragma: no-Cache\r\nServer: nginx\r\nStrict-Transport-Security: max-age=31536000;\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-Xss-Protection: 1; mode=block\r\n\r\n",
    "ecs.version": "1.6.0"
}

Updating the Integration

Follow these steps to update the Elastic Security ThreatConnect integration in Kibana:

  1. In Kibana, open the main menu and navigate to Management > Integrations > Installed integrations.
  2. If an update is available for the ThreatConnect integration, the ThreatConnect card will display an Update available label. Click the ThreatConnect card to open the ThreatConnectintegration page.
    Note
    On the ThreatConnect integration page, aUpdate Available icon_Elastic Kibanaicon will be displayed next to the version number if an update is available.
  3. Select the Settings tab on the ThreatConnect integration page. On this tab, you can view the version of the integration that is currently installed, the latest version of the integration that is available, and a changelog for the latest version (Figure 19). To upgrade the integration to the latest version, click Upgrade to latest version.
    Figure 19_Elastic Security Integration User Guide_Software Version 1.0

     

    Note
    If you select the Upgrade integration policies checkbox, a Upgrade ThreatConnect and policies window that prompts you to confirm the upgrade will be displayed after you click Upgrade to latest version. Click the Upgrade ThreatConnect and policies button in this window to proceed with the upgrade.

Deleting the Integration

If a situation arises where the Elastic Security ThreatConnect integration needs to be deleted, you can do so through the associated agent policy's page. For example, if the integration’s Initial Interval parameter is not set to the desired value and the integration already ran once, you will need to delete the integration and re-add it because, at this time, it is not possible to reset the Initial Interval timestamp after the integration has been deployed and run.

Follow these steps to delete the ThreatConnect integration in Kibana:

  1. In Kibana, open the main menu and navigate to Management > Fleet.
  2. On the Agents tab, select the agent policy associated with the ThreatConnect integration.
  3. On the agent policy page, click the menu in the Actions column for the ThreatConnect integration and select Delete integration.

Data Mappings

The data mappings in Table 3 illustrate how data are mapped from the ThreatConnect data model to the Elastic Common Scheme (ECS). ThreatConnect fields also come in as Elastic custom fields for reference. Depending on the size of the device you are viewing this article on, you may need to scroll horizontally to view all table columns.

 

ThreatConnect API FieldECS FieldElastic Custom Mapping FieldSample Value
active threat_connect.indicator.active.valueTRUE
activeLocked threat_connect.indicator.active.lockedFALSE
addressthreat.indicator.email.addressthreat_connect.indicator.addressjelod40108@newnime.com
AS Numberthreat.indicator.as.numberthreat_connect.indicator.as_numberASN1234
associatedArtifacts threat_connect.indicator.associated_artifacts{"data": [{"id": 12345,"summary": "71.6.135.131","type": "Address","intelType": "indicator-Address","dateAdded": "2023-03-28T13:49:32Z","derivedLink": true,"hashCode": "DQdYXoSZXiWM2shjCXxdsHTcflvuoi12dli1oo7oFHs="}]}
associatedCases threat_connect.indicator.associated_cases{"data": [{"id": 12345,"xid": "a1a1a1a1-a1a1-a1a1-a1a1-a1a1a1a1a1a1","name": "Test Case","dateAdded": "2022-10-04T12:49:08Z","lastUpdated": "2023-08-28T21:00:30Z","caseOpenTime": "2022-10-04T12:49:08Z","caseOpenUser": {"id": 1,"userName": "djones","firstName": "Douglas","lastName": "Jones","pseudonym": "DMJ","owner": "Demo Organization"},"status": "Open","severity": "High","resolution": "Not Specified","assignee": {"type": "User","data": {"id": 2,"userName": "hhodges","firstName": "Herschel","lastName": "Hodges","pseudonym": "HRH","owner": "Demo Organization"}},"createdBy": {"id": 1,"userName": "djones","firstName": "Douglas","lastName": "Jones","pseudonym": "DMJ","owner": "Demo Organization"},"owner": "Demo Organization","ownerId": 5}]}
associatedGroups.data[].assignments.data.type threat_connect.indicator.associated_groups.data[].assignments.data.typeAssigned
associatedGroups.data[].assignments.data.user.id threat_connect.indicator.associated_groups.data[].assignments.data.user.id12
associatedGroups.data[].body threat_connect.indicator.associated_groups.data[].bodyPlease visit bad.com to see your order and give us all your money
associatedGroups.data[].createdBy.firstName threat_connect.indicator.associated_groups.data[].created_by.first_nameJohn
associatedGroups.data[].createdBy.id threat_connect.indicator.associated_groups.data[].created_by.id3
associatedGroups.data[].createdBy.lastName threat_connect.indicator.associated_groups.data[].created_by.last_nameSmith
associatedGroups.data[].createdBy.owner threat_connect.indicator.associated_groups.data[].created_by.ownerDemo Organization
associatedGroups.data[].createdBy.pseudonym threat_connect.indicator.associated_groups.data[].created_by.pseudonymjsmith
associatedGroups.data[].createdBy.userName threat_connect.indicator.associated_groups.data[].created_by.user_namejsmith@companyabc.com
associatedGroups.data[].dateAdded threat_connect.indicator.associated_groups.data[].date_added2021-11-03T14:57:45Z
associatedGroups.data[].documentDateAdded threat_connect.indicator.associated_groups.data[].document.date_added2021-10-21T19:54:59Z
associatedGroups.data[].documentType threat_connect.indicator.associated_groups.data[].document.typeText
associatedGroups.data[].downVoteCount threat_connect.indicator.associated_groups.data[].down_vote_count0
associatedGroups.data[].dueDate threat_connect.indicator.associated_groups.data[].due_date2021-09-17T12:50:19Z
associatedGroups.data[].emailDate threat_connect.indicator.associated_groups.data[].email_date2021-09-17T12:50:19Z
associatedGroups.data[].escalationDate threat_connect.indicator.associated_groups.data[].escalation_date2021-09-17T12:50:19Z
associatedGroups.data[].eventDate threat_connect.indicator.associated_groups.data[].event_date2021-09-17T12:50:19Z
associatedGroups.data[].externalDateAdded threat_connect.indicator.associated_groups.data[].external.date.added2021-09-17T12:50:19Z
associatedGroups.data[].externalDateExpires threat_connect.indicator.associated_groups.data[].external.date.expires2021-09-17T12:50:19Z
associatedGroups.data[].externalLastModified threat_connect.indicator.associated_groups.data[].external.last_modified2021-09-17T12:50:19Z
associatedGroups.data[].fileName threat_connect.indicator.associated_groups.data[].file.nameindicators.txt
associatedGroups.data[].fileSize threat_connect.indicator.associated_groups.data[].file.size36
associatedGroups.data[].fileText threat_connect.indicator.associated_groups.data[].file.textFile text
associatedGroups.data[].fileType threat_connect.indicator.associated_groups.data[].file.typeHash
associatedGroups.data[].firstSeen threat_connect.indicator.associated_groups.data[].first_seen2021-09-17T12:50:19Z
associatedGroups.data[].from threat_connect.indicator.associated_groups.data[].fromauto-confirm@bad.com
associatedGroups.data[].header threat_connect.indicator.associated_groups.data[].headerDelivered-To: smithj@threatconnect.com
Received: by…
associatedGroups.data[].id threat_connect.indicator.associated_groups.data[].id6
associatedGroups.data[].lastModified threat_connect.indicator.associated_groups.data[].last_modified2021-10-21T19:54:59Z
associatedGroups.data[].lastSeen threat_connect.indicator.associated_groups.data[].last_seen2021-09-17T12:50:19Z
associatedGroups.data[].legacyLink threat_connect.indicator.associated_groups.data[].legacy_linkhttps://companyabc.threatconnect.com/auth/document/document.xhtml?document=10
associatedGroups.data[].malware threat_connect.indicator.associated_groups.data[].malwareTRUE
associatedGroups.data[].name threat_connect.indicator.associated_groups.data[].nameBad Incident
associatedGroups.data[].ownerId threat_connect.indicator.associated_groups.data[].owner.id1
associatedGroups.data[].ownerName threat_connect.indicator.associated_groups.data[].owner.nameDemo Organization
associatedGroups.data[].password threat_connect.indicator.associated_groups.data[].passwordduwyhfsjhsi
associatedGroups.data[].publishDate threat_connect.indicator.associated_groups.data[].publish_date2021-09-17T12:50:19Z
associatedGroups.data[].reminderDate threat_connect.indicator.associated_groups.data[].reminder_date2021-09-17T12:50:19Z
associatedGroups.data[].scoreBreakdown threat_connect.indicator.associated_groups.data[].score_breakdownRule SPF Neutral was matched against 'neutral'.
associatedGroups.data[].scoreIncludesBody threat_connect.indicator.associated_groups.data[].score_includes_bodyTRUE
associatedGroups.data[].status threat_connect.indicator.associated_groups.data[].statusNew
associatedGroups.data[].subject threat_connect.indicator.associated_groups.data[].subjectYour Amazon.com order for demo@sample.com
associatedGroups.data[].to threat_connect.indicator.associated_groups.data[].todemo@sample.com
associatedGroups.data[].type threat_connect.indicator.associated_groups.data[].typeIncident
associatedGroups.data[].upVote threat_connect.indicator.associated_groups.data[].up_voteFALSE
associatedGroups.data[].upVoteCount threat_connect.indicator.associated_groups.data[].up_vote_count0
associatedGroups.data[].webLink threat_connect.indicator.associated_groups.data[].web_linkhttps://companyabc.threatconnect.com/#/details/groups/3/overview
associatedGroups.data[].xid threat_connect.indicator.associated_groups.data[].xida1a1a1a1-a1a1-a1a1-a1a1-a1a1a1a1a1a1
associatedGroups.generatedReport  TRUE
associatedIndicators.data[].active threat_connect.indicator.associated_indicators.data[].active.valueTRUE
associatedIndicators.data[].activeLocked threat_connect.indicator.associated_indicators.data[].active.lockedFALSE
associatedIndicators.data[].address threat_connect.indicator.associated_indicators.data[].addressjelod40108@newnime.com
associatedIndicators.data[].AS Number threat_connect.indicator.associated_indicators.data[].as_numberASN1234
associatedIndicators.data[].Block threat_connect.indicator.associated_indicators.data[].block216.160.83.56/29
associatedIndicators.data[].confidence threat_connect.indicator.associated_indicators.data[].confidence20
associatedIndicators.data[].dateAdded threat_connect.indicator.associated_indicators.data[].date_added2021-11-02T13:07:08Z
associatedIndicators.data[].description threat_connect.indicator.associated_indicators.data[].descriptionA bad email address
associatedIndicators.data[].dnsActive threat_connect.indicator.associated_indicators.data[].dns_activeFALSE
associatedIndicators.data[].externalDateAdded threat_connect.indicator.associated_indicators.data[].external.date.added2023-10-04T12:34:56Z
associatedIndicators.data[].externalDateExpires threat_connect.indicator.associated_indicators.data[].external.date.expires2023-10-04T12:34:56Z
associatedIndicators.data[].externalLastModified threat_connect.indicator.associated_indicators.data[].external.last_modified2023-10-04T12:34:56Z
associatedIndicators.data[].firstSeen threat_connect.indicator.associated_indicators.data[].first_seen2023-10-04T12:34:56Z
associatedIndicators.data[].Hashtag threat_connect.indicator.associated_indicators.data[].hashtag#testabc
associatedIndicators.data[].hostName threat_connect.indicator.associated_indicators.data[].host_namesamplehost.com
associatedIndicators.data[].id threat_connect.indicator.associated_indicators.data[].id10
associatedIndicators.data[].ip threat_connect.indicator.associated_indicators.data[].ip175.16.199.22
associatedIndicators.data[].Key Name threat_connect.indicator.associated_indicators.data[].key_nameHKEY_LOCAL_MACHINE\\SYSTEM\\Setup\\Status\\ChildCompletion
associatedIndicators.data[].lastModified threat_connect.indicator.associated_indicators.data[].last_modified"2021-11-02T13:07:08Z"
associatedIndicators.data[].lastSeen threat_connect.indicator.associated_indicators.data[].last_seen2023-10-04T12:34:56Z
associatedIndicators.data[].legacyLink threat_connect.indicator.associated_indicators.data[].legacy_linkhttps://companyabc.threatconnect.com/auth/indicators/details/file.xhtml?file=F5A2496CF66CB8CFFE66CB1B27D7DEDE&owner=Demo+Organization
associatedIndicators.data[].md5 threat_connect.indicator.associated_indicators.data[].md5F5A2496CF66CB8CFFE66CB1B27D7DEDE
associatedIndicators.data[].Mutex threat_connect.indicator.associated_indicators.data[].mutexTest.Mutex()
associatedIndicators.data[].ownerId threat_connect.indicator.associated_indicators.data[].owner.id1
associatedIndicators.data[].ownerName threat_connect.indicator.associated_indicators.data[].owner.nameDemo Organization
associatedIndicators.data[].privateFlag threat_connect.indicator.associated_indicators.data[].private_flagFALSE
associatedIndicators.data[].rating threat_connect.indicator.associated_indicators.data[].rating3
associatedIndicators.data[].sha1 threat_connect.indicator.associated_indicators.data[].sha1640AB2BAE07BEDC4C163F679A746F7AB7FB5D1FA
associatedIndicators.data[].sha256 threat_connect.indicator.associated_indicators.data[].sha2567D5FFFBFE8D098E369466164F705B4D692517A2B4659A03901DAF67CF78553FB
associatedIndicators.data[].size threat_connect.indicator.associated_indicators.data[].size123
associatedIndicators.data[].Subject threat_connect.indicator.associated_indicators.data[].subjectSpam
associatedIndicators.data[].summary threat_connect.indicator.associated_indicators.data[].summaryF5A2496CF66CB8CFFE66CB1B27D7DEDE
associatedIndicators.data[].text threat_connect.indicator.associated_indicators.data[].texthttp://www.testingmcafeesites.com/testcat_pc.html
associatedIndicators.data[].type threat_connect.indicator.associated_indicators.data[].typeFile
associatedIndicators.data[].User Agent String threat_connect.indicator.associated_indicators.data[].user_agent_stringPostmanRuntime/7.32.3
associatedIndicators.data[].Value Name threat_connect.indicator.associated_indicators.data[].value.name0
associatedIndicators.data[].Value Type threat_connect.indicator.associated_indicators.data[].value.typeREG_QWORD
associatedIndicators.data[].webLink threat_connect.indicator.associated_indicators.data[].web_linkhttps://companyabc.threatconnect.com/#/details/indicators/10/overview
associatedIndicators.data[].whoisActive threat_connect.indicator.associated_indicators.data[].whois_activeTRUE
attributes.data[].createdBy.firstName threat_connect.indicator.attributes.data[].created_by.first_nameJohn
attributes.data[].createdBy.id threat_connect.indicator.attributes.data[].created_by.id3
attributes.data[].createdBy.lastName threat_connect.indicator.attributes.data[].created_by.last_nameSmith
attributes.data[].createdBy.owner threat_connect.indicator.attributes.data[].created_by.ownerDemo Organization
attributes.data[].createdBy.pseudonym threat_connect.indicator.attributes.data[].created_by.pseudonymjsmith
attributes.data[].createdBy.userName threat_connect.indicator.attributes.data[].created_by.user_namesmithj@companyabc.com
attributes.data[].dateAdded threat_connect.indicator.attributes.data[].dateAdded2021-11-09T13:32:37Z
attributes.data[].default threat_connect.indicator.attributes.data[].defaultFALSE
attributes.data[].id threat_connect.indicator.attributes.data[].id10
attributes.data[].lastModified threat_connect.indicator.attributes.data[].last_modified2021-11-09T13:32:37Z
attributes.data[].pinned threat_connect.indicator.attributes.data[].pinnedFALSE
attributes.data[].source threat_connect.indicator.attributes.data[].sourcePhase of Intrusion
attributes.data[].type threat_connect.indicator.attributes.data[].typeAdditional Analysis and Context
attributes.data[].value threat_connect.indicator.attributes.data[].valueHost used by hacker conglomerate traced to Iran.
Block threat_connect.indicator.block216.160.83.56/29
confidence threat_connect.indicator.confidence20
customAssociations threat_connect.indicator.custom_associations{"data": [{"id": 12345,"dateAdded": "2018-09-18T17:46:56Z","ownerId": 5,"ownerName": "Demo Organization","webLink": "https://companyabc.threatconnect.com/#/details/indicators/12345","type": "Address","lastModified": "2024-01-18T15:27:41Z","summary": "209.15.13.134","privateFlag": false,"active": true,"activeLocked": false,"ip": "209.15.13.134","legacyLink": "https://companyabc.threatconnect.com/auth/indicators/details/address.xhtml?address=209.15.13.134&owner=Demo+Organization"}]}
dateAdded threat_connect.indicator.date_added2021-11-02T13:07:08Z
descriptionthreat.indicator.descriptionthreat_connect.indicator.descriptionA bad email address
dnsActive threat_connect.indicator.dns_activeFALSE
dnsResolution threat_connect.indicator.dns_resolution{"data": [{"addresses": {"data": [{"id": 9962038,"ip": "162.55.172.212","geoLocation": {"country": "Germany","countryCode": "DE","state": "Rheinland-Pfalz","timezone": "Europe/Berlin","city": "Mammelzen","asn": 24940,"registeringOrg": "Hetzner Online GmbH","lat": 50.7,"lon": 7.6661}}],"count": 0},"dateOccurred": "2023-07-06T14:08:31Z"}],"count": 0}
enrichment threat_connect.indicator.enrichment{"data": [{"type": "VirusTotal","vtMaliciousCount": 14}]}
externalDateAdded threat_connect.indicator.external.date.added2023-10-04T12:34:56Z
externalDateExpires threat_connect.indicator.external.date.expires2023-10-04T12:34:56Z
externalLastModifiedthreat.indicator.modified_atthreat_connect.indicator.external_last.modified2023-10-04T12:34:56Z
falsePositiveReportedByUser threat_connect.indicator.false_positive_reported_by_userFALSE
falsePositives threat_connect.indicator.false_positives1
fileActions threat_connect.indicator.file_actions{"data": [{"relationship": "File DNS Query","indicator": {"id": 54321,"dateAdded": "2018-09-18T17:46:56Z","ownerId": 5,"ownerName": "Demo Organization","webLink": "https://companyabc.threatconnect.com/#/details/indicators/54321","type": "Host","lastModified": "2024-05-09T18:48:34Z","rating": 0.00,"confidence": 0,"source": "Host used by hacker conglomerate traced to China.","description": "Malicious actor with ties to hacker networks.","summary": "badguy.com","privateFlag": false,"active": false,"activeLocked": false,"fileActions": {"count": 0},"hostName": "badguy.com","dnsActive": true,"whoisActive": true,"legacyLink": "https://companyabc.threatconnect.com/auth/indicators/details/host.xhtml?host=badguy.com&owner=Demo+Organization"}}],"count": 1}
fileOccurrences threat_connect.indicator.file_occurrences{"data": [{"fileName": "win999301.dll","path": "C:\\\\Windows\\System","date": "2022-06-14T10:00:00Z"}]}
firstSeenthreat.indicator.first_seenthreat_connect.indicator.first_seen2023-10-04T12:34:56Z
genericCustomIndicatorValues threat_connect.indicator.generic_custom_indicator_valuesHKEY_CURRENT_USER\\Software\\MyApp
geoLocation threat_connect.indicator.geo_location{"country": "Germany","countryCode": "DE","state": "Land Berlin","timezone": "Europe/Berlin","city": "Berlin","asn": 24940,"registeringOrg": "Hetzner Online GmbH","lat": 52.5203,"lon": 13.3849}
Hashtag threat_connect.indicator.hashtag#testabc
hostNameobserver.hostnamethreat_connect.indicator.host_namesamplehost.com
id threat_connect.indicator.id10
investigationLinks threat_connect.indicator.investigation_links{"data": [{"name": "#totalhash","path": "https://totalhash.cymru.com/search/?ip:49.13.76.144"}]}
ipthreat.indicator.ipthreat_connect.indicator.ip175.16.199.22
Key Namethreat.indicator.registry.paththreat_connect.indicator.key_nameHKEY_LOCAL_MACHINE\\SYSTEM\\Setup\\Status\\ChildCompletion
lastFalsePositive threat_connect.indicator.last_false_positive2023-10-04T12:34:56Z
lastModified@timestampthreat_connect.indicator.last_modified"2021-11-02T13:07:08Z"
lastSeenthreat.indicator.last_seenthreat_connect.indicator.last_seen2023-10-04T12:34:56Z
legacyLinkthreat.indicator.referencethreat_connect.indicator.legacy_linkhttps://companyabc.threatconnect.com/auth/indicators/details/file.xhtml?file=F5A2496CF66CB8CFFE66CB1B27D7DEDE&owner=Demo+Organization
md5threat.indicator.file.hash.md5threat_connect.indicator.md5F5A2496CF66CB8CFFE66CB1B27D7DEDE
Mutex threat_connect.indicator.mutexTest.Mutex()
observations threat_connect.indicator.observations2
ownerId threat_connect.indicator.owner.id1
ownerName threat_connect.indicator.owner.nameDemo Organization
privateFlag threat_connect.indicator.private_flagFALSE
rating threat_connect.indicator.rating3
securityLabels.data[].color threat_connect.indicator.security_labels.data[].colorFFFFFF
securityLabels.data[].dateAdded threat_connect.indicator.security_labels.data[].date_added2016-08-31T00:00:00Z
securityLabels.data[].description threat_connect.indicator.security_labels.data[].descriptionThis Security Label is used for information that carries minimal risk
securityLabels.data[].id threat_connect.indicator.security_labels.data[].id1
securityLabels.data[].name threat_connect.indicator.security_labels.data[].nameTLP:WHITE
securityLabels.data[].owner threat_connect.indicator.security_labels.data[].ownerSystem
sha1threat.indicator.file.hash.sha1threat_connect.indicator.sha1640AB2BAE07BEDC4C163F679A746F7AB7FB5D1FA
sha256threat.indicator.file.hash.sha256threat_connect.indicator.sha2567D5FFFBFE8D098E369466164F705B4D692517A2B4659A03901DAF67CF78553FB
sizethreat.indicator.file.sizethreat_connect.indicator.size123
source threat_connect.indicator.sourcehttps://fp.tools/api/v4/indicators/attribute/pN0psYjPUQ6a_sxPSW5XjQ
Subjectemail.subjectthreat_connect.indicator.subjectSpam
summaryevent.idthreat_connect.indicator.summaryF5A2496CF66CB8CFFE66CB1B27D7DEDE
tags.data[].description threat_connect.indicator.tags.data[].descriptionApply this Tag to objects related to ransomware attacks
tags.data[].id threat_connect.indicator.tags.data[].id1
tags.data[].lastUsed threat_connect.indicator.tags.data[].last_used2023-08-25T13:15:30Z
tags.data[].name threat_connect.indicator.tags.data[].nameuser execution: malicious link
tags.data[].owner threat_connect.indicator.tags.data[].ownerDemo Organization
tags.data[].platforms.count threat_connect.indicator.tags.data[].platforms.count1
tags.data[].platforms.data threat_connect.indicator.tags.data[].platforms.data["Windows"]
tags.data[].techniqueId threat_connect.indicator.tags.data[].technique.idT1055.005
textthreat.indicator.url.originalthreat_connect.indicator.texthttp://www.testingmcafeesites.com/testcat_pc.html
threatAssessConfidence threat_connect.indicator.threat_assess.confidence58.1
threatAssessRating threat_connect.indicator.threat_assess.rating4.5
threatAssessScore threat_connect.indicator.threat_assess.score.value678
threatAssessScoreFalsePositive threat_connect.indicator.threat_assess.score.false_positive0
threatAssessScoreObserved threat_connect.indicator.threat_assess.score.observed139
trackedUsers threat_connect.indicator.tracked_users{"jsmith API": {"observations": 0,"falsePositives": 0}}
typethreat.indicator.typethreat_connect.indicator.typeFile
User Agent Stringuser_agent.originalthreat_connect.indicator.user_agent_stringPostmanRuntime/7.32.3
Value Namethreat.indicator.registry.valuethreat_connect.indicator.value.name0
Value Typethreat.indicator.registry.data[].typethreat_connect.indicator.value.typeREG_QWORD
webLinkthreat.indicator.referencethreat_connect.indicator.web_linkhttps://companyabc.threatconnect.com/#/details/indicators/10/overview
whoIs threat_connect.indicator.who_is{"data": [{"fullRecord": "Domain Name: ZEVERCO.COM\n      Registry...","registrant": "Dynadot Inc","created": "2024-01-30T00:00:00Z","updated": "2024-01-30T00:00:00Z","expires": "2025-01-30T00:00:00Z","contacts": [{"type": "Registrant","name": "REDACTED FOR PRIVACY","organization": "Super Privacy Service LTD c/o Dynadot","street1": "PO Box 701","city": "San Mateo","state": "California","postalCode": "94401","country": "UNITED STATES","phone": "16505854708"}]}],"count": 0}
whoisActive threat_connect.indicator.whois_activeTRUE
 event.kind event
 event.category threat
 event.type indicator

Frequently Asked Questions (FAQ)

How do I debug the Elastic Security ThreatConnect integration if it is not working as expected?

See the “Integration Troubleshooting” section for instructions on how to troubleshoot issues with the Elastic Security ThreatConnect integration.


I receive an error message from the log data when I inspect them with Kibana Discover. What should I do?

The following is the most common error message that occurs in the Elastic Security ThreatConnect integration:

"error.message": [
   "failed eval: internal error: runtime error: invalid memory address or nil pointer dereference",
   "Processor json with tag json_event_original in pipeline logs-ti_threatconnect.indicator-0.2.0 failed with message: field [original] not present as part of path [event.original]"
]

This is a generic error that usually signifies a configuration or connection failure between Elastic Security and ThreatConnect. If you encounter this error, verify there are no issues with the values entered for following integration configuration parameters:

  • URL: Ensure /api is not included in the URL.
  • ThreatConnect Access ID: Check for extraneous characters (e.g., letters, special characters), as the API user’s Access ID should contain only numbers.
  • Secret Key: If using Elastic Stack version 8.13.4 or older, ensure that the ThreatConnect API user’s Secret Key does not contain the $ character.

If you confirm that there are no issues with the values entered for these parameters, see the “Integration Troubleshooting” section to further troubleshoot the integration.


I do not see data on the built-in Kibana dashboard or in Kibana Discover. What should I do?

Change the time range from the default value of Last 15 minutes to the value entered for the integration’s Initial Interval parameter. Alternatively, set the time range to Last 90 days to quickly validate whether Elastic Security is ingesting ThreatConnect data.

If you update the time range and data are still not displayed, see the “Integration Troubleshooting” section to further troubleshoot the integration.


The value of the ThreatConnect integration’s Initial Interval parameter was too small and Elastic Security did not ingest some ThreatConnect data. What should I do?

Delete the ThreatConnect integration and then redeploy it with a larger value for the Initial Interval parameter. At this time, there is no way to reset the initial interval date for the integration.


Should I add more than one instance of the ThreatConnect integration to an agent policy?

It is recommended that an agent policy have an equal number of Elastic Agents per integration instance. Otherwise, there may be issues with bandwidth on the Elastic Stack. This limitation can also be overcome by having a large Elastic Agent.


Why am I receiving a “Cannot retrieve search results” error when viewing records in Kibana Discover, and how can I resolve this issue?

The “Cannot retrieve search results” error (Figure 20) occurs when the size of a response from ThreatConnect is larger than 10 megabytes. Changing the time range can exclude some of the larger records for a certain time interval as a temporary fix. You can also change the max_async_search_response_size via an API call to the Elastic Cluster.

Figure 20_Elastic Security Integration User Guide_Software Version 1.0

 


ThreatConnect® is a registered trademark of ThreatConnect, Inc.
Elastic®, Filebeat®, and Kibana® are registered trademarks of Elasticsearch BV.

30085-01 EN Rev. A


Was this article helpful?

What's Next