Searching Polarity Integrations

This guide walks through searching integrations via the Polarity REST API

Overview

The Polarity Web Search capability (i.e., browser-based search), is powered by the Polarity REST API.  You can use the same REST API to make similar calls as the Polarity Web Search interface.

The general flow when running searches against the API is as follows:

1. Authenticate to Polarity

2. Parse search text into entities (i.e., Parsed Entity Objects) using /api/parse-entities

3. Lookup Entity Objects against an integration using /api/integrations/<INTEGRATION_ID>/lookup

4. For each entity, this endpoint returns a Lookup Result Object

5. Optionally, run a lookup against the onDetails endpoint (if implemented by the integration), passing in the Lookup Result Object from step 3.

6. Optionally, run additional onMessage lookups to fetch data from additional integration endpoints

Authenticating

To begin, you will need to authenticate to the Polarity Server to obtain a reusable bearer token that will be used with subsequent authenticated requests.

See Authentication for information on how to authenticate to the Polarity Platform and acquire an authentication token.  The authentication token will be used in all the requests outlined below.

Parsing Data

The first step in searching integrations is to parse the text you wish to search.  Text in this case is a string which may contain one or more entities of interest (e.g., IP addresses, domains, URLS etc.)

POST /api/parsed-entities
{
  "text": "Text to parse"
}

Examples

For example, to parse the text "Google DNS is 8.8.8.8" you could use the following:

CURL

curl -v -X POST \
https://<polarity.server.url>/api/parsed-entities \
--header 'Authorization: Bearer <AUTH_TOKEN>' \
--header 'Content-Type: application/vnd.api+json' \
--data '{"data":{"attributes":{"text":"Google DNS is 8.8.8.8"}}}'

Python

import requests
import json

def parse_text(token, host, text):
    url = f'{host}/api/parsed-entities'

    payload = json.dumps({
        'data': {
            'attributes': {
                'text': text
            }
        }
    })
    headers = {
        'Content-Type': 'application/vnd.api+json',
        'Authorization': f'Bearer {token}'
    }

    response = requests.request('POST', url, headers=headers, data=payload)
    response.raise_for_status()

    return response.json()

body = parse_text('TOKEN', 'https://polarity.server.url', 'Google DNS is 8.8.8.8')    

The parsing endpoint will then return a list of parsed entities from the provided text.  The key information are the entities found under data.attributes.entities. This is an array of parsed entity objects which will be passed to the integrations to be searched.

Here is a sample response where the IPv4 address "8.8.8.8" has been parsed from the input text:

{
  "data": {
    "attributes": {
      "annotations": [],
      "entities": [
        {
          "channels": [],
          "display-value": "8.8.8.8",
          "start-index": 0,
          "type": "IPv4",
          "value": "8.8.8.8"
        }
      ]
    },
    "id": "aec6e5a3-f8a9-4339-84ab-09d32f70afc2",
    "type": "parsed-entities"
  },
  "jsonapi": {
    "version": "1.0"
  }
}

Creating Entity Objects Manually

In some cases, you will not need to parse any text as you will be looking up a single indicator or a list of indicators.  In these cases, you can bypass the parse-entities endpoint and generate the parsed entity objects manually.  

As an example, if you wanted to lookup the IP address 8.8.4.4 you could create a simplified parsed entity object like this:

{
  "type": "IPv4",
  "value": "8.8.4.4"
}          

The type attribute is case sensitive

In addition to IPv4, the other supported types are:

• IPv4

• IPv6

• domain

• email

• IPv4CIDR

• url

• MD5

• SHA1

• SHA256

Searching an Integration

Once you have the parsed entity objects, you can send those objects to specific integrations for enrichment using the integration-lookups endpoint:

POST /api/integrations/<INTEGRATION_ID>/lookup
{
    "data":{
         "type": "integration-lookups",
         "attributes": {
             "entities":[
                // one or more entity objects
            ]  
        }   
    }
}

For example, if we have the following entity object returned from the parsed-entities endpoint:

{
    "display-value": "8.8.8.8",
    "value": "8.8.8.8",
    "type": "IPv4",
    "start-index": 0
}

You would send the following request:

{
    "data":{
         "type": "integration-lookups",
         "attributes": {
             "entities":[
                {
                    "display-value": "8.8.8.8",
                    "value": "8.8.8.8",
                    "type": "IPv4",
                    "start-index": 39
                }
            ]  
        }   
    }
}

The INTEGRATION_ID is assigned when an integration is first installed.  The simplest way to find the id for the integration you're interested in is to login via the Polarity Web Interface and check for the ID in the URL of your browser.

Alternatively, you can return all installed integrations and their IDs by querying the /api/integrations endpoint:

CURL

curl -v -X GET \
'https://<polarity.server.url>/api/integrations' \
--header 'Authorization: Bearer <AUTH_TOKEN>' \
--header 'Content-Type: application/vnd.api+json'

Python

import requests
import json

def get_integrations(token, host):
    url = f'{host}/api/integrations'

    payload = {}
    headers = {
        'Content-Type': 'application/vnd.api+json',
        'Authorization': f'Bearer {token}'
    }

    response = requests.request('GET', url, headers=headers, data=payload)
    response.raise_for_status()

    return response.json()
  
integrations = get_integrations(token, 'https://polarity.server.url')    

See Retrieving Integration Information for more details on fetching integration related data.

Examples

As an example, to search the VirusTotal integration for the parsed IP address 8.8.8.8 you could send a payload like this:

CURL

curl -v -X POST \
'https://<polarity.server.url>/api/integrations/virustotal/lookup' \
--header 'Authorization: Bearer <AUTH_TOKEN>' \
--header 'Content-Type: application/vnd.api+json'
--data-binary @- <<EOF
{
    "data":{
         "type": "integration-lookups",
         "attributes": {
             "entities":[
                {
                    "display-value": "8.8.8.8",
                    "value": "8.8.8.8",
                    "type": "IPv4",
                    "start-index": 39
                }
            ]  
        }   
    }
}
EOF

Python

import requests
import json

def search_integration(token, host, integration_id, entity_objects):
    url = f'{host}/api/integrations/{integration_id}/lookup'

    payload = json.dumps({
        'data': {
            'type': 'integration-lookups',
            'attributes': {
                'entities': entity_objects
            }
        }
    })

    headers = {
        'Content-Type': 'application/vnd.api+json',
        'Authorization': f'Bearer {token}'
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    response.raise_for_status()

    return response.json()
  
search_result = search_integration(token, HOST, 'virustotal_3_7_4_node_18_63e5110da4_1697729362', [
    {
        'value': '8.8.8.8',
        'type': 'IPv4'
    }
])  

The resulting data will include a data.attributes.results array which contains a result object for each searched entity. These result objects will include the following top-level keys:

{
  "entity": {} // expanded entity value for the results
  "data": {
    "summary": [] // array of summary tags as strings
    "details": {} // JSON object containing details data from integration
  }
}

If the integration did not have a result for the specified entity than the data object will be null:

// lookup result object with no result for the specified entity
{
  "entity": {},
  "data": null
}

Entities included in the payload that are not supported by an integration will be ignored

Fetching Additional Details (onDetails)

Certain integrations fetch additional data in what is known as the onDetails hook. You can query the onDetails hook using the integration-messages endpoint:

POST /api/integrations/<INTEGRATION_ID/message
{
  "data": {
    "type": "integration-message",
    "attributes": {
      "payload": {
        // the lookup result object should be inserted here
      }
    }  
  },
  "meta": {
    "loadDetailsBlock": true
  }
}

Currently, the fastest way to tell if an integration implements an onDetails hook is to check the bottom of the integration’s integration.js file to see if an onDetails method is exported, as in the example below:

module.exports = {
  doLookup,
  startup,
  onDetails, //<-- includes onDetails
  onMessage,
  validateOptions
};

If this method is exported, then you can use the /api/integrations/<INTEGRATION_ID/message endpoint to query the onDetails data.

The endpoint will return a Lookup Result Object containing any additional data loaded by the onDetails hook.

As of 4/03/2024 the following integrations include an onDetails lookup:

• Sophos

• ServiceNow

• ServiceNow SIR

• RipeStat

• Salesforce

• Slashnext

• ThreatConnect

• Rapid7 Nexpose

• Crowdstrike Intel

• ThreatStream

• ReversingLabs

• Analyst1

• Gigamon

• MISP

• Redmine

• FIR Search

• Splunk SOAR (Phantom)

• Flashpoint

• ThreatQuotient

• HYAS Insight

Per Integration Custom Data (onMessage)

Certain integrations also can return additional data via the onMessage hook.  This is custom per integration and requires an understanding of the integration configuration to implement.  The onMessage data can be queried via the integration-messages endpoint.

As an example, the following query can be sent to retrieve “resolutions” information from the PassiveTotal integration.

POST /v2/integration-messages/passivetotal
{
  "data":{
    "type":"integration-messages",
      "attributes": {
        "payload": {
      "entity": {
            "value": "8.8.8.8"
          },
          "searchType": "pdns"
    }
    }
  }
}