Searching Polarity Integrations

This guide walks through searching integrations via the Polarity REST API

Overview

The Polarity Web Search capability (i.e., browser-based search), is powered by the Polarity REST API.  You can use the same REST API to make similar calls as the Polarity Web Search interface.

The general flow when running searches against the API is as follows:

1. Authenticate to Polarity

2. Parse search text into entities (i.e., Parsed Entity Objects) using /api/parse-entities

3. Lookup Entity Objects against an integration using /api/integrations/<INTEGRATION_ID>/lookup

4. For each entity, this endpoint returns a Lookup Result Object

5. Optionally, run a lookup against the onDetails endpoint (if implemented by the integration), passing in the Lookup Result Object from step 3.

6. Optionally, run additional onMessage lookups to fetch data from additional integration endpoints

Authenticating

To begin, you will need to authenticate to the Polarity Server to obtain a reusable bearer token that will be used with subsequent authenticated requests.

See Authentication for information on how to authenticate to the Polarity Platform and acquire an authentication token.  The authentication token will be used in all the requests outlined below.

Parsing Data

The first step in searching integrations is to parse the text you wish to search.  Text in this case is a string which may contain one or more entities of interest (e.g., IP addresses, domains, URLS etc.)

POST /api/parsed-entities
{
  "text": "Text to parse"
}

Examples

To parse the text "Google DNS is 8.8.8.8" you could use the following curl command or Python code:

curl -v -X POST \
https://<polarity.server.url>/api/parsed-entities \
--header 'Authorization: Bearer <AUTH_TOKEN>' \
--header 'Content-Type: application/vnd.api+json' \
--data '{"data":{"attributes":{"text":"Google DNS is 8.8.8.8"}}}'  

import requests
import json

def parse_text(token, host, text):
    url = f'{host}/api/parsed-entities'

    payload = json.dumps({
        'data': {
            'attributes': {
                'text': text
            }
        }
    })
    headers = {
        'Content-Type': 'application/vnd.api+json',
        'Authorization': f'Bearer {token}'
    }

    response = requests.request('POST', url, headers=headers, data=payload)
    response.raise_for_status()

    return response.json()

body = parse_text('TOKEN', 'https://polarity.server.url', 'Google DNS is 8.8.8.8')    

Be sure to include the Content-Type header with a value of application/vnd.api+json when sending your request.

The parsing endpoint will then return a list of parsed entities from the provided text.  The key information are the entities found under data.attributes.entities. This is an array of parsed entity objects which will be passed to the integrations to be searched.

Here is a sample response where the IPv4 address "8.8.8.8" has been parsed from the input text:

{
  "data": {
    "attributes": {
      "annotations": [],
      "entities": [
        {
          "channels": [],
          "display-value": "8.8.8.8",
          "start-index": 0,
          "type": "IPv4",
          "value": "8.8.8.8"
        }
      ]
    },
    "id": "aec6e5a3-f8a9-4339-84ab-09d32f70afc2",
    "type": "parsed-entities"
  },
  "jsonapi": {
    "version": "1.0"
  }
}

Searching Annotations

The parsed-entities endpoint is also used for fetching annotations (also known as tags) for any entities extracted from the provided text attribute sent to the endpoint.  Annotations are returned on the data.attributes.annotations property.  The annotations property is an array of annotation objects.  Each annotation object includes an array of context objects where each context object includes information about an entity-annotation pair.

The following return payload provides an example which includes the entity “localhost” which has the annotation “127.0.0.1” applied to it.  Note that due to historical naming conventions, the annotation for a particular entity is referred to as a “tag” in the returned payload.

{
  "data": {
    "attributes": {
      "annotations": [
        {
          "channel": {
            "channel-id": -1,
            "channel-name": "General"
          },
          "contexts": [
            {
              "active": true,
              "applied": "2025-01-22T12:54:26Z",
              "channel": {
                "id": -1,
                "name": "General"
              },
              "channel-id": -1,
              "confidence": 0,
              "doi-end": null,
              "doi-start": null,
              "downvote-total": 0,
              "entity": {
                "channel": {
                  "channel-id": -1,
                  "channel-name": "General"
                },
                "entity-name": "localhost",
                "id": 12,
                "is-ip": false,
                "last-tagged": "2025-01-22T12:54:26.789972Z"
              },
              "entity-id": 12,
              "id": 13,
              "num-comments": 0,
              "tag": {
                "created": "2025-01-22T12:54:26.799550Z",
                "id": 11,
                "last-applied": "2025-01-22T12:54:26.789972Z",
                "tag-name": "127.0.0.1"
              },
              "tag-id": 11,
              "upvote-total": 0,
              "user": {
                "id": 1,
                "user-stats": {
                  "id": 1
                }
              },
              "user-id": 1,
              "user-stats": {
                "comment-count": 5,
                "downvote-total": 0,
                "id": 1,
                "tag-count": 10,
                "upvote-total": 0,
                "user": {
                  "id": 1
                },
                "user-id": 1
              },
              "user-stats-id": 1,
              "vote-total": 0
            }
          ],
          "entity": {
            "channel": {
              "channel-id": -1,
              "channel-name": "General"
            },
            "entity-name": "localhost",
            "id": 12,
            "is-ip": false,
            "last-tagged": "2025-01-22T12:54:26.789972Z"
          },
          "entity-name": "localhost",
          "id": 12,
          "is-ip": false,
          "last-tagged": "2025-01-22T12:54:26.789972Z"
        }
      ],
      "entities": [
         //..relevant entities returned    
      ]
    },
    "id": "da679bc0-32a0-4713-8d3f-6e195d4b1268",
    "type": "parsed-entities"
  },
  "jsonapi": {
    "version": "1.0"
  }
}

The parse-entities endpoint will only return annotations for channels your API token has been subscribed to.  For information on subscribing and unsubscribing to channels please see the REST API section on Channels.

The annotations array includes a contexts array which is a list of tag-entity pair objects.  Tag-entity pairs represent an entity and its corresponding tag or annotation. The id of the context object is also known as the tag-entity pair id. This id value can be used to update or delete the annotation.

Creating Entity Objects Manually

In some cases, you will not need to parse any text as you will be looking up a single indicator or a list of indicators.  In these cases, you can bypass the parse-entities endpoint and generate the parsed entity objects manually.  

As an example, if you wanted to lookup the IP address 8.8.4.4 you could create a simplified parsed entity object like this:

{
  "type": "IPv4",
  "value": "8.8.4.4"
}          

The type attribute is case sensitive

In addition to IPv4, the other supported types are:

• IPv4

• IPv6

• domain

• email

• IPv4CIDR

• url

• MD5

• SHA1

• SHA256

Searching an Integration

Once you have the parsed entity objects, you can send those objects to specific integrations for enrichment using the integration-lookups endpoint:

POST /api/integrations/<INTEGRATION_ID>/lookup
{
    "data":{
         "type": "integration-lookups",
         "attributes": {
             "entities":[
                // one or more entity objects
            ]  
        }   
    }
}

For example, if we have the following entity object returned from the parsed-entities endpoint:

{
    "display-value": "8.8.8.8",
    "value": "8.8.8.8",
    "type": "IPv4",
    "start-index": 0
}

You would send the following request:

{
    "data":{
         "type": "integration-lookups",
         "attributes": {
             "entities":[
                {
                    "display-value": "8.8.8.8",
                    "value": "8.8.8.8",
                    "type": "IPv4",
                    "start-index": 39
                }
            ]  
        }   
    }
}

The INTEGRATION_ID is assigned when an integration is first installed.  The simplest way to find the id for the integration you're interested in is to login via the Polarity Web Interface and check for the ID in the URL of your browser.

Alternatively, you can return all installed integrations and their IDs by querying the /api/integrations endpoint:

curl -v -X GET \
'https://<polarity.server.url>/api/integrations' \
--header 'Authorization: Bearer <AUTH_TOKEN>' \
--header 'Content-Type: application/vnd.api+json'

import requests
import json

def get_integrations(token, host):
    url = f'{host}/api/integrations'

    payload = {}
    headers = {
        'Content-Type': 'application/vnd.api+json',
        'Authorization': f'Bearer {token}'
    }

    response = requests.request('GET', url, headers=headers, data=payload)
    response.raise_for_status()

    return response.json()
  
integrations = get_integrations(token, 'https://polarity.server.url')  

See Retrieving Integration Information for more details on fetching integration related data.

Examples

As an example, to search the VirusTotal integration for the parsed IP address 8.8.8.8 you could send a payload like this:

curl -v -X POST \
'https://<polarity.server.url>/api/integrations/virustotal/lookup' \
--header 'Authorization: Bearer <AUTH_TOKEN>' \
--header 'Content-Type: application/vnd.api+json'
--data-binary @- <<EOF
{
    "data":{
         "type": "integration-lookups",
         "attributes": {
             "entities":[
                {
                    "display-value": "8.8.8.8",
                    "value": "8.8.8.8",
                    "type": "IPv4",
                    "start-index": 39
                }
            ]  
        }   
    }
}
EOF 

import requests
import json

def search_integration(token, host, integration_id, entity_objects):
    url = f'{host}/api/integrations/{integration_id}/lookup'

    payload = json.dumps({
        'data': {
            'type': 'integration-lookups',
            'attributes': {
                'entities': entity_objects
            }
        }
    })

    headers = {
        'Content-Type': 'application/vnd.api+json',
        'Authorization': f'Bearer {token}'
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    response.raise_for_status()

    return response.json()
  
search_result = search_integration(token, HOST, 'virustotal_3_7_4_node_18_63e5110da4_1697729362', [
    {
        'value': '8.8.8.8',
        'type': 'IPv4'
    }
])

The resulting data will include a data.attributes.results array which contains a result object for each searched entity. These result objects will include the following top-level keys:

{
  "entity": {} // expanded entity value for the results
  "data": {
    "summary": [] // array of summary tags as strings
    "details": {} // JSON object containing details data from integration
  }
}

If the integration did not have a result for the specified entity than the data object will be null:

// lookup result object with no result for the specified entity
{
  "entity": {},
  "data": null
}

Entities included in the payload that are not supported by an integration will be ignored

Fetching Additional Details (onDetails)

Certain integrations fetch additional data in what is known as the onDetails hook. You can query the onDetails hook using the integration-messages endpoint:

POST /api/integrations/<INTEGRATION_ID/message
{
  "data": {
    "type": "integration-message",
    "attributes": {
      "payload": {
        // the lookup result object should be inserted here
      }
    }  
  },
  "meta": {
    "loadDetailsBlock": true
  }
}

Currently, the fastest way to tell if an integration implements an onDetails hook is to check the bottom of the integration’s integration.js file to see if an onDetails method is exported, as in the example below:

module.exports = {
  doLookup,
  startup,
  onDetails, //<-- includes onDetails
  onMessage,
  validateOptions
};

If this method is exported, then you can use the /api/integrations/<INTEGRATION_ID/message endpoint to query the onDetails data.

The endpoint will return a Lookup Result Object containing any additional data loaded by the onDetails hook.

As of 4/03/2024 the following integrations include an onDetails lookup:

• Sophos

• ServiceNow

• ServiceNow SIR

• RipeStat

• Salesforce

• Slashnext

• ThreatConnect

• Rapid7 Nexpose

• Crowdstrike Intel

• ThreatStream

• ReversingLabs

• Analyst1

• Gigamon

• MISP

• Redmine

• FIR Search

• Splunk SOAR (Phantom)

• Flashpoint

• ThreatQuotient

• HYAS Insight

Per Integration Custom Data (onMessage)

Certain integrations also can return additional data via the onMessage hook.  This is custom per integration and requires an understanding of the integration configuration to implement.  The onMessage data can be queried via the integration-messages endpoint.

As an example, the following query can be sent to retrieve “resolutions” information from the PassiveTotal integration.

POST /v2/integration-messages/passivetotal
{
  "data":{
    "type":"integration-messages",
      "attributes": {
        "payload": {
      "entity": {
            "value": "8.8.8.8"
          },
          "searchType": "pdns"
    }
    }
  }
}