Custom Entity Types

As of Polarity v5.4.0, administrators have the ability to add Custom Entity Types for many Integrations via the UI.  The information below is the manual process to add a Custom Entity Type (which applies to any Polarity Server v5).  This page was distilled from two or more documents.

Add Custom Entity Types to an Integration

What is a Custom Entity Type?

You can add Custom Entity Types to extend the functionality of a supported integration. For certain integrations (eg. LDAP), you may want to add a Custom Entity Type so the integration will search using additional data types.

Custom Entity Type objects have two properties:

• “key” is used by the integration to identify the Custom Entity Type.

  • The “key” value should only include alphabetic characters (regardless of case) and underscores.

  • You should not include spaces in the “key” value.

• “regex” is a regular expression (regex).

  • We recommend using an efficient regex and boundaries in your regex to ensure your Custom Entity Type only matches items you want to query.

Adding a Custom Entity Type

To add a Custom Entity Type, you will need to modify two files in the target integration’s directory structure:

• the config.json file (located in the target integration's config directory)

  • /app/integrations/<INTEGRATION_NAME>/config/config.json

• the package.json file (located in the integration’s home directory)

  • /app/integrations/<INTEGRATION_NAME>/package.json

Modify the config.json file

In the config.json file, you will see an “entityTypes” property near the top of the file. If the “customTypes” option already exists, then you will add your additional type to the “customTypes” array.

To add a Custom Entity Type, you will add or modify the “customTypes” option to the config.json file. Open it with a text editor on your server:

vi /app/integrations/<INTEGRATION_NAME>/config/config.json

For example, the Polarity Splunk integration’s config.json file looks like this (parts truncated for clarity):

{
  "polarityIntegrationUuid": "b59a2740-cce1-11ed-aeee-075d3490155d",
  "name": "Splunk",
  "acronym": "SPLNK",
  "description": "Splunk allows you to aggregate, analyze …",
  "entityTypes": [
    "IPv4",
    "IPv6",
    "hash",
    "email",
    "domain",
    "cve"
  ],
  "defaultColor": "light-gray",

…
}

Here is what a single Custom Entity Type called “myCustomType” might look like when added to the file.  This custom type matches any letter or numeral combination, 4 to 9 characters in length, between natural word boundaries:

{
…

    “domain”,
    “cve”
  ],
  “customTypes”: [
    {
      “key”: “myCustomType”,
      “regex”: “\\b[a-z0-9]{4,9}\\b”
    }
  ],

…
}

Again, “myCustomType” would match entities that are within two word boundaries (\b, matching spaces or other built-in word boundaries) that contain any letter or numeral combination ([a-z0-9]) between 4 and 9 characters long ( {4,9} ).

NOTES:

- Polarity recognition is case insensitive, so providing separate cases is unnecessary.

- To accommodate the JSON format of the config.json file, you must “double-escape” regex built-ins (such as \b in the example above, making it \\b).

For an integration with multiple Custom Entity Types, it might look like this:

{
…

  ],
  “customTypes”: [
    {
      “key”: “myCustomType”,
      “regex”: “\\b[a-z0-9]{4,9}\\b”
    },
    {
      “key”: “anotherCustomType”,
      “regex”: “\\b\\d{,7}\\b”
    }
  ],

…
}

The biggest difference between single and multiple entries is the comma character ( ,) following the closing curly brace ( }) that bounds each entry (except for the final entry).

As a third and final example, you could use the following Custom Entity Type to match a hostname that consists of 8 letters followed by 2 numbers:

{
…

  ],
  “customTypes”: [
    {
      “key”: “hostname”,
      “regex”: “\\b[a-z]{8}\\d{2}\\b”
    }
  ],

…
}

Once you've made your changes, save the file.

Increment the Integration Version

For the Polarity Server to recognize the change in the integration’s baseline configuration, you must increment its version.  The version of the integration is designated within the integration's package.json file.  Open it with a text editor on your server:

vi /app/integrations/<INTEGRATION_NAME>/package.json

For example, the Polarity Splunk integration’s package.json file looks something like this:

{
  "name": "Splunk",
  "version": "3.6.4",
  "main": "./integration.js",
  "description": "Splunk Query Engine",
  "private": true,
  "dependencies": {
    "async": "^3.2.5",
    "lodash": "^4.17.21",
    "node-cache": "^5.1.2",
    "postman-request": "^2.88.1-postman.33"
  }
}

Once open, find the “version” property, and append a plus character ( +) and a numeric counter to signify that you have made a modification to the integration.  Leave the base version alone so we know which integration version was modified.

From the example above with a base version of "3.6.4", we can simply add a “ +1” to the end of it for the first change, making the version “3.6.4+1”.

We would increment the count if we made another change to the integration’s config.json (such as adding another Custom Entity Type), which would make the version “3.6.4+2”, and so on.

The changed package.json file (based on the Polarity Splunk integration’s file above) would be this:

{
  "name": "Splunk",
  "version": "3.6.4+1",
  "main": "./integration.js",
  "description": "Splunk Query Engine",
  "private": true,
  "dependencies": {
    "async": "^3.2.5",
    "lodash": "^4.17.21",
    "node-cache": "^5.1.2",
    "postman-request": "^2.88.1-postman.33"
  }
}

Save and close the package.json file.

Restart the Integration

Finally, you must restart the integration from the Polarity User Interface for the new Custom Entity Type to be identified and assimilated into the Server.

1. Log into the Polarity Server as an “admin” user.  

2. Navigate to the “Advanced Settings” view in the Polarity User Interface.

3. Select “Integrations” in the Nav Menu on the left.

4. Find the integration that has been modified.

5. Select the “Settings” link on the right.

6. In the integration’s Settings view, select “Restart integration” from the Nav Menu on the left.

7. Click the green “Restart integration” button.

After the integration has restarted, you should see the version number reflect the counter added in the package.json file.

NOTE:

If the integration fails to restart and ends up in a “Stopped” state, a configuration error has occurred. This is usually a syntax error in the config.json file.

Check it thoroughly for errors, and feel free to contact us at Support@polarity.io for assistance.

Best Practices

Using Custom Entity Types inherently increases the number of queries to the resource the integration interfaces with. To mitigate the effects of this, we recommend setting the integration to be “On-Demand Only” for all users.  This helps by limiting the search context that will trigger a query to only deliberate searches.

You can configure this by checking the "On-demand only" option and setting the permission on that option to "Users can view only".

Creating a New Entity Format

The Basics

What is an Entity Type?

An entity type in Polarity is a data type or format that forms a template for Polarity software to recognize specific strings from within a bulk or batch submission of text. The most common format for this “recognition template” is a regular expression, or “RegEx”.

How to Figure Out Your New Entity Type’s Format

The best way to establish a new entity type is by taking a sampling of the data you wish to recognize and looking for a pattern that maximizes recognition (ie. true positives) while minimizing misrecognition (ie. false positives) and complete misses (ie. false negatives). Using this pattern, you can create a RegEx to match your samples, accounting for as many potential variances as possible but avoiding looking for “all the things”.

Creating Your RegEx

Gathering Samples

To maximize your time and effort, your data samples should cover a variety of cases. For example, if the data type is a hostname for end user computers, look for as many different hostname variations as possible and add them to a running list for testing.

Looking for the Pattern

The pattern can be recognized in many ways, and the best method varies between individuals.

One simple method is to use the running list of samples: place each sample on its own line in a text file (ensure you are using a monospace font to display your samples) and look for differences in length and placement of character type (ie. letters vs. numerals vs. special characters). This can provide a simple starting pattern.

Use a Tool!

Using a RegEx helper utility, such as the online resource RegEx101.com, is the best way to refine your RegEx for your application. This simple, easy to use website allows you to place sample data in one field and fine-tune your RegEx in another, all the while receiving immediate feedback on what your RegEx pattern means in plain language and which samples matched your pattern.

Refining Your Work

While refining your RegEx, we recommend adding some “false samples” that are close to your actual data samples but don’t represent desired matches. This can help avoid false positives, yielding a tighter, more efficient pattern.

Formatting Your RegEx for Polarity

Polarity uses the JSON format in integration configuration files. The basic JSON block that your Custom Entity Type will fit into will be in the integration’s ./config/config.json file, and looks like this example:

  “customTypes”: [
    {
      “key”: “customEntityTypeName”,
      “regex”: “customRegEx”
    }
  ],

“customEntityTypeName” will be a unique, short name describing the data type your pattern recognizes. “customRegEx” will be your custom RegEx.

To accommodate the JSON formatting of the integration configuration file, any RegEx built-in codes used which are normally “escaped” (ie. prepended with a backslash character, “ \”) must be “double-escaped” by adding another backslash character to escape the original backslash character.  This simply means that the RegEx code for “numbers” or “digits” ( \d) would become \\d.

External Resources

https://www.regex101.com is a good resource to assist in creating and refining your RegEx and has built-in tipping and resources to help learn how to create RegEx patterns efficiently and effectively.

Note on External Resources

Please be aware that neither Polarity nor ThreatConnect owns or is involved with the RegEx101 site, and it is only listed here as an example of a potentially useful resource when creating and testing regular expressions.

Use of this resource is at your own risk!  We do not recommend using live data to test your regular expressions.

Please reach out to your Customer Success representative, Support@Polarity.IO, or CustomerSuccess@Polarity.IO for support, assistance, or questions.