Enabling Polarity Source Analytics

Enable or adjust source analytics logging levels

Enabling Source Analytics Logging

Polarity Source Analytics logging is enabled by default at the verbose level on Polarity Server v5.  

To adjust Polarity Source Analytics logging on the Polarity server edit the /app/.env file.

In the .env file you will need to edit and/or add the following line:

POLARITY_LOG_INTEGRATION_LOOKUPS=verbose

There are four logging levels for lookups: off, minimal, info, verbose.  Each logging level is additive, data wise. 

To leverage the Polarity Source Analytics integration, set this logging level to verbose.

When enabled, a JSON log entry will be generated for each lookup performed by the server.  There will be one log entry per integration per entity.  The generated logs are newline delimited JSON.

After modifying the .env file you will need to restart the Polarity Server containers for the change to take effect.

cd /app && ./down.sh && ./up.sh

After the server restarts, you will find telemetry log entries appearing in the polarity_platform Docker container's log file.  You can verify that telemetry logs are being captured by running the following search:

docker logs polarity_platform |& grep "\"msgType\":\"integration-lookup\""

You should see results that looks like this:

{"email":"support@polarity.io","entityTypes":["IP","IPv4"],"entityValue":"8.8.8.8","hasResult":true,"integrationId":"maxmind","isFromCache":false,"isOnDemand":true,"msgType":"integration-lookup","primaryEntityType":"IPv4","process":null,"timestamp":"2024-02-19T15:42:43.288989Z","userId":1,"username":"admin","windowTitle":null}

If you do not see any logs, ensure that you have restarted the Polarity Platform docker containers and that you have run a search since enabling telemetry logging.

Logging Levels

off

The default logging level is off which will not log any integration lookups

minimal

The minimal logging level will log the following fields:

• timestamp
  • An ISO8601 compliant timestamp which represents the time at which the lookup occurred.
• msgType
  • A static value of integration-lookup designed to make it easy to identify source analytic logging messages.
• integrationId
  • The id of the integration that performed the lookup
• hasResult
  • A boolean true or false indicating whether or not the lookup returned a result
• isFromCache
  • A boolean true or false indicating whether or not the result was returned from the integration cache
• isOnDemand
  • A boolean true or false indicating whether or not the lookup was an on-demand lookup
• entityTypes
  • An array (list) of all entity types that the looked up entity matches
• primary EntityType
  • The primary entity type (e.g., IPv4, MD5, domain etc.)

info

Contains all the information logged at the minimal level but also includes the userId , username, and email properties.

• userId
  • The immutable integer id of the user that performed the lookup
• username
  • The username of the user making the search request.  Note that a user's username can be changed over time.
• email
  • The email of the user making the search request. Note that a user's email address can be changed over time.  

verbose

Contains all information logged at the minimal and info level but also includes the following entity information.

• entityValue
  • The value of the entity that was looked up.
• windowTitle
  • If the lookup was not an on-demand lookup, windowTitle will be the title of the window where the entity was extracted from via Polarity's OCR capabilities.
• process
  • If the lookup was not an on-demand lookup (i.e., was a Stream of Highlight lookup), the process will be the name of the application process where the entity was extracted from via Polarity's OCR capabilities.

Example Source Analytics Log Entry

The following is an example of a log entry generated when POLARITY_LOG_INTEGRATION_LOOKUPS is set to verbose:

{
   "hasResult":true,
   "isFromCache":false,
   "integrationId":"cyberchef",
   "msgType":"integration-lookup",
   "isOnDemand":true,
   "primaryEntityType":"custom",
   "entityTypes":[
      "custom.encodedString",
      "IPv4",
      "IP"
   ],
   "entityValue":"8.8.8.8",
   "userId":1,
   "username": "admin",
   "email": "support@polarity.io",
   "level":"info",
   "message":"Integration Lookup",
   "timestamp":"2021-02-09T21:51:25.358Z"
}