ZeekJS-Redis

This zkg package implements a method of parsing Zeek logs to Redis. ZeekJS is utilised to achieve this. Using a Unix socket for data transmission, this package builds upon the log-bypass example, to enhance storage options and faster data analysis.

The intent with the development of this package was to "kick the tyres" and gain familiarity with ZeekJS. The overall experience was positive; it was possible to create a working version of features that would have taken me much longer to develop in Zeek's standard C++ plugin architecture. Time saved enabled implemention of more involved config options.

Example

1. Run the test PCAP through Zeek:
After installing the package run the test pcap with zeek.

/opt/zeek/var/lib/zkg/clones/package/zeekjs-redis.git# zeek -C LogAscii::use_json=T -r testing/Traces/zeekjs-redis-test.pcap ./scripts/index.js

-C avoids checksum errors, LogAscii::use_json=T enables JSON formatted logs. Note, if installing from a local git pull, the pathway is likely:

PREFIX_PATH=$(zeek-config --prefix)
FULL_PATH="${PREFIX_PATH}/var/lib/zkg/clones/package/zeekjs-redis"

2. Access Redis using the CLI:
Connect to Redis through the Unix socket to query the results.

/opt/zeek/var/lib/zkg/clones/package/zeekjs-redis.git# redis-cli -s /var/run/redis/redis.sock

3. View the keys stored in Redis:
Check what keys are in Redis after running Zeek.

redis /var/run/redis/redis.sock> KEYS *
1) "zeek_packet_filter_logs"
2) "zeek_conn_logs"

4. Retrieve log data from Redis:
Fetch the stored conn logs.

redis /var/run/redis/redis.sock> LRANGE zeek_conn_logs 0 -1

Example Output:

1) "{\"ts\":1616775350.763199,\"uid\":\"XXX\",\"id\":{\"orig_h\":\"192.168.220.35\",\"orig_p\":53537,\"resp_h\":\"192.168.220.1\",\"resp_p\":31981},\"proto\":\"tcp\",\"conn_state\":\"S0\",\"local_orig\":true,\"local_resp\":true,\"missed_bytes\":0,\"history\":\"S\",\"orig_pkts\":1,\"orig_ip_bytes\":44,\"resp_pkts\":0,\"resp_ip_bytes\":0}"
...
24) "{\"ts\":...

Build

Install with zkg:

zkg install https://github.com/mbispham/zeekjs-redis.git

The following options can be used to create a suitable environmental for the package

--install-npm-dependencies Install npm dependencies from package.json (N/y)
--install-redis-cli        Install redis-cli if not present (N/y)
--redis-conf-path=PATH     The path to Redis config (Default = /etc/redis/redis.conf)
--redis-socket-path=PATH   The path to Redis socket (Default = /var/run/redis/redis.sock)
--start-redis-server       Start redis server (N/y)

Install with --force if you want to install npm dependencies separately, use default settings, or modify scipts/.env directly:

zkg install https://github.com/mbispham/zeekjs-redis.git --force

Filtered Log Usage

If the intention is to export specific fields from Zeek logs to Redis, an example modification to index.js is outlined below:

  ...
  logID = logID.replace(/([a-z0-9])([A-Z])/g, '$1_$2').toLowerCase();

  const logFile = `${logID}.log`;
  const redisKey = `zeek_${logID}_logs_orig_ip_bytes`;
  const logRec = zeek.select_fields(logData, zeek.ATTR_LOG);
  if (!('orig_ip_bytes' in logRec)) {
    return;
  }

  const origBytes = logRec.orig_ip_bytes;
  const serializedData = stringify(origBytes);
  ...

Dependencies

• Zeek > 6.0.2 - Experimental ZeekJS supported as a builtin package
• Redis-cli
• NPM - Ensure Node.js is configured with shared OpenSSL
  • dotenv
  • redis
  • safe-stable-stringify
  • validator
  • winston

Acknowledgements

• Christian Kreibich: Zeek project technical lead
• Arne Welzel: Main author of ZeekJS
• Simeon Miteff: The structure of this project was inspired by Simeon's work on integrating telegram with Zeek zeekjs-notice-telegram
• WRCCDC: zeekjs-redis-test.pcap is carved from a publicly available trace released under a creative commons license

License

zeekjs-redis is free and open-source software licensed under the 3-clause BSD license.

Feedback and Contributions

Feedback and/or contributions are welcome.