Zeek Prometheus Exporter
This is a Zeek plugin which will track performance and health metrics in real-time, and run a web server which can be scraped with Prometheus.
Install the Package
$ zkg install zeek-exporter
Configure Binds and Scraping
The scripts will assign each node a unique port. By default, it will bind to all addresses (0.0.0.0).
A Prometheus scrape target example for file discovery is available in prometheus/target.yml.
The easiest way to get the list of ports is via
$ zeekctl print Exporter::bind_port
zeek-logger Exporter::bind_port = 9101/tcp
zeek-manager Exporter::bind_port = 9102/tcp
zeek-proxy Exporter::bind_port = 9103/tcp
zeek-worker-1 Exporter::bind_port = 9104/tcp
zeek-worker-2 Exporter::bind_port = 9105/tcp
zeek-worker-3 Exporter::bind_port = 9106/tcp
zeek-worker-4 Exporter::bind_port = 9107/tcp
Scrapes can occur as often as you need, and you can track how long scrapes take in the
Our scrapes take about 0.2 seconds, and we scrape every 10 seconds, to detect some micro-bursts.
The cost of frequent scrapes is disk space on the Prometheus system, and increased memory/computation when generating the graphs.
A Grafana dashboard is included in prometheus/dashboard.json.
How It Works
To get the necessary visibility, each node in a Zeek cluster will run this plugin and a Prometheus exporter.
Metrics are tracked individually for each node.
The plugin uses some of the available hooks to inspect function calls, log writes, and even itself and other plugin hooks.
What It Can Measure
The following are instrumented at a high granularity:
- Built In Functions (BIFs),
- Plugins implemented via hooks
What It Can't Measure
While we have total CPU run-time, we don't have fine-grained visibility into:
- The core I/O loop,
- Protocol and file analyzers,
What Impact Will This Have?
Don't know, but you can measure it!
On ESnet systems, the impact ranges from 0.05% to 15%. Workers are impacted most heavily, and the impact is determined by the volume of event and function calls. The function call hook is the most expensive, adding about 2 microseconds to every function call. However, on a very busy system, with > 100k calls/second, this will add up to 200ms each second.
For increased visibility into some functions, simply having the function name isn't enough. For instance, for SumStats,
we'd like to be able to measure the cost of each SumStat (
detect-ssh-bruteforcing). To do this,
we augment the metrics with additional labels for Prometheus.
There's a Zeek option available,
Exporter::addl_functions which allows you to define which events and functions to add labels for.
There are two labels available,
This supports the case of, for instance, for the
unknown_protocol weird, grabbing the
addl value telling you which protocol was unknown.
For more information, see the Zeek script documentation.
Detailed Metrics Information
Wall-Clock and CPU Time
zeek_start_time_secondsThe epoch timestamp of when the process was started. Used to detect periodic crashes.
zeek_total_cpu_time_secondsThe total amount of CPU time spent in this process. This uses the standard library
clock()function call, which returns an approximation. This can give an idea of how much "headroom" there is before more resources are needed.
Note: The logger node runs multiple threads, which will result in an inaccurate count in most cases, as they get scheduled on different CPUs and execute in parallel.
Number of Invocations
zeek_function_calls_totalThe number of times Zeek functions were called, by function and parent function. This can be used to identify the most common execution path in scripts.
zeek_hooks_totalThe number of times Zeek plugin hooks were called. This can be used to identify the most common execution paths in plugins.
zeek_log_writes_totalThe number of log writes per log, writer and filter. This is mainly used to understand the network traffic profile.
Function and Plugin Hook Durations, by Type
zeek_cpu_time_per_function_type_secondsThe amount of time spent in Zeek functions, by type. Types are Built In Functions (BIFs), and the three script-land function types: events, hooks, and functions. Note that script hooks are different from plugin hooks.
zeek_hook_cpu_time_secondsThe amount of time spent in Zeek plugin hooks, by hook name.
Function Durations, by Function and Parent Function
There are two very similar metrics, one which includes execution time in child functions, and one which doesn't.
zeek_cpu_time_per_function_secondsThe amount of time spent in Zeek functions, by function and parent function. Note that this includes the time any child functions take to execute, and as such will count those executions multiple times when summed.
zeek_absolute_cpu_time_per_function_secondsThe "absolute" amount of time spent in Zeek functions. These metrics do not include the time spent in child functions, and thus will give valid data when summed.