HostProfile

DOI: https://doi.org/10.5281/zenodo.23313580 (all versions; see CITATION.cff)

Turns raw traffic into one row of 17 features per local host every 5 minutes, so that a model can spot a host whose behavior changes.

A host profile is the behavior of one local host (Site::local_nets) during one window:

• Input: everything the host originates, whatever the destination: its connections, DNS queries and TLS server names (SNI).
• Windows: tumbling, 5 min by default (HostProfile::window_len).
• Per host and window, 17 statistics:
  • volume: connections, bytes out and in;
  • diversity: distinct destinations and ports (HyperLogLog estimates);
  • novelty: share of destinations, domains and SNIs never seen before for this host (Bloom filter memory, forgotten after a day of inactivity);
  • structure: share of internal destinations, share of failed connections;
  • rhythm: regularity of the gaps between connections, overall and towards the most regular destination (near 0 = beaconing);
  • DNS: queries, distinct domains, NXDOMAIN share;
  • idle time since the host's previous active window.
• Output: one row per active host and window in hostprofile_features.log (Zeek TSV or JSON), ready for an anomaly detection model, identical on a standalone Zeek and on a cluster.

Columns

ts start of the window host the profiled host idle_before seconds since the end of the host's previous active window; -1 for its first window n_conns connections whose last packet falls in the window bytes_out payload bytes sent / received over those connections bytes_in n_dst distinct destination addresses / ports (estimates) n_dst_ports frac_new_dst share of connections to a destination never seen before for this host frac_internal_dst share of connections to a local destination frac_failed share of failed attempts (S0, REJ, ...) start_cv regularity of connection starts: std / mean of the gaps (0 = clockwork); -1 under 3 connections beacon_cv lowest start_cv among the destinations contacted at least 4 times; -1 if none n_dns DNS queries sent n_domains distinct query names (estimate) frac_new_domain share of queries for a name never queried before frac_nxdomain share of queries answered NXDOMAIN n_sni TLS client hellos carrying a server name frac_new_sni share of server names never seen before

When a family has no activity in the window, its counts and shares are 0 and its regularity columns -1. Example row (CICIDS-2017, a workstation):

ts 1499428800.0 host 192.168.10.14 idle_before -1.0 n_conns 204 bytes_out 787363 bytes_in 941544 n_dst 43 n_dst_ports 18 frac_new_dst 0.22 frac_internal_dst 0.71 frac_failed 0.02 start_cv 3.56 beacon_cv 0.69 n_dns 366 n_domains 74 frac_new_domain 0.21 frac_nxdomain 0.01 n_sni 57 frac_new_sni 0.54

Configuration

By default Zeek treats every private or reserved range as local, so hosts of those ranges are profiled too. Declare your own networks explicitly:

redef Site::private_address_space_is_local = F; redef Site::local_nets = { 10.1.0.0/16 }; # your networks

Cluster

Works on any Zeek cluster, Broker or ZeroMQ, with the same output as a standalone Zeek. Workers send each observation (connection, DNS message, SNI) to the proxy owning its host (chosen by hashing the host), which keeps the windows and writes the rows. Without a proxy, the manager does it and warns at startup: add proxies to spread the load.

Resources

Measured on Zeek 9.0 (one process, CICIDS-2017 Friday capture, 8.3 GB, 28 profiled hosts), two runs each:

CPU +1 to +9 % user time over plain Zeek (runs vary by ~10 %) memory +1 to 2 MB peak on that capture

Memory grows with the number of profiled hosts, mostly the novelty memory (one Bloom filter of about 18 KB per kind: destinations, domains, SNIs; two after a rotation). Measured per host:

long-lived state 54 KB, up to 107 KB once filters have rotated open window (active) 7 KB

So about 110 MB per 1,000 hosts in the worst case. Hosts are forgotten after HostProfile::host_expire (1 day) of inactivity. The filter size scales with HostProfile::novelty_capacity (10,000 keys) and log(1 / novelty_fp): e.g. novelty_capacity = 2000 divides it by 5, at the cost of forgetting sooner. In a cluster this memory lives on the proxies, spread across them.

Evaluation

Reproducible with one command (eval/, pinned images, fixed seeds, public data; two runs give identical results). An unsupervised detector (IsolationForest) learns CICIDS-2017 Monday (benign) and scores the other days. Baseline: NetFlow-like counters only (connections, bytes). Values: baseline -> HostProfile, mean of 10 seeds.

Ranking (AUC): the features separate attacks from benign windows better than counters, also on a day nothing was tuned on:

Tuesday FTP-Patator 0.67 -> 0.93 SSH-Patator 0.74 -> 0.95 Thu/Fri Web Attack 0.61 -> 0.94 Infiltration 0.83 -> 0.88 PortScan 0.98 -> 0.99 DDoS 0.92 -> 0.92 Bot 0.55 -> 0.54

Alerts, at equal false-alarm rates (share of attack windows detected):

               at 0.1 %     at 0.5 %     at 1 %

Infiltration 11 -> 3 % 19 -> 44 % 47 -> 52 % PortScan 25 -> 18 % 42 -> 57 % 70 -> 68 % DDoS 0 -> 3 % 0 -> 62 % 82 -> 80 % FTP-Patator 0 -> 0 % 7 -> 0 % 7 -> 4 % SSH-Patator 0 -> 0 % 0 -> 0 % 0 -> 1 %

Holdout, Wednesday: attacks never seen during development, detector fixed beforehand, run once (eval/README):

                 AUC           at 0.1 %     at 0.5 %     at 1 %

DoS Hulk 0.99 -> 1.00 0 -> 8 % 0 -> 97 % 65 -> 100 % DoS Slowhttptest 0.84 -> 0.99 0 -> 6 % 0 -> 70 % 0 -> 80 % DoS slowloris 0.90 -> 0.98 0 -> 3 % 0 -> 18 % 11 -> 28 % DoS GoldenEye 0.97 -> 0.99 0 -> 0 % 0 -> 30 % 0 -> 70 % Heartbleed 0.95 -> 0.94 0 -> 0 % 0 -> 0 % 0 -> 0 %

beacon_cv alone separates real botnets (CTU-13) from CICIDS benign hosts: AUC 0.83 Neris, 0.89 Murlo, 0.72 Donbot; Rbot rarely contacts one destination 4 times in a window (0.14).

What it shows: the features carry the signal (better ranking, also on unseen data). On the holdout, the same generic detector turns it into alerts: at 0.5 % false alarms it catches 18 to 97 % of the DoS windows, counters 0 %. Heartbleed is missed. On the development days it does not improve alerts at 0.1 %. Slow or bursty beacons (CICIDS Ares bot) are missed. Attack windows are few (2 to 16, except Bot): one window moves a percentage.

Layout

scripts/main.zeek windowing engine, cluster dispatch, distinct counters (HLL), novelty memory (rotating Bloom filters) scripts/features/conn.zeek volume, diversity, novelty, failures, rhythm scripts/features/dns.zeek queries, domains, novelty, NXDOMAIN rate scripts/features/tls.zeek SNI count and novelty

A new feature family adds columns with redef record Features += {...} and its data with redef record Observation += {...}. Its event handlers build an Observation and pass it to HostProfile::send; a hook HostProfile::observe handler accumulates it into WindowState/HostState, and a hook HostProfile::finalize handler fills the columns.

Testing

Tests use btest (testing/tests/) and run in a Zeek container, so no local Zeek is needed:

make test # run the tests make test-update # accept new outputs as baselines (review the diff first) make traces # regenerate the synthetic traces

The traces come from testing/Scripts/gen-traces.py, which documents each scenario and its expected values. The cluster tests replay every trace through a local cluster and compare with standalone output (testing/Scripts/cluster-vs-standalone).