Get started

Administration

Administration / Metrics & health

Overview
Metric event structure
Key/value format
JSON format
Field reference

Overview

DnsStream writes periodic metrics covering which include message collection, queue health, and forwarding metrics to the Windows Event Log on Windows and the system log on Linux.

By default metrics are written every 10 minutes (600 seconds).

metrics-interval 600

Metric event format can be controlled using the metrics-format in the dnsstream.conf. DnsStream supports a key/value format using metrics-format kv (the default) and a JSON format using metrics-format json.

metrics-format kv

DnsStream always writes metrics to the Windows Event Log under the Telemity DnsStream provider, and the Linux system log using the dnsstream.bin process identity. In both cases, regardless of format used, the raw metrics payload is prefixed with Metrics: .

Metrics: listener[id=0 type=etw numberOfBuffers=16 freeBuffers=16 eventsLost=0 buffersWritten=310875 logBuffersLost=0 realTimeBuffersLost=0] queue[eventsProcessed=42 eventsQueued=0 eventsDropped=0] forwarder[id=0 type=ipv4 host=host1.internal port=514 state=connected bytesSent=5629 eventsSent=42 dnstapErrors=0 connects=1]

Metric event structure

The following metrics sections are present.

Section	Covers
`listener`	One block per configured listener, i.e. one for ETW on Windows or one for UNIX domain socket on Linux.
`receiver`	One block per inbound connection.
`queue`	The in-memory ring buffer sitting between capture and forwarding.
`forwarder`	One block per configured forwarding target, reporting connection state and forwarding throughput.

These sections map onto stages of the DnsStream data path - capture, buffer, forward - making it straightforward to identify which stage a problem is occurring in.

Metrics appear in the order: one listener block per configured listener, one or more receiver blocks for each inbound connection, queue, then one forwarder block per configured forwarding target.

Key/value format

Format

When metrics-formant kv is configured, DnsStream formats metrics using a key=value format with keys grouped into sections. Sections are each enclosed in square brackets with the space-separated key=value fields inside.

# Windows
metrics: listener[id=0 type=etw ...] queue[...] forwarder[id=0 ...] forwarder[id=1 ...]

# Linux
metrics: listener[id=0 type=unix ...] receiver[id=0 ...] receiver[id=1 ...] queue[...] forwarder[id=0 ...] forwarder[id=1 ...]

Single-target example on Windows

A deployment with one forwarding target:

Metrics: listener[id=0 type=etw numberOfBuffers=16 freeBuffers=16 eventsLost=0 buffersWritten=310875 logBuffersLost=0 realTimeBuffersLost=0] queue[eventsProcessed=42 eventsQueued=0 eventsDropped=0] forwarder[id=0 type=ipv4 host=host1.internal port=514 state=connected bytesSent=5629 eventsSent=42 dnstapErrors=0 connects=1]

Multi-target example on Windows

A deployment with two forwarding targets - one connected, one attempting to connect:

Metrics: listener[id=0 type=etw numberOfBuffers=16 freeBuffers=16 eventsLost=0 buffersWritten=310875 logBuffersLost=0 realTimeBuffersLost=0] queue[eventsProcessed=42 eventsQueued=0 eventsDropped=0] forwarder[id=0 type=ipv4 host=host1.internal port=514 state=connected bytesSent=5629 eventsSent=42 dnstapErrors=0 connects=1] forwarder[id=1 type=ipv4 host=host2.internal port=6514 state=not-connected bytesSent=0 eventsSent=0 connects=0]

Single-target example on Linux

A deployment with one forwarding target:

Metrics: listener[id=0 type=unix path=/run/telemity/dnsstream/dnstap.sock accepts=1 activeReceivers=1 peakReceivers=1] receiver[id=0 listener=0 type=unix uid=1000 gid=1000 pid=4324 bytesReceived=5968 eventsReceived=32] queue[eventsProcessed=32 eventsQueued=0 eventsDropped=0] forwarder[id=0 type=ipv4 host=localhost port=514 state=not-connected bytesSent=9183 eventsSent=32 dnstapErrors=0 connects=0]

JSON format

When metrics-format json is configured, DnsStream formats metrics as a single line JSON object. The following top-level keys are included which corresond to the section:

listeners - Array of listener metrics
receivers - Array of receiver metrics
queue
forwarders - Array of forwarder metrics

Single-target example on Windows

{
  "listeners": [
    {
      "id":                   0,
      "type":                 "etw",
      "numberOfBuffers":      16,
      "freeBuffers":          16,
      "eventsLost":           0,
      "buffersWritten":       310875,
      "logBuffersLost":       0,
      "realTimeBuffersLost":  0
    }
  ],
  "queue": {
    "eventsProcessed": 42,
    "eventsQueued":    0,
    "eventsDropped":   0
  },
  "forwarders": [
    {
      "id":           0,
      "host":         "host1.internal",
      "port":         514,
      "state":        "connected",
      "bytesSent":    5629,
      "eventsSent":   42,
      "dnstapErrors": 0,
      "connects":     1
    }
  ]
}

Multi-target example on Windows

{
  "listeners": [
    {
      "id":                   0,
      "type":                 "etw",
      "numberOfBuffers":      16,
      "freeBuffers":          16,
      "eventsLost":           0,
      "buffersWritten":       310875,
      "logBuffersLost":       0,
      "realTimeBuffersLost":  0
    }
  ],
  "queue": {
    "eventsProcessed": 42,
    "eventsQueued":    0,
    "eventsDropped":   0
  },
  "forwarders": [
    {
      "id":           0,
      "host":         "host1.internal",
      "port":         514,
      "state":        "connected",
      "bytesSent":    5629,
      "eventsSent":   42,
      "dnstapErrors": 0,
      "connects":     1
    },
    {
      "id":           1,
      "host":         "host2.internal",
      "port":         6514,
      "state":        "not-connected",
      "bytesSent":    0,
      "eventsSent":   0,
      "dnstapErrors": 0,
      "connects":     0
    }
  ]
}

Single-target example on Linux

{
  "listeners": [
    {
      "id":              0,
      "type":            "unix",
      "path":            "/run/telemity/dnsstream/dnstap.sock",
      "accepts":         1,
      "activeReceivers": 1,
      "peakReceivers":   1
    }
  ]
  "receivers": [
    {
      "id":             0,
      "listener":       0,
      "type":           "unix",
      "uid":            1000,
      "gid":            1000,
      "pid":            4234,
      "bytesReceived":  5968,
      "eventsReceived": 32
    }
  ],
  "queue": {
    "eventsProcessed": 32,
    "eventsQueued":    0,
    "eventsDropped":   0
  },
  "forwarders": [
    {
      "id":           0,
      "type":         "ipv4",
      "host":         "localhost",
      "port":         514,
      "state":        "not-connected",
      "bytesSent":    9183,
      "eventsSent":   32,
      "dnstapErrors": 0,
      "connects":     0
    }
  ]
}

Field reference

`listener`

The listener section has several different formats.

For listener etw ..., the listener section reports the state of the Windows ETW session through which DnsStream receives DNS events from the DNS Server service. Note that for this type of listener there are no related receiver sections like there would be for other listener types.

Field	Type	Example	Description
`id`	integer	0	Unique ID for this listener.
`type`	string	`etw`	Listener type, always `etw` for ETW sessions.
`numberOfBuffers`	integer	`16`	Total number of ETW buffers allocated to the DnsStream trace session. Controlled by `trace-maximum-buffers` in `dnsstream.conf`.
`freeBuffers`	integer	`16`	Number of ETW buffers not currently in use at the time of sampling. A value consistently equal to `numberOfBuffers` indicates the ETW session is draining buffers faster than they fill. A value of `0` indicates ETW buffer pressure.
`eventsLost`	integer	`0`	Number of DNS events dropped by the ETW subsystem during this interval because all buffers were full when the event was generated. This is ETW-side loss, distinct from queue-side loss reported in `queue.eventsDropped`. Any non-zero value means DNS events were lost before DnsStream could receive them.
`buffersWritten`	integer	`310875`	Number of ETW buffers delivered to DnsStream during this interval. A consistently non-zero value confirms the ETW session is active and receiving events.
`logBuffersLost`	integer	`0`	ETW buffers lost to disk-based logging during this interval. Should always be `0` - DnsStream uses real-time ETW delivery, not file-based logging. A non-zero value indicates an unexpected ETW session configuration issue.
`realTimeBuffersLost`	integer	`0`	ETW buffers lost in real-time delivery to DnsStream during this interval. Distinct from `eventsLost` - this counts buffer-level loss rather than event-level loss. Non-zero values indicate the DnsStream consumer thread is not draining buffers fast enough.

For all listener unix ..., the listener section reports the state of the listening socket accepting inbound dnstap producers.

Field	Type	Example	Description
`id`	integer	0	Unique ID for this listener.
`type`	string	`unix`	Listener type (currently only `unix`).
`path`	string	`/run/telemety/dnsstream/dnstap.sock`	UNIX socket.
`accepts`	integer	`1`	Number of inbound connections accepted during the current interval.
`activeReceivers`	integer	`1`	Current number of active receiver connections.
`peakReceivers`	integer	`1`	Highest number of concurrent receiver connections observed.

`receiver`

One block per inbound producer connection.

Field	Type	Example	Description
`id`	integer	`0`	Receiver identifier.
`listener`	integer	0	Unique ID of the associated listener the connected was accepted through.
`type`	string	`unix`	Connection type (currently only `unix`).
`uid`	string	1000	User ID of connecting process.
`gid`	string	1001	Group ID of connecting process.
`pid`	string	4324	Process ID of connecting process.
`bytesReceived`	integer	`5968`	Bytes received during the current interval.
`eventsReceived`	integer	`32`	Events received during the current interval.

`queue`

The queue section reports the state of the in-memory ring buffer that sits between capture and network forwarding.

Field	Type	Example	Description
`type`	string	`message`	Event type the queue manages (currently only `message` to indicate DNS messages).
`eventsProcessed`	integer	`42`	Number of events removed from the queue and handed to the forwarding thread during this interval. Under normal operation this tracks closely with ingestion rate.
`eventsQueued`	integer	`0`	Number of events currently sitting in the ring buffer awaiting forwarding at the time of sampling. Under normal operation this should be zero or near zero. A persistently growing value indicates forwarding cannot keep pace with capture.
`eventsDropped`	integer	`0`	Number of events discarded during this interval because the ring buffer was full when they arrived. This is queue-side loss, distinct from the ETW-side loss reported in `etw.eventsLost`. Any non-zero value means events were captured from ETW but could not be queued for forwarding and were permanently discarded.

`forwarder`

One block per configured forwarding target. In syslog format, each target appears as a separate forwarder[...] block identified by id. In JSON, all targets appear as objects in the forwarders array.

Field	Type	Example	Description
`id`	integer	`0`	Zero-based index identifying this forwarding target. Corresponds to the order targets are listed in `dnsstream.conf`. Stable across metrics intervals for the lifetime of the service process.
`type`	string	`ipv4`	Connection type. Either `ipv4` or `ipv6`.
`host`	string	`"host1.internal"`	Hostname or IP address of the forwarding target as configured in `forwarder`.
`port`	integer	`514`	TCP port of the forwarding target.
`state`	string	`"connected"`	Current connection state. Either `connected` or `not-connected`.
`bytesSent`	integer	`5629`	Number of bytes successfully written to this target’s TCP connection during this interval.
`eventsSent`	integer	`42`	Number of events successfully forwarded to this target during this interval. This is the most operationally useful counter for throughput and health monitoring.
`dnstapErrors`	integer	`0`	Number of unparseable dnstap messages during this interval. This is seperate from DNS message parsing and decoding errors. These errors are forwarded as `error` type messages in the normal forwarding stream.
`connects`	integer	`1`	Number of TCP connections established to this target during this interval. A value greater than `0` indicates at least one connection or reconnection occurred. Persistent non-zero values across intervals indicate connection instability.

This page