Diagnostics and Load Testing

Updated: Oct 30, 2026

Author: Erkin Khidirov

Configuring classic TURN servers often turns into working "blind" - administrators have to install third-party packages, manually verify UDP packet traversal through NAT, and guess how many concurrent video calls a virtual machine can handle.

The Slyber Turn Server binary includes two built-in professional CLI tools that cover all server verification and profiling tasks without installing external dependencies:

slyber_turn doctor - an automated scanner for configuration, network ports, NAT rules, and TURN allocation health.
slyber_turn bench (or benchmark) - a multi-threaded generator of WebRTC-like media traffic that calculates throughput, packet loss, round-trip and one-way latency (RTT / One-Way Latency), as well as media jitter according to RFC 3550.

1. Deep Diagnostics Tool

The doctor command performs a comprehensive node check: from .env syntax validity and CORS configuration to an actual test allocation of a relay port from the pool.

Running Diagnostics

To check the server using the current configuration (/opt/slyber_turn/.env), run:

Bash

sudo slyber_turn doctor

If needed, you can override any configuration parameter on the fly using command-line flags:

Bash

sudo slyber_turn doctor -public-ip 203.0.113.50 -udp-port 3478 -realm rtc.example.com

What `doctor` Checks Step by Step

Diagnostics run in 4 sequential stages:

Stage 1. Configuration and Port Pool Capacity Validation

The utility loads the settings, verifies the secret key length (TURN_SECRET must be at least 16 characters), checks for conflicts between UDP, TCP, TLS, and HTTP ports, and then calculates the maximum capacity of concurrent relay sessions using the formula RELAY_MAX_PORT - RELAY_MIN_PORT + 1.

Stage 2. HTTP API Security Audit

Checks the security configuration of the temporary credentials issuance endpoint /api/turn-credentials:

If the ALLOWED_ORIGINS list is empty and strict mode REQUIRE_SECRET=true is not enabled, the utility outputs a [WARN] warning because browser CORS requests will be rejected.
If protection is configured properly, an [OK] status is displayed along with the number of allowed origins and the require_secret status.

Stage 3. External STUN Check and NAT Verification

The utility queries external STUN servers (stun.l.google.com:19302, stun.cloudflare.com:3478, stun1.l.google.com:19302) with a 3-second timeout, measures network RTT, and determines the actual external IP address your server uses to access the internet:

If the detected external IP matches the configured PUBLIC_IP, the check passes ([OK]).
If the addresses differ (for example, the server is behind a cloud 1:1 NAT or an incorrect IP is specified in .env), a [WARN] IP mismatch warning is displayed.

Analytics Pollution Protection: For test connections, doctor and bench use the reserved system user identifier doctor_probe. The server core and telemetry system automatically recognize this name and do not record diagnostic checks in user connection counters, the active session list, or the events.jsonl event log. You can run slyber_turn doctor from cron or monitoring systems every minute without skewing business metrics.

`slyber_turn doctor` Output Examples

Successful check of a running server:

Code

[INFO] Running slyber_turn doctor diagnostics...
--------------------------------------------------
[OK] Configuration valid (realm: turn.local, relay port pool: 49152-65535, capacity: 16384 sessions)
[OK] HTTP API protection active (origins: 1, require_secret: false)
[OK] STUN check passed in 18ms: PUBLIC_IP (203.0.113.50) matches external IP
[INFO] UDP port 3478 is already in use - switching to running server verification (Live Probe)...
[OK] HTTP API (:8080/healthz) responded normally (200 OK)
[OK] TURN Allocate succeeded in 412µs (allocated relay address: 203.0.113.50:51024)
--------------------------------------------------
[OK] Diagnostics completed successfully. Server is ready.

Successful check before starting the service (Self-Test Mode):

Plaintext

Code

[INFO] Running slyber_turn doctor diagnostics...
--------------------------------------------------
[OK] Configuration valid (realm: turn.local, relay port pool: 49152-65535, capacity: 16384 sessions)
[OK] HTTP API protection active (origins: 1, require_secret: false)
[OK] STUN check passed in 21ms: PUBLIC_IP (203.0.113.50) matches external IP
[INFO] Ports UDP 3478 / TCP 3478 are free - starting isolated Self-Test...
[OK] TURN Allocate succeeded in 385µs (allocated relay address: 203.0.113.50:49152)
--------------------------------------------------
[OK] Diagnostics completed successfully. Server is ready.

Diagnostic Messages Reference and Troubleshooting

doctor Message	Cause	How to Fix
`[WARN] IP mismatch: configured PUBLIC_IP=..., but external STUN reported ...`	The IP address specified in `.env` does not match the server's actual outbound IP.	Set `PUBLIC_IP="auto"` in `/opt/slyber_turn/.env` or specify the correct public IPv4 address of the server and restart the service.
`[FAIL] UDP port 3478 is free, but TCP port 3478 is already in use`	The TURN server is stopped, but TCP port `3478` (or `TLS_PORT` `5349`) is occupied by another third-party process.	Check which process is using the port (`sudo ss -tulpn \| grep 3478`), stop the conflicting process, or change the port in `.env`.
`[WARN] HTTP API (:8080/healthz) is not responding`	The TURN server is running, but the HTTP API is not responding (or was started with different flags/port).	Check the service status (`sudo systemctl status slyber_turn`) and the `HTTP_ENABLED` / `HTTP_PORT` settings in `.env`.
`[FAIL] TURN Allocate failed on 127.0.0.1:3478`	The running server did not respond to the port allocation request or rejected authentication.	Make sure port `3478` is not occupied by another STUN/TURN server (e.g., an old `coturn` instance) and that you are running `doctor` with the same `.env` file used by the running service.

2. Load Testing and Media Quality Assessment: `slyber_turn bench`

The slyber_turn bench command (alias: slyber_turn benchmark) generates realistic load on the TURN server, simulating dozens and hundreds of concurrent WebRTC audio and video calls.

Benchmark Architecture and Algorithm

Testing in bench is executed in 5 phases:

Determining the Testing Mode (mode):

self-test: If the -target flag is not specified and UDP_PORT is free, the utility automatically spins up an internal isolated TURN server on 127.0.0.1 (temporarily enabling loopback traffic via SetAllowLoopback(true)) and tests it.
live-local: If the -target flag is not specified and UDP_PORT is already in use by a running slyber_turn server, the test automatically runs against the live local server (PUBLIC_IP:UDP_PORT).
live-remote: If the -target <host:port> flag is explicitly passed, the test runs over the network against the specified remote TURN server.

Mass Parallel Allocation (Allocation Setup):

N clients (-clients) are launched concurrently in separate goroutines; each opens a UDP socket, completes HMAC-SHA1 authentication, and requests a relay port (Allocate) from the server.
The exact port allocation time is recorded for each client, which is used to calculate the avg, p95, and max metrics.

Pairing and Channel Warmup (Warmup):

All successfully connected clients are grouped into pairs (Client A ↔ Client B).
Each pair exchanges an initial 24-byte packet via their allocated relay addresses and waits 80 ms to establish TURN permissions (CreatePermission) in both directions.

Bidirectional Media Packet Forwarding via the SLYB Binary Protocol:

For the specified duration (-duration), each client sends UDP packets of the specified size (-size) at the specified rate (-pps) to its peer through the TURN relay.
The first 24 bytes of each packet contain a special header:
Bytes 0..3: Protocol magic signature 0x534C5942 (ASCII: "SLYB").
Bytes 4..7: Direction flag (0 = PING, 1 = PONG).
Bytes 8..15: Packet sequence number (seq, uint64).
Bytes 16..23: Exact send timestamp in nanoseconds (UnixNano, uint64).
When Client B receives a PING packet, it calculates the one-way transit time, updates the smoothed media jitter according to the RFC 3550 formula ($J = J + (\vert{}D(i-1, i)\vert{} - J)/16$), flips the packet flag to PONG (1), and immediately sends it back through the TURN relay to Client A.
When Client A receives the returning PONG (having traversed 4 relay legs: A -> TURN -> B -> TURN -> A), it records the delivery and calculates the full round-trip RTT.

Waiting for In-Flight Packets and Generating the Report:

Once sending finishes, bench waits up to 1500 ms (waitForInFlightPackets) to properly receive the last in-flight packets, and then generates a text or JSON report.

`slyber_turn bench` Launch Flags

Benchmark-Specific Flags:

CLI Flag	Default	Description
`-target`	`""` (auto-select)	Target TURN server address in `host:port` format. If omitted, the local server is tested (`live-local` or `self-test`).
`-clients`	`10`	Number of concurrent TURN clients (allocations). If an odd number is specified, it is automatically incremented by `1` to form complete peer pairs.
`-duration`	`5s`	Duration of the active media traffic generation phase (e.g., `5s`, `30s`, `1m`).
`-size`	`1200`	Size of each UDP packet in bytes (minimum `24` bytes; `1200` bytes matches the typical WebRTC VP8/VP9/H.264 video packet size).
`-pps`	`50`	Number of packets per second sent by each client (`50 pps` = 1 packet every 20 ms, which matches a WebRTC Opus audio stream).
`-json`	`false`	Output the final report in JSON format (convenient for CI/CD pipelines and automated monitoring).

Additional Configuration Flags (for remote or custom testing):

The bench command also supports all base configuration flags: -secret (server secret key), -realm (domain realm), -udp-port, -tcp-port, -public-ip, and -max-kbps (testing traffic shaper behavior under load).

Usage Examples

1. Quick default local test (10 clients, 5 seconds):

Bash

sudo slyber_turn bench

2. High video-load stress test (100 clients / 50 pairs, 30 seconds, 100 packets/sec at 1200 bytes):

Bash

sudo slyber_turn bench -clients 100 -duration 30s -size 1200 -pps 100

3. Remote testing of a production server from another machine with JSON output:

Code

slyber_turn bench \
  -target 203.0.113.50:3478 \
  -secret "your-production-turn-secret-key" \
  -realm "turn.yourdomain.com" \
  -clients 20 \
  -duration 10s \
  -json

Analyzing Benchmark Results

1. Standard Text Report:

Code

[INFO] Starting Slyber Turn benchmark against 127.0.0.1:3478 (mode: live-local, clients: 20, duration: 5s, payload: 1200 B)...
----------------------------------------------------------------------
  Allocation Setup : 20/20 succeeded | avg: 1.42 ms | p95: 2.15 ms | max: 2.40 ms
  Relay Throughput : 19.20 Mbps (2000 packets/sec, 11.44 MB relayed)
  Packet Delivery  : 5000 sent | 5000 recv | 0.00% loss (0 dropped)
  Relay Latency    : ~0.35 ms one-way (4-leg loop RTT avg: 0.70 ms | p50: 0.65 ms | p95: 1.10 ms)
  Media Jitter     : 0.12 ms (RFC 3550)
  Quality Verdict  : [OK] EXCELLENT (Ready for low-latency HD Voice & Video)
----------------------------------------------------------------------

2. JSON Report (`-json` flag):

When the -json flag is specified, the utility outputs the complete metrics structure (including additional percentiles rtt_min_ms, rtt_p99_ms, rtt_max_ms, and the exact number of relayed bytes bytes_relayed):

JSON

{
  "target": "127.0.0.1:3478",
  "mode": "live-local",
  "clients": 20,
  "duration_sec": 5.0,
  "alloc_success": 20,
  "alloc_failed": 0,
  "alloc_avg_ms": 1.42,
  "alloc_p95_ms": 2.15,
  "alloc_max_ms": 2.4,
  "packets_sent": 5000,
  "packets_received": 5000,
  "packets_lost": 0,
  "packet_loss_percent": 0.0,
  "bytes_relayed": 12000000,
  "throughput_mbps": 19.2,
  "packets_per_sec": 2000.0,
  "rtt_min_ms": 0.41,
  "rtt_avg_ms": 0.7,
  "rtt_p50_ms": 0.65,
  "rtt_p95_ms": 1.1,
  "rtt_p99_ms": 1.48,
  "rtt_max_ms": 1.85,
  "one_way_relay_avg_ms": 0.35,
  "jitter_ms": 0.12,
  "verdict": "[OK] EXCELLENT (Ready for low-latency HD Voice & Video)"
}

What Each Report Metric Means:

Allocation Setup (alloc_avg_ms / alloc_p95_ms / alloc_max_ms): Handshake and relay port allocation speed under load. Indicates how quickly new call participants connect.
Relay Throughput (throughput_mbps / packets_per_sec / bytes_relayed): Total media traffic throughput across the TURN server in megabits per second and packets per second (counts both inbound PING and return PONG packets).
Packet Delivery (packet_loss_percent / packets_lost): Percentage of lost UDP packets. For high-quality communication on a local server, loss should be 0.00%.
Relay Latency (one_way_relay_avg_ms and rtt_*):
4-leg loop RTT - full round-trip time of a packet across two relay ports (Client A -> TURN -> Client B -> TURN -> Client A).
one-way (RTT / 2) - net one-way relay latency (Client A -> TURN -> Client B) actually experienced by call participants.
Media Jitter (jitter_ms): Packet arrival interval variation calculated using the RTP RFC 3550 algorithm. Low jitter guarantees the absence of "robotic" voice and video stuttering.

Testing Note When Bandwidth Limiting Is Enabled (`MAX_KBPS_PER_SESSION`): If you have a traffic shaper configured in `.env` (for example, `MAX_KBPS_PER_SESSION=1000`) and run `bench` with parameters that exceed this per-client limit, the `bandwidthLimiter` rate limiter will drop excess UDP packets as designed. To test the maximum physical capacity of the server itself without artificial shaping, pass the `-max-kbps 0` flag:

Bash

sudo slyber_turn bench -clients 40 -duration 10s -max-kbps 0

1. Deep Diagnostics Tool

Running Diagnostics

What doctor Checks Step by Step

Stage 1. Configuration and Port Pool Capacity Validation

Stage 2. HTTP API Security Audit

Stage 3. External STUN Check and NAT Verification

slyber_turn doctor Output Examples

Diagnostic Messages Reference and Troubleshooting

2. Load Testing and Media Quality Assessment: slyber_turn bench

Benchmark Architecture and Algorithm

slyber_turn bench Launch Flags

Benchmark-Specific Flags:

Additional Configuration Flags (for remote or custom testing):

Usage Examples

Analyzing Benchmark Results

1. Standard Text Report:

2. JSON Report (-json flag):

What Each Report Metric Means:

What `doctor` Checks Step by Step

`slyber_turn doctor` Output Examples

2. Load Testing and Media Quality Assessment: `slyber_turn bench`

`slyber_turn bench` Launch Flags

2. JSON Report (`-json` flag):