Observability

Metrics#

Built-in Prometheus endpoint#

GET /metrics is always available and returns metrics in the Prometheus text exposition format. No configuration is required.

Server-wide counters updated by the request loop:

Per-route metrics with MetricsLayer#

Add MetricsLayer to any application to record per-route counters and latency histograms:

use rust_web_server::app::App;
use rust_web_server::core::New;
use rust_web_server::metrics::MetricsLayer;

let app = App::new().wrap(MetricsLayer);

MetricsLayer adds:

Circuit breaker state#

Present automatically — no MetricsLayer or other opt-in needed — for every backend known to circuit_breaker::global():

Wire ReverseProxy::with_circuit_breaker(Arc::new(circuit_breaker::global())) to get this metric for free alongside automatic breaker integration.

TLS#

rws_tls_handshakes_total{tls_protocol_version,tls_cipher,tls_group}, rws_tls_handshake_failures_total and rws_tls_certificate_expiry_timestamp_seconds{name} — see TLS metrics. Alert on rws_tls_certificate_expiry_timestamp_seconds - time() < 14 * 86400.

Upstreams#

Config-file proxies export rws_upstream_requests_total, rws_upstream_request_duration_seconds, rws_upstream_retries_total, rws_upstream_backend_up and rws_upstream_circuit_breaker_state per upstream and backend — see Upstream metrics.

OTLP metrics export#

http.server.request.duration can also be pushed to an OpenTelemetry Collector (or any OTLP/HTTP endpoint: Grafana Cloud, Honeycomb, Datadog's OTLP intake) — the same data /metrics serves, so the two never disagree. The rws binary starts the exporter when an endpoint is set:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318   # metrics go to /v1/metrics, spans to /v1/traces
export OTEL_SERVICE_NAME=edge-1                                   # default rws
export OTEL_METRIC_EXPORT_INTERVAL=15000                          # ms, default 60000
export OTEL_EXPORTER_OTLP_HEADERS="api-key=secret"               # optional, key=value,…
rws

Each export is an OTLP/HTTP JSON ExportMetricsServiceRequest with cumulative temporality and the convention's explicit buckets, attributes http.request.method, http.response.status_code, url.scheme, network.protocol.name and network.protocol.version. A last export is sent on SIGTERM/SIGINT. From Rust: otel::metrics::start(MetricsExportConfig { .. }) or otel::metrics::start_from_env(), and otel::shutdown() before exit.

Grafana / PromQL queries#

# Request rate (requests per second over 1 m)
rate(rws_requests_total[1m])

# Error rate
rate(rws_errors_total[1m])

# p99 latency per route
histogram_quantile(0.99, rate(rws_route_duration_seconds_bucket[5m]))

# Top routes by request count
topk(10, sum by (path) (rate(rws_route_requests_total[5m])))

# Backends currently tripped open
rws_circuit_breaker_state == 2

Logs#

JSON structured logs (default)#

By default the server writes one newline-delimited JSON object per request to stdout:

{"time":"2026-07-02T12:00:00Z","remote_addr":"203.0.113.9","method":"GET","path":"/docs/","protocol":"HTTP/2.0","status":200,"bytes":14384,"host":"example.com","user_agent":"Mozilla/5.0 …","referer":"https://www.google.com/"}

host, user_agent and referer come last, each only when the request sent the header (since 18.15.0). JSON logs are well-suited for ingestion by Loki, Fluentd, Datadog, and other structured-log collectors.

Combined Log Format#

Set RWS_CONFIG_LOG_FORMAT=combined (via environment variable or rws.config.toml) for Apache Combined Log Format (CLF):

127.0.0.1 - - [02/Jul/2026:12:00:00 +0000] "GET /healthz HTTP/1.1" 200 2

This format is compatible with GoAccess, AWStats, and most log-analysis tools.

To produce the same line yourself — in your own middleware or a custom server loop — call Log::combined:

use rust_web_server::log::Log;

// request: &Request, response: &Response, peer_addr: &SocketAddr
let line = Log::combined(&request, &response, &peer_addr);
println!("{}", line);
// 192.168.1.1 - - [29/Jun/2026:14:23:05 +0000] "GET /index.html HTTP/1.1" 200 1234

RWS_CONFIG_LOG_FORMAT=off writes no access log at all.

Buffered writes#

Access-log lines are buffered per worker thread and written to stdout by a background thread every 100 ms — at once when a thread's buffer passes 32 KiB, when a thread exits, and when the process exits normally (including a graceful shutdown on SIGTERM/SIGINT). Writing each line with its own write under the process-wide stdout lock cost about a sixth of a small request's CPU. A line can therefore appear up to 100 ms after its request; lines from one thread keep their order, and every line carries its own timestamp. Code that ends the process some other way can call Log::flush_access_log() first.

Writing to a file#

Set access_log (RWS_CONFIG_ACCESS_LOG, since 18.1.0) to write the access log to a file instead of stdout:

log_format          = "combined"
access_log          = "/var/log/rws/access.log"
access_log_max_size = "100M"    # rotate past 100 MiB (the default since 18.15.0)
access_log_keep     = 5         # access.log.1 … access.log.5

• Lines are appended; the file and its directory must be writable by rws.

• Rotation by rws (the default): past access_log_max_size (100 MiB unless set) the file is renamed access.log.1 (older ones shift up, the oldest past access_log_keep is deleted) and a new one is started — at most 600 MiB with the defaults. Before 18.15.0 the default was 0, never rotating.

• Rotation by logrotate: set access_log_max_size to 0 and have logrotate send SIGUSR1 after moving the file; rws reopens the path before its next write:

  /var/log/rws/access.log {
      daily
      rotate 14
      compress
      delaycompress
      missingok
      notifempty
      postrotate
          systemctl kill -s USR1 rws.service
      endscript
  }

  Without an access-log file SIGUSR1 keeps its default action (it ends the process), so don't send it to an rws logging to stdout.

Disk safety#

A busy server writes a lot: at 10,000 requests a second an access log grows by about 170 GB a day. Everything rws writes is bounded:

min_free_disk (RWS_CONFIG_MIN_FREE_DISK, since 18.15.0): before writing the access-log file or a disk-cache entry, rws checks the free space of that filesystem (at most every 5 s). Under the floor it stops writing there — the lines are dropped and counted, the cache serves what it has — logs one warning, and resumes when space is back. A full disk then costs log lines, never the server, a database or a mail spool on the same disk.

min_free_disk = "2G"     # or "5%"; default: the larger of 1% and 1 GiB; "0" turns it off

The space counted is what an unprivileged process may use, so the root reserve most filesystems keep is already set aside. /metrics has rws_disk_free_bytes{dir}, rws_disk_low{dir}, rws_access_log_dropped_lines_total and rws_cache_skipped_writes_total; the admin API's /status lists each directory, and the admin UI's Overview warns while one is low.

Logging to stdout leaves the bound to what collects it:

• journald keeps at most SystemMaxUse (default 10% of the filesystem, up to 4 GiB) and drops lines past RateLimitBurst per RateLimitIntervalSec. For a busy server set them in /etc/systemd/journald.conf.d/rws.conf:

  [Journal]
  SystemMaxUse=2G
  RateLimitIntervalSec=0

  (RateLimitIntervalSec=0 stops journald dropping rws's lines; the size cap still holds.)

• Docker: the default json-file driver never rotates — set --log-opt max-size=100m --log-opt max-file=5 (or the same in daemon.json).

• Kubernetes rotates container logs at 10 MiB × 5 by default (containerLogMaxSize, containerLogMaxFiles).

If you don't read every line, log_format = "off" writes none; metrics and the admin API keep working.

Log aggregation in Kubernetes#

The server writes all logs to stdout. Kubernetes log aggregation pipelines pick them up automatically:

# View logs in real time
kubectl logs -f deployment/my-rws-app

# Ship to Loki via Promtail — no application config required; just label the pod

Tracing#

OtelLayer#

OtelLayer implements Middleware and propagates W3C traceparent headers. Add it to your application and configure an OTLP exporter:

use rust_web_server::app::App;
use rust_web_server::core::New;
use rust_web_server::otel::{self, ExporterConfig, OtelLayer, TracingConfig};

otel::setup(TracingConfig {
    service_name: "my-service".to_string(),
    exporter: ExporterConfig::Otlp {
        endpoint: "http://tempo:4318".to_string(),
    },
    ..TracingConfig::default()
});

let app = App::new().wrap(OtelLayer);

The layer reads the incoming traceparent header, creates a span for each request (a child of the caller's span when one was sent), and exports completed spans to the configured OTLP endpoint. Start nested spans inside a request with otel::span(name) / otel::client_span(name) — see Distributed Tracing.

The rws binary adds OtelLayer and exports spans by itself when OTEL_EXPORTER_OTLP_ENDPOINT is set — see Environment variable setup.

Supported backends#

• Grafana Tempo — point endpoint at the OTLP HTTP port (default 4318)
• Jaeger — enable the OTLP receiver and point at port 4318
• OpenTelemetry Collector — use as a fan-out proxy to multiple backends

Distributed tracing across services#

Include the traceparent header in outbound requests using the synchronous HTTP client:

use rust_web_server::http_client::Client;

let response = Client::new()
    .get("http://downstream-service/api/data")
    .header("traceparent", incoming_traceparent)
    .send()?;

Inside a traced request rws does this itself: the HTTP client, the proxies and database statements get client spans, and traceparent plus baggage are sent on — see Automatic client spans and propagation.

Health endpoints#

Two health endpoints are built into every server instance without any configuration:

GET /healthz — liveness#

Returns 200 OK as long as the binary is running. Use this as the Kubernetes liveness probe:

livenessProbe:
  httpGet:
    path: /healthz
    port: 7878
  initialDelaySeconds: 5
  periodSeconds: 10

GET /readyz — readiness#

Returns 200 OK during normal operation. Returns 503 Service Unavailable during graceful shutdown (after SIGTERM is received, while in-flight requests are being drained). Use as the Kubernetes readiness probe:

readinessProbe:
  httpGet:
    path: /readyz
    port: 7878
  initialDelaySeconds: 3
  periodSeconds: 5

The readiness gate ensures the load balancer stops routing new traffic to a pod as soon as it begins draining, giving in-flight requests time to complete before the pod terminates.