IET Framework

Turn on traces and metrics

The question this guide answers: "Log lines are no longer enough — I need spans, durations, and failure counts per operation."

Add telemetry when you need to answer operational questions about a running process. Skip it while stdout is genuinely sufficient. Choosing capabilities covers that decision. This page shows the current API: config_from_env, OpenTelemetry::init, the Operation recorder, and shutdown.

What you add

iet-telemetry is the vocabulary — Operation, the config, and a compile-time no-op recorder. iet-telemetry-otel installs the OpenTelemetry SDK and the tracing layers. Framework features depend inward on the core crate; only your composition root depends on the exporter. Libraries never install global subscribers.

Minimal usage

Initialize once at startup, run the application, then shut the SDK down after work drains. This example is complete and compiled against this checkout:

rust
use iet_telemetry_otel::{OpenTelemetry, Operation};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let mut config = iet_telemetry_otel::config_from_env()?;
    config.service_name = "fieldnotes".into();
    let telemetry = OpenTelemetry::init(config)?;

    let result = run().await;
    let shutdown = telemetry.shutdown().await;
    result?;
    shutdown?;
    Ok(())
}

async fn run() -> iet_core::AppResult<()> {
    let confirmed = Operation::new("service", "booking", "confirm")
        .record(async {
            // The real operation goes here.
            Ok::<u64, iet_core::AppError>(1)
        })
        .await?;
    let _ = confirmed;
    Ok(())
}

The example runs run(), then shutdown(), then propagates the application result. Shutdown runs even when the application returns an error. The application error takes priority; a shutdown error is returned only when the application itself succeeded. The starter uses the same ordering in examples/htmx-starter/servers/web/src/main.rs.

The startup and shutdown lifecycle

  • config_from_env() snapshots only telemetry-related environment values — FW_TELEMETRY_* and OTEL_EXPORTER_OTLP_ENDPOINT — and returns a validated TelemetryConfig. It installs nothing. Programmatic field assignments on the returned config override the environment values. Invalid settings fail without echoing the values.
  • OpenTelemetry::init(config) installs the global tracing subscriber and the SDK exactly once. Call it once at startup, before serving. A second call fails with telemetry.already_initialized.
  • Export defaults to OTLP/HTTP protobuf at http://127.0.0.1:4318. Endpoints containing credentials or query strings are rejected; exporter authentication belongs at a local collector. Set OTEL_EXPORTER_OTLP_ENDPOINT to your collector base URL in production.
  • telemetry.shutdown().await aborts resource sampling, then performs the blocking SDK flush off the async executor under a bounded timeout (three times the configured export timeout). Drain requests and workers first. Do not skip shutdown on the error path.
  • Logs go to OTLP or JSON stdout, not both. FW_TELEMETRY_JSON_STDOUT selects JSON stdout.

No per-route or global meter setup is required in application code, but do not read that as full coverage. The framework's adapters already emit spans and metrics for HTTP requests, repository CRUD, session pool and transaction work, memory-cache lookups, and supervised workers. Coverage is partial and expanding:

  • Direct SQLx calls bypass repository instrumentation.
  • Authentication and authorization instrumentation is a remaining area, not a finished one.
  • Custom service and repository implementations are never intercepted; they must wrap their own methods.

Treat the coverage checklist in docs/telemetry-coverage.md as the current inventory rather than assuming every path is instrumented.

The recording build prerequisite

Recording is compiled out of iet-telemetry unless its recording feature is enabled. The iet-telemetry-otel dependency enables it, so depending on the exporter is what turns recording on. If you use iet-telemetry without that feature, iet_telemetry::recording_compiled() is false and Operation timers read no clocks and construct no spans; the middleware call sites still exist, but they do nothing. Even with recording compiled in, Operation::record (and start) skip their clocks unless a subscriber accepts the framework metric events — a process that never calls init still compiles the call but records nothing. Check the feature edge in crates/features/iet-telemetry-otel/Cargo.toml before assuming either cost or output.

Instrument your own operations

Operation::new(component, name, operation) builds a static, implementation-owned identity. The available methods:

  • .record(future).await — time an async operation and record ok or error.
  • .record_sync(action) — the same accounting for blocking work.
  • .lifecycle(future).await — emit start / ok / error / cancelled lifecycle logs independently of metric enablement.
  • .family(&'static str) — add a bounded sub-dimension.
  • .start() -> Stage — a manual timer; dropping an unfinished Stage records cancelled, and finish(&result) records the outcome. Most code should use record.

Use static strings only. Never use user IDs, subjects, cache keys, SQL text, or bind values as a component, name, operation, or label. Rust cannot intercept trait calls, so a custom service or repository must wrap its own methods with Operation; the framework only instruments its own adapters.

The recorder emits fw_operation_duration_seconds (histogram), fw_operation_completed (counter), and fw_operation_active (signed gauge), dimensioned by component, name, operation, and outcome. HTTP request timing uses fw_http_duration_seconds with method, route template, phase, status, and outcome.

Sampling and admission are caveats, not performance claims

  • Trace sampling is parent-based with a local trace-ID ratio for roots (FW_TELEMETRY_TRACE_SAMPLE_RATIO). Setting FW_TELEMETRY_TRACES=false stops span export; it does not stop metrics, and metrics are not trace-sampled. Head sampling cannot guarantee that error traces are retained.
  • Framework metric events are admitted before SDK aggregation. Admission enforces a process-lifetime series budget per instrument (FW_TELEMETRY_METRIC_SERIES_LIMIT, default 2000) and a 256-instrument ceiling. A new dimension that exceeds the budget drops the whole event, not just the new series; already-admitted dimensions stay eligible. Inspect telemetry.metric_series_dropped to see drops. This is an operability bound against label explosion, not a statement about throughput — do not turn it into a performance claim.
  • Export runs on SDK threads with bounded queues and per-request timeouts. Failures are counted in the trace_exports, log_exports, and metric_exports snapshots (attempts, failures, failed_records); a metric failed record is one aggregation batch, not one observation. SDK internals allocate, so the enabled path is not allocation-free.

The full settings table, profile presets, and the exported name translation for a Prometheus-compatible backend are in the technical reference in docs/telemetry.md.

Test without touching global state

iet_telemetry_otel::testing::TestTelemetry captures completed spans, events, and raw metric events under a scoped subscriber, so isolated tests do not install a global provider:

rust,ignore
use tracing::instrument::WithSubscriber;

let capture = iet_telemetry_otel::testing::TestTelemetry::new(1000);
async {
    exercise_application().await;
}.with_subscriber(capture.dispatch()).await;
let snapshot = capture.snapshot();
snapshot.assert_span_exists("fw.operation");
snapshot.assert_no_dropped_records();

For assertions on actual SDK export data, testing::sdk::TestProviders assembles test-local trace, log, and metric providers you can flush and inspect. Neither helper changes global state, and both have bounded capacity. See the technical reference in docs/telemetry.md for the exact capture API.

Compile the example, and where runtime behavior is tested

The complete startup example above was compiled (not run) in an isolated crate with local path dependencies on iet-core and iet-telemetry-otel, plus Tokio's macros and rt-multi-thread features. To reproduce that check, put it in your application's main.rs with those dependencies and run cargo check.

That check proves the API and the shutdown ordering type-check. It does not prove export. Running the example with no collector still builds the exporters and attempts network export to the default http://127.0.0.1:4318; those attempts fail, are counted in the export snapshots, and are not fatal to the application. Set OTEL_EXPORTER_OTLP_ENDPOINT to a running collector for useful output.

Runtime export is exercised by the framework's own tests, not here: crates/features/iet-telemetry-otel/tests/otlp.rs exports all three signals to a real in-process HTTP receiver, and tests/outage.rs covers collector unavailability.

IET Framework documentation — built with the IET Framework.