Skip to content
Prague
Improve this page

Health and Metrics

Goal: an orchestrator that restarts a dead consumer and stops routing to a replica that is still loading, and a dashboard that shows cache sizes and message rates.

Prerequisite: a host with AddKafkaCaches. The code is in HealthAndMetrics.cs. The ASP.NET mapping is compiled-only; the evaluator, the health-check service and the meter registration run as tests without a broker.

1. Register the checks

Two checks, two tags, two endpoints. Map Degraded to 503 on the readiness endpoint; the default maps it to 200.

public static void AddPragueHealth(this IServiceCollection services) {
    services.AddHealthChecks()
        .AddPragueKafkaLiveness(tags: ["live"])              // name "prague-kafka-live"; failure → Unhealthy
        .AddPragueKafkaReadiness(tags: ["ready"]);           // name "prague-kafka-ready"; failure → Degraded

    services.Configure<KafkaCachesHealthOptions>(o => {
        o.PollLoopHeartbeatTimeout = TimeSpan.FromSeconds(3);   // liveness: consumer poll loop silent this long
        o.HandlerProcessingTimeout = TimeSpan.FromSeconds(5);   // liveness: one message in a handler this long
        o.MinBrokersUp = 1;                                     // readiness: needs StatisticsEnabled (default); 0 to disable
    });
}

public static void MapPragueHealth(this WebApplication app) {
    app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = r => r.Tags.Contains("live") });
    app.MapHealthChecks("/health/ready", new HealthCheckOptions {
        Predicate = r => r.Tags.Contains("ready"),
        ResultStatusCodes = {
            [HealthStatus.Healthy] = StatusCodes.Status200OK,
            [HealthStatus.Degraded] = StatusCodes.Status503ServiceUnavailable,   // the default returns 200 for Degraded
            [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable,
        },
    });
}
Check Fails as Reasons
prague-kafka-live Unhealthy FatalLatched, PollLoopStalled, HandlerLoopFaulted, HandlerProcessingTimeout
prague-kafka-ready Degraded (Unhealthy if liveness also fails) InitialLoadIncomplete, NoPartitionAssigned, BrokersDown, PartitionsLost

The thresholds live in KafkaCachesHealthOptions: PollLoopHeartbeatTimeout (3 s), HandlerProcessingTimeout (5 s), MinBrokersUp (1). BrokersDown needs librdkafka statistics, which are on by default; set MinBrokersUp = 0 if you turn them off. The full matrix is in Kafka health checks.

2. Read a verdict

The checks are thin wrappers over KafkaCachesHealthEvaluator.Evaluate, which you can call yourself on the statistics object, for a test or a custom check.

var statistics = new KafkaCachesStatistics();           // the singleton AddKafkaCaches registers
var consumer = statistics.GetOrAddConsumer("kafkaCaches");   // one per AddKafkaCaches section, never polled yet

var verdict = KafkaCachesHealthEvaluator.Evaluate(consumer, new KafkaCachesHealthOptions());

// Liveness: the poll loop has never reported in. Readiness: no broker is known to be UP.
Console.WriteLine(string.Join(", ", verdict.LivenessFailures));    // PollLoopStalled
Console.WriteLine(string.Join(", ", verdict.ReadinessFailures));   // BrokersDown

Through HealthCheckService the same verdict shows up as entry status plus a Data dictionary with the failure lists per consumer:

var health = provider.GetRequiredService<HealthCheckService>();

var noConsumers = await health.CheckHealthAsync();      // before AddKafkaCaches started anything: nothing to judge
provider.GetRequiredService<KafkaCachesStatistics>().GetOrAddConsumer("kafkaCaches");
var stalled = await health.CheckHealthAsync();

foreach (var (name, entry) in stalled.Entries)
    Console.WriteLine($"{name}: {entry.Status} {string.Join(" ", entry.Data.Select(d => $"{d.Key}={d.Value}"))}");
// prague-kafka-live: Unhealthy kafkaCaches.failures=[PollLoopStalled]
// prague-kafka-ready: Unhealthy kafkaCaches.liveness_failures=[PollLoopStalled] kafkaCaches.readiness_failures=[BrokersDown]

3. Export metrics

AddPragueKafkaInstrumentation registers the Prague.Kafka meter with any OpenTelemetry metrics pipeline; an optional prefix namespaces the instruments.

public static void AddPragueMetrics(this IServiceCollection services) {
    services.AddOpenTelemetry().WithMetrics(m => m
        .AddPragueKafkaInstrumentation()                     // meter "Prague.Kafka": prague.kafka.* instruments
        // .AddPragueKafkaInstrumentation("acme")            // → acme.prague.kafka.*
        // .AddPrometheusExporter()                          // or OTLP; any exporter works
    );
}

The meter publishes eight observable instruments; the test lists them with a MeterListener:

var names = new SortedSet<string>();
using var listener = new MeterListener {
    InstrumentPublished = (instrument, _) => {
        if (instrument.Meter.Name == "Prague.Kafka")
            names.Add(instrument.Name);
    },
};
listener.Start();
provider.GetRequiredService<MeterProvider>();           // builds the provider, which creates the instruments
Instrument Tags
prague.kafka.consumer.partitions.assigned, consumer.broker.rtt.p99, consumer.health consumer
prague.kafka.cache.size, cache.messages.received, cache.health consumer, cache
prague.kafka.cache.index.size consumer, cache, index, kind
prague.kafka.cache.index.capacity consumer, cache, index

consumer.health and cache.health are 1 or 0 and use the same predicates as the checks (consumer-level failures on the first, per-cache ones on the second), so an alert on them and a 503 from /health/ready agree.

Pitfalls

  • One readiness endpoint for everything. Readiness fails during the initial load by design; if the same endpoint gates your database check, a long replay looks like a database outage. Tag and split.
  • Degraded returning 200. Kubernetes keeps routing. Set ResultStatusCodes as in step 1.
  • A handler that awaits I/O. HandlerProcessingTimeout fires after 5 s and liveness restarts the pod; the fix is in the handler (React to changes), not in the timeout.
  • No consumers yet. Before AddKafkaCaches has created a consumer, both checks report Healthy: there is nothing to judge. Readiness becomes meaningful once the host starts.
Prague documentation