Observability

Diagnose a live FFDB deployment

Observability is the retained project and instance performance workspace plus the lower-level Prometheus, log, and request-correlation interfaces.

Operators need time-windowed QPS, latency, errors, saturation, route hot spots, and query hot spots without copying customer SQL into a telemetry store.

Use it for routine capacity review, performance regression analysis, incident triage, and alert validation.

Requirements for Observability

• Prerequisite — A platform session with organization membership for project scope or instance administration for instance scope.
• Prerequisite — Migration 14 applied to control-plane PostgreSQL and enough database capacity for minute aggregates.
• Required value — Project or instance scope and a 1h, 6h, 24h, 7d, or 30d range.
• Required value — For request-specific diagnosis, a safe X-Request-Id and bounded log window.

Performance workspace

Open Observability in the portal for retained traffic, latency, errors, saturation, storage, route rankings, and privacy-safe query fingerprints. The current-project scope is available to organization members. The entire-instance scope and optional project filter require an instance owner or administrator.

• Choose 1 hour, 6 hours, 24 hours, 7 days, or the full 30-day retention window.
• Charts use a server-selected bounded resolution and refresh every 30 seconds.
• Route tables use stable templates, so concrete project and resource IDs are excluded.
• Capacity reports live worker processes, execution slots in use, and database and backup filesystem headroom.
• A dropped-sample warning means recorder capacity was exceeded; the portal never silently presents an incomplete sample as complete.

# Project scope: any member of the owning organization
curl --fail \
  -H "Authorization: Bearer $FFDB_PLATFORM_SESSION" \
  "http://127.0.0.1:5173/v1/projects/$FFDB_PROJECT_ID/observability?range=24h"

# Instance scope: owner or administrator only
curl --fail \
  -H "Authorization: Bearer $FFDB_PLATFORM_SESSION" \
  "http://127.0.0.1:5173/v1/instance/observability?range=7d"

Query privacy boundary

FFDB records execution duration and row counts inside the isolated database worker, but never persists raw SQL. A bounded lexer preserves keywords, operators, and structure while replacing identifiers, comments, strings, numbers, blobs, and bind parameters. The normalized shape is capped at 96 tokens and 320 characters and hashed with SHA-256 for grouping.

• No raw SQL, table or column names, comments, literal values, or bound parameters enter the telemetry tables.
• Successful statements and failed executions are counted; idempotency replays are not timed twice.
• Logical database size is sampled after successful database operations, so an idle upgraded project may initially have no size sample.
• Retained minute aggregates live in control-plane PostgreSQL and are deleted after 30 days.

Prometheus and request correlation

The retained API powers the portal. The separate /metrics endpoint remains an instance-wide, current-process Prometheus scrape for external alerting. Every inbound API request also receives X-Request-Id for correlation with structured logs and immutable audit events.

• ffdb_http_requests_total is labeled by method, stable route, and status class.
• ffdb_http_request_duration_seconds is labeled by method and stable route.
• ffdb_http_requests_inflight, authentication failures, and rate-limit denials remain available.
• Prometheus labels never include project, user, request, object key, SQL, or token IDs.

curl --fail http://127.0.0.1:5173/healthz
curl --fail http://127.0.0.1:5173/readyz
curl --fail http://127.0.0.1:5173/metrics
ffdb logs 100
ffdb health

Observability workflow

• 1. Open Observability and select the narrowest useful scope and time range.
• 2. Compare QPS, latency percentiles, and error rate across the chart window.
• 3. Inspect worker execution-slot and filesystem saturation.
• 4. Sort routes and normalized query fingerprints by frequency or p95 latency.
• 5. Correlate a specific failure by request ID, resolve the cause, and confirm recovery in the next retained buckets.

Verify observability

Operators can identify traffic, latency, error, worker, disk, route, and query-shape pressure without retaining raw SQL, identifiers, or values.

Troubleshoot observability

• Dropped samples is nonzero — inspect recorder/PostgreSQL and worker saturation before trusting the affected window as complete.
• Charts stay empty after traffic — verify migration 14 and wait for the five-second aggregate flush.
• Health is green while readiness is red — investigate dependencies rather than restarting blindly.

Continue from Observability

• Define thresholds from measured production baselines.
• Connect request-level findings to incident response and scaling.