ADR-023: CLARISSA Portal - Observability¶
| Status | Proposed |
|---|---|
| Date | 2026-01-22 |
| Authors | Wolfram Laube, Claude (AI Assistant) |
| Supersedes | - |
| Related | ADR-021 (System Architecture), ADR-022 (Software Architecture) |
Context¶
The CLARISSA Portal is being built during the experimental phase by ~4 developers, but should later be used productively by significantly more users. The observability strategy must:
- Jetzt: Minimal, kostenfrei, schnell einsatzbereit
- Later: Scalable for production workloads
- Immer: Von Anfang an richtig angelegt (Structured Logging, Correlation IDs)
Decision¶
Strategie: "Design for Scale, Start Simple"¶
| Phase | User | Stack | Kosten |
|---|---|---|---|
| Phase 1 (jetzt) | 4 Devs | GCP Native | $0 |
| Phase 2 (50+ User) | Early Adopters | + Sentry | ~$0-26/mo |
| Phase 3 (500+ User) | Production | Full Stack | $50-500/mo |
Key Decisions¶
| Decision | Choice | Rationale |
|---|---|---|
| Logging | Structured JSON (structlog) | Machine-readable, filterable |
| Correlation | UUID, generated by Portal API | Request tracing across services |
| Metrics | GCP Cloud Monitoring | Automatic for Cloud Run |
| Alerting | GCP Alerts โ Email/Slack | Minimal aber effektiv |
| Tracing | OpenTelemetry (prepared) | Future-proof, not yet active |
Phase 1: Experimentierphase (Jetzt)¶
Architektur¶
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ GCP (kostenlos, automatisch) โ
โ โ
โ Cloud Run Services โ
โ โ โ
โ โ stdout/stderr (JSON) โ
โ โผ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Cloud Logging โ โ
โ โ โ โ
โ โ โข Structured JSON Logs โ โ
โ โ โข 30 Tage Retention (Free) โ โ
โ โ โข Filter: correlation_id, user_id, level โ โ
โ โ โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โ โ Log-based Metrics (custom) โ โ โ
โ โ โ โข error_count{service, endpoint} โ โ โ
โ โ โ โข auth_failures{reason} โ โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Cloud Monitoring โ โ
โ โ โ โ
โ โ Built-in Metrics (automatisch): โ โ
โ โ โข request_count โ โ
โ โ โข request_latencies (P50, P95, P99) โ โ
โ โ โข instance_count โ โ
โ โ โข cpu_utilization โ โ
โ โ โข memory_utilization โ โ
โ โ โ โ
โ โ Alerts: โ โ
โ โ โข Error Rate > 10% โ Email + Slack โ โ
โ โ โข Service Down โ Email + Slack โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Error Reporting โ โ
โ โ โ โ
โ โ โข Stack Traces (automatisch aus Logs) โ โ
โ โ โข Gruppierung nach Error-Typ โ โ
โ โ โข Trending/New Errors โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ
โ Kosten: $0 (Free Tier) โ
โ Setup: ~2h โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
What we are NOT building (Phase 1)¶
- โ Custom Dashboards (GCP Console reicht)
- โ Distributed Tracing (nur 2 Services)
- โ SIEM / Security Analytics
- โ Self-hosted Prometheus/Grafana
Structured Logging¶
Warum von Anfang an?¶
โ Refactor later:
print(f"User {user_id} created invoice {inv_id}")
โ Not filterable, not machine-readable
โ
Von Anfang an richtig:
logger.info("invoice_created", user_id=user_id, invoice_id=inv_id)
โ {"event": "invoice_created", "user_id": "123", "invoice_id": "INV-001"}
Implementation¶
# services/portal-api/src/core/logging.py
import structlog
from google.cloud import logging as gcp_logging
def setup_logging():
"""
Configure structured logging for Cloud Run.
Logs go to stdout as JSON โ Cloud Logging picks them up.
"""
structlog.configure(
processors=[
# Add timestamp
structlog.processors.TimeStamper(fmt="iso"),
# Add log level
structlog.processors.add_log_level,
# Format as JSON for Cloud Logging
structlog.processors.JSONRenderer()
],
wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
context_class=dict,
logger_factory=structlog.PrintLoggerFactory(),
cache_logger_on_first_use=True
)
# Usage
logger = structlog.get_logger()
logger.info("invoice_created",
user_id="user_123",
invoice_id="INV-001",
amount=1500.00
)
# Output (JSON):
# {
# "timestamp": "2026-01-22T21:30:00Z",
# "level": "info",
# "event": "invoice_created",
# "user_id": "user_123",
# "invoice_id": "INV-001",
# "amount": 1500.00
# }
Log Levels¶
| Level | Wann | Beispiel | Alert? |
|---|---|---|---|
| ERROR | Braucht Aufmerksamkeit | PDF Generation failed | โ Ja |
| WARNING | Unusual but OK | Retry succeeded, Rate limit near | No |
| INFO | Normale Operationen | Request completed, User logged in | Nein |
| DEBUG | Development only | SQL queries, Token details | No |
Correlation IDs¶
Warum?¶
Ohne Correlation ID:
Portal API Log: "PDF generation triggered"
Worker Log: "PDF generation failed: timeout"
โ Which request? For which user? ๐คท
With Correlation ID:
Portal API Log: {"correlation_id": "abc-123", "event": "pdf_triggered", ...}
Worker Log: {"correlation_id": "abc-123", "event": "pdf_failed", ...}
โ Filter: correlation_id="abc-123" โ All logs for this request โ
Wer erzeugt die Correlation ID?¶
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ
โ Browser Portal API Worker โ
โ โ โ โ โ
โ โ (optional) โ โ โ
โ โ X-Request-ID: abc โ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโบโ โ โ
โ โ โ โ โ
โ โ Middleware: โ โ
โ โ if header missing: โ โ
โ โ correlation_id = uuid4() โ โ
โ โ else: โ โ
โ โ correlation_id = validate(header) โ โ
โ โ โ โ โ
โ โ Bind to structlog context โ โ
โ โ โ โ โ
โ โ โ X-Correlation-ID: abc โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโบโ โ
โ โ โ โ โ
โ โ โ Bind to context โ
โ โ โ All logs include it โ
โ โ โ โ โ
โ โ X-Correlation-ID: abc โ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โ (for error reports) โ โ โ
โ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Implementation¶
# services/portal-api/src/core/middleware.py
import uuid
import re
import structlog
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
# Validation: alphanumeric + hyphens, max 64 chars
CORRELATION_ID_PATTERN = re.compile(r'^[a-zA-Z0-9\-]{1,64}$')
logger = structlog.get_logger()
class CorrelationIdMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
# 1. Check if client sent one
correlation_id = (
request.headers.get("X-Correlation-ID") or
request.headers.get("X-Request-ID")
)
# 2. Validate or generate
if not correlation_id or not CORRELATION_ID_PATTERN.match(correlation_id):
correlation_id = str(uuid.uuid4())
# 3. Bind to logging context (all subsequent logs include it)
structlog.contextvars.bind_contextvars(
correlation_id=correlation_id
)
# 4. Also bind user_id if authenticated
user = getattr(request.state, "user", None)
if user:
structlog.contextvars.bind_contextvars(user_id=user.get("id"))
# 5. Log request start
logger.info("request_started",
method=request.method,
path=request.url.path,
client_ip=request.client.host
)
# 6. Process request
start_time = time.perf_counter()
response = await call_next(request)
duration_ms = (time.perf_counter() - start_time) * 1000
# 7. Log request end
logger.info("request_completed",
status_code=response.status_code,
duration_ms=round(duration_ms, 2)
)
# 8. Return correlation ID to client
response.headers["X-Correlation-ID"] = correlation_id
# 9. Clear context for next request
structlog.contextvars.unbind_contextvars("correlation_id", "user_id")
return response
Propagation zum Worker¶
# services/portal-api/src/services/worker_client.py
import structlog
class WorkerClient:
async def generate_pdf(self, invoice_id: str) -> dict:
# Get correlation ID from current context
ctx = structlog.contextvars.get_contextvars()
correlation_id = ctx.get("correlation_id", str(uuid.uuid4()))
async with httpx.AsyncClient() as client:
response = await client.post(
f"{self.worker_url}/worker/billing/generate-pdf",
json={"invoice_id": invoice_id},
headers={
"Authorization": f"Bearer {token}",
"X-Correlation-ID": correlation_id # Propagate!
}
)
return response.json()
Log Output Beispiel¶
// Portal API
{
"timestamp": "2026-01-22T21:30:00.000Z",
"level": "info",
"event": "request_started",
"correlation_id": "550e8400-e29b-41d4-a716-446655440000",
"user_id": "user_123",
"method": "POST",
"path": "/api/v1/billing/invoices/INV-001/generate-pdf"
}
{
"timestamp": "2026-01-22T21:30:00.150Z",
"level": "info",
"event": "pdf_generation_triggered",
"correlation_id": "550e8400-e29b-41d4-a716-446655440000",
"user_id": "user_123",
"invoice_id": "INV-001"
}
// Worker Service
{
"timestamp": "2026-01-22T21:30:00.200Z",
"level": "info",
"event": "pdf_generation_started",
"correlation_id": "550e8400-e29b-41d4-a716-446655440000",
"invoice_id": "INV-001"
}
{
"timestamp": "2026-01-22T21:30:05.500Z",
"level": "info",
"event": "pdf_generation_completed",
"correlation_id": "550e8400-e29b-41d4-a716-446655440000",
"invoice_id": "INV-001",
"pdf_size_bytes": 145230,
"duration_ms": 5300
}
Cloud Logging Query¶
-- All logs for a request
resource.type="cloud_run_revision"
jsonPayload.correlation_id="550e8400-e29b-41d4-a716-446655440000"
-- All errors for a user
resource.type="cloud_run_revision"
jsonPayload.user_id="user_123"
jsonPayload.level="error"
Alerting¶
Phase 1 Alerts¶
# GCP Cloud Monitoring Alert Policies
alerts:
- name: "CLARISSA - High Error Rate"
condition:
filter: |
resource.type="cloud_run_revision"
resource.labels.service_name="clarissa-portal-api"
metric.type="run.googleapis.com/request_count"
metric.labels.response_code_class="5xx"
comparison: COMPARISON_GT
threshold: 0.1 # 10%
duration: 300s # 5 Minuten
notification:
- email: wolfram.laube@blauweiss-edv.at
- slack_webhook: $SLACK_WEBHOOK_URL
- name: "CLARISSA - Service Unavailable"
condition:
filter: |
resource.type="cloud_run_revision"
resource.labels.service_name="clarissa-portal-api"
metric.type="run.googleapis.com/request_count"
absence_duration: 600s # 10 Minuten kein Traffic
notification:
- email: wolfram.laube@blauweiss-edv.at
- name: "CLARISSA - Auth Failures Spike"
condition:
# Log-based metric
filter: |
resource.type="cloud_run_revision"
jsonPayload.event="auth_failed"
comparison: COMPARISON_GT
threshold: 50
duration: 60s # 50 failures/minute = possible brute force
notification:
- email: wolfram.laube@blauweiss-edv.at
- slack_webhook: $SLACK_WEBHOOK_URL
Frontend Error Reporting¶
// frontend/portal/assets/js/api.js
async function apiCall(endpoint, options = {}) {
const response = await fetch(`${API_BASE}${endpoint}`, {
...options,
credentials: 'include'
});
const correlationId = response.headers.get('X-Correlation-ID');
if (!response.ok) {
// User can send us this for support
console.error(`Request failed. Correlation ID: ${correlationId}`);
// Optional: Show to user
showError(`Something went wrong. Reference: ${correlationId.slice(0, 8)}`);
throw new Error(`API Error: ${response.status}`);
}
return response.json();
}
Phase 2: Early Adopters (50+ User)¶
+ Sentry for Error Tracking¶
# services/portal-api/src/main.py
import sentry_sdk
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.httpx import HttpxIntegration
sentry_sdk.init(
dsn=settings.SENTRY_DSN,
integrations=[
FastApiIntegration(),
HttpxIntegration(),
],
traces_sample_rate=0.1, # 10% of requests
environment=settings.ENVIRONMENT,
)
# Correlation ID in Sentry
@app.middleware("http")
async def sentry_context(request: Request, call_next):
correlation_id = request.headers.get("X-Correlation-ID", str(uuid.uuid4()))
with sentry_sdk.configure_scope() as scope:
scope.set_tag("correlation_id", correlation_id)
scope.set_user({"id": get_user_id(request)})
return await call_next(request)
Sentry bringt: - Error Grouping (gleiche Errors zusammengefasst) - Stack Traces mit Context - Release Tracking - Performance Monitoring - User Feedback Widget
Kosten: $0 (Developer) / $26/mo (Team)
Phase 3: Production (500+ User)¶
Option A: GCP Native (einfacher)¶
Cloud Logging โ Log Analytics (SQL queries)
Cloud Monitoring โ Custom Dashboards
Cloud Trace โ Distributed Tracing
Cloud Profiler โ Performance Analysis
Alerting โ PagerDuty/Opsgenie Integration
Option B: Grafana Stack (mehr Kontrolle)¶
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Grafana Cloud (SaaS) โ
โ oder Self-Hosted โ
โ โ
โ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโ โ
โ โ Loki โ โ Prometheus โ โ Tempo โ โ
โ โ (Logs) โ โ (Metrics) โ โ (Traces) โ โ
โ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโ โ
โ โ โ โ โ
โ โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโ โ
โ โผ โ
โ โโโโโโโโโโโโโโโ โ
โ โ Grafana โ โ
โ โ Dashboards โ โ
โ โโโโโโโโโโโโโโโ โ
โ โ โ
โ โโโโโโโโโโโโโโโ โ
โ โ OnCall โ โ
โ โ (Alerting) โ โ
โ โโโโโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Decision will be made when we get there - important is that we are prepared through OpenTelemetry.
OpenTelemetry Vorbereitung¶
Even though we do not actively use tracing in Phase 1, we instrument from the start with OpenTelemetry. This makes later switching to any backend trivial.
# services/portal-api/src/core/telemetry.py
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
def setup_telemetry(app, export_traces: bool = False):
"""
Setup OpenTelemetry instrumentation.
Phase 1: Instrumentation only (no export)
Phase 3: Add exporter (Cloud Trace, Jaeger, Tempo)
"""
# Set up provider
provider = TracerProvider()
trace.set_tracer_provider(provider)
# Phase 3: Uncomment to export traces
# if export_traces:
# from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
# exporter = CloudTraceSpanExporter()
# provider.add_span_processor(BatchSpanProcessor(exporter))
# Instrument FastAPI (automatic spans for all requests)
FastAPIInstrumentor.instrument_app(app)
# Instrument HTTPX (automatic spans for outgoing requests)
HTTPXClientInstrumentor().instrument()
Summary¶
| Was | Phase 1 | Phase 2 | Phase 3 |
|---|---|---|---|
| Logging | structlog โ Cloud Logging | gleich | + Log Analytics |
| Metrics | Cloud Run built-in | gleich | + Custom Dashboards |
| Errors | Cloud Error Reporting | + Sentry | Sentry |
| Tracing | (prepared) | (prepared) | Cloud Trace / Tempo |
| Alerting | Email + Slack | gleich | PagerDuty/Opsgenie |
| Kosten | $0 | ~$26/mo | $50-500/mo |
Nicht verhandelbar (alle Phasen)¶
- โ Structured Logging (JSON, nicht print())
- โ Correlation IDs (Portal API erzeugt, propagiert)
- โ User ID in Logs (for support)
- โ Error Alerting (Email minimum)
Implementation Checklist¶
Phase 1 (jetzt)¶
- [ ] structlog konfigurieren
- [ ] CorrelationIdMiddleware implementieren
- [ ] Worker: Correlation ID aus Header lesen
- [ ] Log-based Metric: auth_failures
- [ ] Alert: Error Rate > 10%
- [ ] Alert: Service Down
- [ ] Frontend: Correlation ID in Errors anzeigen
Phase 2 (bei 50+ User)¶
- [ ] Sentry Account erstellen
- [ ] Sentry SDK integrieren
- [ ] Release Tracking einrichten
Phase 3 (bei 500+ User)¶
- [ ] Decision: GCP Native vs Grafana
- [ ] OpenTelemetry Exporter aktivieren
- [ ] Custom Dashboards
- [ ] PagerDuty/Opsgenie Integration