profit/agent-governance
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Phase 10-12 implementation: multi-tenant, marketplace, observability

Browse Source 
Major additions:
- marketplace/: Agent template registry with FTS5 search, ratings, versioning
- observability/: Prometheus metrics, distributed tracing, structured logging
- ledger/migrations/: Database migration scripts for multi-tenant support
- tests/governance/: 15 new test files for phases 6-12 (295 total tests)
- bin/validate-phases: Full 12-phase validation script

New features:
- Multi-tenant support with tenant isolation and quota enforcement
- Agent marketplace with semantic versioning and search
- Observability with metrics, tracing, and log correlation
- Tier-1 agent bootstrap scripts

Updated components:
- ledger/api.py: Extended API for tenants, marketplace, observability
- ledger/schema.sql: Added tenant, project, marketplace tables
- testing/framework.ts: Enhanced test framework
- checkpoint/checkpoint.py: Improved checkpoint management

Archived:
- External integrations (Slack/GitHub/PagerDuty) moved to .archive/
- Old checkpoint files cleaned up

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
profit
2026-01-24 18:39:47 -05:00
co-authored by Claude Opus 4.5
parent 4b5b3b0a2d
commit 8c6e7831e9
184 changed files with 16249 additions and 13673 deletions
Download Patch File Download Diff File Expand all files Collapse all files
observability/README.md
+151
View File
@@ -0,0 +1,151 @@
# Observability
> Metrics, tracing, and structured logging for the agent governance system
## Overview
This module provides comprehensive observability infrastructure including Prometheus-format metrics, distributed tracing with span propagation, and structured JSON logging with trace correlation.
## Key Files
| File | Description |
|------|-------------|
| `metrics.py` | Prometheus metrics (Counter, Gauge, Histogram) with FastAPI router |
| `tracing.py` | Distributed tracing with Span/Trace classes and context propagation |
| `logging.py` | Structured JSON logging with SQLite persistence |
| `__init__.py` | Module exports and unified API |
## Components
### Metrics (`metrics.py`)
Prometheus-format metrics with automatic collection:
| Metric | Type | Description |
|--------|------|-------------|
| `agent_executions_total` | Counter | Total executions by tier, action, status |
| `agent_execution_duration_seconds` | Histogram | Execution latency distribution |
| `agent_violations_total` | Counter | Violations by type and severity |
| `agent_promotions_total` | Counter | Tier promotions |
| `api_requests_total` | Counter | API requests by method, endpoint, status |
| `api_request_duration_seconds` | Histogram | Request latency |
| `component_health` | Gauge | Health status (1=healthy, 0=unhealthy) |
| `tenant_quota_usage_ratio` | Gauge | Quota usage per tenant |
### Tracing (`tracing.py`)
Distributed tracing with automatic context propagation:
- **Span**: Individual operation with timing, attributes, events
- **Trace**: Collection of related spans
- **Context Propagation**: Thread-local storage + HTTP headers (`X-Trace-ID`, `X-Span-ID`)
### Logging (`logging.py`)
Structured JSON logging with:
- Automatic trace/span ID correlation
- SQLite persistence with full-text search
- Multi-tenant support
- Configurable retention (default: 7 days)
## API Endpoints
### Metrics
- `GET /metrics` - Prometheus format export
### Tracing
- `GET /traces` - List traces with filters
- `GET /traces/{trace_id}` - Full trace details
### Logging
- `GET /logs` - Search logs with filters
- `GET /logs/trace/{trace_id}` - Logs for a trace
- `GET /logs/stats` - Log statistics
- `POST /logs/cleanup` - Clean old logs
### Health
- `GET /health/detailed` - Component health with details
## Usage
```python
from observability import (
# Metrics
registry,
record_agent_execution,
record_violation,
record_promotion,
MetricsMiddleware,
# Tracing
get_tracer,
get_current_trace_id,
# Logging
get_logger
)
# Create a logger
logger = get_logger("my_agent")
# Get the tracer
tracer = get_tracer()
# Trace an operation
with tracer.trace("agent_execution", agent_id="agent-123") as span:
logger.info("Starting execution", agent_id="agent-123")
try:
# Do work...
with tracer.span("sub_operation") as child:
# Child span automatically linked
pass
record_agent_execution(tier=1, action="update_config", success=True, duration=0.45)
except Exception as e:
span.set_error(e)
record_violation("unauthorized_action", "high")
logger.error("Execution failed", error=str(e))
```
## FastAPI Integration
```python
from fastapi import FastAPI
from observability import metrics_router, tracing_router, logging_router, MetricsMiddleware
app = FastAPI()
# Add metrics middleware
app.add_middleware(MetricsMiddleware)
# Mount routers
app.include_router(metrics_router)
app.include_router(tracing_router)
app.include_router(logging_router)
```
## Configuration
| Setting | Default | Description |
|---------|---------|-------------|
| `DB_PATH` | `/opt/agent-governance/ledger/governance.db` | SQLite database |
| `LOG_LEVEL` | `INFO` | Minimum log level |
| `LOG_RETENTION_DAYS` | `7` | Days to retain logs |
## Status
**Complete**
See [STATUS.md](./STATUS.md) for detailed progress tracking.
## Architecture Reference
Part of the [Agent Governance System](/opt/agent-governance/docs/ARCHITECTURE.md).
Parent: [Project Root](/opt/agent-governance)
---
*Last updated: 2026-01-24 UTC*
observability/STATUS.md
+82
View File
@@ -0,0 +1,82 @@
# Status: Observability
## Current Phase
**COMPLETE**
## Tasks
| Status | Task | Updated |
|--------|------|---------|
| ✅ | Prometheus metrics module (Counter, Gauge, Histogram) | 2026-01-24 |
| ✅ | Distributed tracing with span hierarchy | 2026-01-24 |
| ✅ | Structured JSON logging with trace correlation | 2026-01-24 |
| ✅ | SQLite persistence for logs and traces | 2026-01-24 |
| ✅ | FastAPI routers for metrics, tracing, logging | 2026-01-24 |
| ✅ | HTTP header context propagation (X-Trace-ID, X-Span-ID) | 2026-01-24 |
| ✅ | Multi-tenant support | 2026-01-24 |
| ✅ | MetricsMiddleware for automatic request tracking | 2026-01-24 |
| ✅ | Module exports and unified API | 2026-01-24 |
## Metrics Implemented
- `agent_executions_total` - Counter by tier, action, status
- `agent_execution_duration_seconds` - Histogram
- `agent_violations_total` - Counter by type, severity
- `agent_promotions_total` - Counter by tier transition
- `api_requests_total` - Counter by method, endpoint, status
- `api_request_duration_seconds` - Histogram
- `component_health` - Gauge (Vault, DragonflyDB, Ledger)
- `tenant_quota_usage_ratio` - Gauge
- `governance_uptime_seconds` - Gauge
- `marketplace_template_downloads_total` - Counter
- `orchestration_requests_total` - Counter by model, status
- `orchestration_tokens_total` - Counter by model
## Dependencies
| Dependency | Status | Purpose |
|------------|--------|---------|
| SQLite (ledger) | ✅ Available | Log/trace storage |
| Vault | ✅ Available | Health check target |
| DragonflyDB | ✅ Available | Health check target |
## API Endpoints
| Endpoint | Method | Status |
|----------|--------|--------|
| `/metrics` | GET | ✅ Prometheus format |
| `/traces` | GET | ✅ List with filters |
| `/traces/{trace_id}` | GET | ✅ Full details |
| `/logs` | GET | ✅ Search with filters |
| `/logs/trace/{trace_id}` | GET | ✅ Logs for trace |
| `/logs/stats` | GET | ✅ Statistics |
| `/logs/cleanup` | POST | ✅ Retention cleanup |
| `/health/detailed` | GET | ✅ Component health |
## Issues / Blockers
*No current issues or blockers.*
## Future Enhancements
- Grafana dashboard templates
- Jaeger/Zipkin export integration
- Alert rule engine
- SLO/SLI tracking
- Trace sampling strategies
## Activity Log
### 2026-01-24 UTC
- **Phase**: COMPLETE
- **Action**: Documentation added
- **Details**: Created README.md and STATUS.md for observability module
### 2026-01-24 12:36 UTC
- **Phase**: COMPLETE
- **Action**: Module implementation complete
- **Details**: metrics.py, tracing.py, logging.py implemented with full functionality
---
*Last updated: 2026-01-24 UTC*
observability/__init__.py
+73
View File
@@ -0,0 +1,73 @@
"""
Observability Module
====================
Provides metrics, tracing, and logging for the agent governance system.
Components:
- metrics: Prometheus-format metrics endpoint
- tracing: Distributed tracing with spans
- logging: Structured JSON logging with trace correlation
"""
from .metrics import (
registry,
router as metrics_router,
record_agent_execution,
record_violation,
record_promotion,
record_orchestration,
record_template_download,
MetricsMiddleware
)
from .tracing import (
Tracer,
Span,
Trace,
get_tracer,
get_current_trace_id,
get_current_span,
router as tracing_router,
TRACE_ID_HEADER,
SPAN_ID_HEADER,
extract_trace_context,
inject_trace_context
)
from .logging import (
get_logger,
StructuredLogger,
LogStorage,
router as logging_router
)
__all__ = [
# Metrics
'registry',
'metrics_router',
'record_agent_execution',
'record_violation',
'record_promotion',
'record_orchestration',
'record_template_download',
'MetricsMiddleware',
# Tracing
'Tracer',
'Span',
'Trace',
'get_tracer',
'get_current_trace_id',
'get_current_span',
'tracing_router',
'TRACE_ID_HEADER',
'SPAN_ID_HEADER',
'extract_trace_context',
'inject_trace_context',
# Logging
'get_logger',
'StructuredLogger',
'LogStorage',
'logging_router'
]
observability/logging.py
+412
View File
@@ -0,0 +1,412 @@
#!/usr/bin/env python3
"""
Structured Logging Module
=========================
Provides JSON-structured logging with trace correlation.
Usage:
from observability.logging import get_logger
logger = get_logger("my_module")
logger.info("Operation completed", user_id="123", duration_ms=45.2)
"""
import json
import logging
import sys
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional, Any, Dict
from dataclasses import dataclass
import threading
from .tracing import get_current_trace_id, get_current_span
# =============================================================================
# Configuration
# =============================================================================
DB_PATH = Path("/opt/agent-governance/ledger/governance.db")
LOG_LEVEL = logging.INFO
LOG_TO_DB = True
LOG_RETENTION_DAYS = 7
# =============================================================================
# Structured Log Entry
# =============================================================================
@dataclass
class LogEntry:
"""Structured log entry"""
timestamp: str
level: str
logger: str
message: str
trace_id: Optional[str] = None
span_id: Optional[str] = None
tenant_id: str = "default"
attributes: Dict[str, Any] = None
def to_dict(self) -> dict:
result = {
"timestamp": self.timestamp,
"level": self.level,
"logger": self.logger,
"message": self.message
}
if self.trace_id:
result["trace_id"] = self.trace_id
if self.span_id:
result["span_id"] = self.span_id
if self.tenant_id:
result["tenant_id"] = self.tenant_id
if self.attributes:
result.update(self.attributes)
return result
def to_json(self) -> str:
return json.dumps(self.to_dict())
# =============================================================================
# Log Storage
# =============================================================================
class LogStorage:
"""Stores logs in SQLite for querying"""
_instance = None
_lock = threading.Lock()
def __new__(cls):
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._initialized = False
return cls._instance
def __init__(self):
if self._initialized:
return
self._ensure_table()
self._initialized = True
def _ensure_table(self):
"""Create logs table if not exists"""
try:
conn = sqlite3.connect(DB_PATH)
conn.execute("""
CREATE TABLE IF NOT EXISTS logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL,
level TEXT NOT NULL,
logger TEXT NOT NULL,
message TEXT NOT NULL,
trace_id TEXT,
span_id TEXT,
tenant_id TEXT DEFAULT 'default',
attributes TEXT, -- JSON
created_at TEXT DEFAULT CURRENT_TIMESTAMP
)
""")
conn.execute("CREATE INDEX IF NOT EXISTS idx_logs_timestamp ON logs(timestamp)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_logs_level ON logs(level)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_logs_trace ON logs(trace_id)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_logs_tenant ON logs(tenant_id)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_logs_logger ON logs(logger)")
conn.commit()
conn.close()
except Exception as e:
print(f"Error creating logs table: {e}", file=sys.stderr)
def save(self, entry: LogEntry):
"""Save log entry to database"""
if not LOG_TO_DB:
return
try:
conn = sqlite3.connect(DB_PATH)
conn.execute("""
INSERT INTO logs (timestamp, level, logger, message, trace_id, span_id, tenant_id, attributes)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", (
entry.timestamp, entry.level, entry.logger, entry.message,
entry.trace_id, entry.span_id, entry.tenant_id,
json.dumps(entry.attributes) if entry.attributes else None
))
conn.commit()
conn.close()
except Exception as e:
print(f"Error saving log: {e}", file=sys.stderr)
def search(
self,
level: str = None,
logger: str = None,
trace_id: str = None,
tenant_id: str = None,
message_contains: str = None,
start_time: str = None,
end_time: str = None,
limit: int = 100
) -> list:
"""Search logs"""
try:
conn = sqlite3.connect(DB_PATH)
conn.row_factory = sqlite3.Row
query = "SELECT * FROM logs WHERE 1=1"
params = []
if level:
query += " AND level = ?"
params.append(level)
if logger:
query += " AND logger = ?"
params.append(logger)
if trace_id:
query += " AND trace_id = ?"
params.append(trace_id)
if tenant_id:
query += " AND tenant_id = ?"
params.append(tenant_id)
if message_contains:
query += " AND message LIKE ?"
params.append(f"%{message_contains}%")
if start_time:
query += " AND timestamp >= ?"
params.append(start_time)
if end_time:
query += " AND timestamp <= ?"
params.append(end_time)
query += " ORDER BY timestamp DESC LIMIT ?"
params.append(limit)
cursor = conn.execute(query, params)
results = []
for row in cursor.fetchall():
entry = dict(row)
entry["attributes"] = json.loads(entry["attributes"]) if entry.get("attributes") else {}
results.append(entry)
conn.close()
return results
except Exception as e:
print(f"Error searching logs: {e}", file=sys.stderr)
return []
def cleanup(self, days: int = LOG_RETENTION_DAYS):
"""Remove old logs"""
try:
conn = sqlite3.connect(DB_PATH)
conn.execute(f"""
DELETE FROM logs WHERE timestamp < datetime('now', '-{days} days')
""")
conn.commit()
deleted = conn.total_changes
conn.close()
return deleted
except Exception as e:
print(f"Error cleaning up logs: {e}", file=sys.stderr)
return 0
# =============================================================================
# JSON Formatter
# =============================================================================
class JSONFormatter(logging.Formatter):
"""Format log records as JSON"""
def format(self, record: logging.LogRecord) -> str:
# Get trace context
trace_id = get_current_trace_id()
span = get_current_span()
span_id = span.span_id if span else None
# Build entry
entry = LogEntry(
timestamp=datetime.now(timezone.utc).isoformat(),
level=record.levelname,
logger=record.name,
message=record.getMessage(),
trace_id=trace_id,
span_id=span_id,
tenant_id=getattr(record, 'tenant_id', 'default'),
attributes=getattr(record, 'extra_attrs', None)
)
# Store in DB
storage = LogStorage()
storage.save(entry)
return entry.to_json()
# =============================================================================
# Custom Logger
# =============================================================================
class StructuredLogger:
"""Logger with structured output and trace correlation"""
def __init__(self, name: str, tenant_id: str = "default"):
self.name = name
self.tenant_id = tenant_id
self._logger = logging.getLogger(name)
self._logger.setLevel(LOG_LEVEL)
# Add JSON handler if not already present
if not any(isinstance(h, logging.StreamHandler) and isinstance(h.formatter, JSONFormatter)
for h in self._logger.handlers):
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JSONFormatter())
self._logger.addHandler(handler)
def _log(self, level: int, message: str, **kwargs):
"""Internal log method"""
extra = {
'tenant_id': kwargs.pop('tenant_id', self.tenant_id),
'extra_attrs': kwargs if kwargs else None
}
self._logger.log(level, message, extra=extra)
def debug(self, message: str, **kwargs):
"""Log debug message"""
self._log(logging.DEBUG, message, **kwargs)
def info(self, message: str, **kwargs):
"""Log info message"""
self._log(logging.INFO, message, **kwargs)
def warning(self, message: str, **kwargs):
"""Log warning message"""
self._log(logging.WARNING, message, **kwargs)
def error(self, message: str, **kwargs):
"""Log error message"""
self._log(logging.ERROR, message, **kwargs)
def critical(self, message: str, **kwargs):
"""Log critical message"""
self._log(logging.CRITICAL, message, **kwargs)
def exception(self, message: str, **kwargs):
"""Log exception with traceback"""
import traceback
kwargs['traceback'] = traceback.format_exc()
self._log(logging.ERROR, message, **kwargs)
# =============================================================================
# Logger Factory
# =============================================================================
_loggers: Dict[str, StructuredLogger] = {}
def get_logger(name: str, tenant_id: str = "default") -> StructuredLogger:
"""Get or create a structured logger"""
key = f"{name}:{tenant_id}"
if key not in _loggers:
_loggers[key] = StructuredLogger(name, tenant_id)
return _loggers[key]
# =============================================================================
# FastAPI Router
# =============================================================================
from fastapi import APIRouter, Query, HTTPException
router = APIRouter(prefix="/logs", tags=["Logging"])
@router.get("")
async def search_logs(
level: Optional[str] = None,
logger: Optional[str] = None,
trace_id: Optional[str] = None,
message: Optional[str] = None,
start_time: Optional[str] = None,
end_time: Optional[str] = None,
limit: int = Query(100, le=500)
):
"""Search logs"""
storage = LogStorage()
logs = storage.search(
level=level,
logger=logger,
trace_id=trace_id,
message_contains=message,
start_time=start_time,
end_time=end_time,
limit=limit
)
return {"logs": logs, "count": len(logs)}
@router.get("/trace/{trace_id}")
async def get_logs_by_trace(trace_id: str, limit: int = Query(100, le=500)):
"""Get all logs for a trace"""
storage = LogStorage()
logs = storage.search(trace_id=trace_id, limit=limit)
return {"logs": logs, "count": len(logs), "trace_id": trace_id}
@router.get("/stats")
async def get_log_stats():
"""Get log statistics"""
try:
conn = sqlite3.connect(DB_PATH)
conn.row_factory = sqlite3.Row
stats = {}
# Count by level
cursor = conn.execute("""
SELECT level, COUNT(*) as count
FROM logs
WHERE timestamp > datetime('now', '-1 hour')
GROUP BY level
""")
stats["last_hour_by_level"] = {row["level"]: row["count"] for row in cursor.fetchall()}
# Count by logger
cursor = conn.execute("""
SELECT logger, COUNT(*) as count
FROM logs
WHERE timestamp > datetime('now', '-1 hour')
GROUP BY logger
ORDER BY count DESC
LIMIT 10
""")
stats["last_hour_by_logger"] = {row["logger"]: row["count"] for row in cursor.fetchall()}
# Total count
cursor = conn.execute("SELECT COUNT(*) as total FROM logs")
stats["total_logs"] = cursor.fetchone()["total"]
# Error count last 24h
cursor = conn.execute("""
SELECT COUNT(*) as count FROM logs
WHERE level IN ('ERROR', 'CRITICAL')
AND timestamp > datetime('now', '-1 day')
""")
stats["errors_last_24h"] = cursor.fetchone()["count"]
conn.close()
return stats
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@router.post("/cleanup")
async def cleanup_logs(days: int = Query(7, ge=1, le=365)):
"""Clean up old logs"""
storage = LogStorage()
deleted = storage.cleanup(days)
return {"deleted": deleted, "retention_days": days}
observability/metrics.py
+584
View File
@@ -0,0 +1,584 @@
#!/usr/bin/env python3
"""
Prometheus Metrics Module
=========================
Exposes metrics in Prometheus format for monitoring agent governance system.
Metrics:
- agent_executions_total - Total agent executions by tier and status
- agent_execution_duration_seconds - Execution duration histogram
- agent_violations_total - Violations by type and severity
- agent_promotions_total - Promotions by tier transition
- api_requests_total - API requests by endpoint and status
- tenant_quota_usage - Quota usage by tenant and resource
- system_health - Component health status
"""
import sqlite3
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional, Dict, List
from dataclasses import dataclass, field
from contextlib import contextmanager
import threading
# =============================================================================
# Configuration
# =============================================================================
DB_PATH = Path("/opt/agent-governance/ledger/governance.db")
# =============================================================================
# Metric Types
# =============================================================================
@dataclass
class Counter:
"""Prometheus counter metric"""
name: str
help: str
labels: List[str] = field(default_factory=list)
values: Dict[tuple, float] = field(default_factory=dict)
_lock: threading.Lock = field(default_factory=threading.Lock)
def inc(self, labels: dict = None, value: float = 1):
"""Increment counter"""
key = tuple(labels.values()) if labels else ()
with self._lock:
self.values[key] = self.values.get(key, 0) + value
def to_prometheus(self) -> str:
"""Format as Prometheus text"""
lines = [f"# HELP {self.name} {self.help}", f"# TYPE {self.name} counter"]
for key, value in self.values.items():
if self.labels and key:
label_str = ",".join(f'{l}="{v}"' for l, v in zip(self.labels, key))
lines.append(f"{self.name}{{{label_str}}} {value}")
else:
lines.append(f"{self.name} {value}")
return "\n".join(lines)
@dataclass
class Gauge:
"""Prometheus gauge metric"""
name: str
help: str
labels: List[str] = field(default_factory=list)
values: Dict[tuple, float] = field(default_factory=dict)
_lock: threading.Lock = field(default_factory=threading.Lock)
def set(self, value: float, labels: dict = None):
"""Set gauge value"""
key = tuple(labels.values()) if labels else ()
with self._lock:
self.values[key] = value
def inc(self, labels: dict = None, value: float = 1):
"""Increment gauge"""
key = tuple(labels.values()) if labels else ()
with self._lock:
self.values[key] = self.values.get(key, 0) + value
def dec(self, labels: dict = None, value: float = 1):
"""Decrement gauge"""
key = tuple(labels.values()) if labels else ()
with self._lock:
self.values[key] = self.values.get(key, 0) - value
def to_prometheus(self) -> str:
"""Format as Prometheus text"""
lines = [f"# HELP {self.name} {self.help}", f"# TYPE {self.name} gauge"]
for key, value in self.values.items():
if self.labels and key:
label_str = ",".join(f'{l}="{v}"' for l, v in zip(self.labels, key))
lines.append(f"{self.name}{{{label_str}}} {value}")
else:
lines.append(f"{self.name} {value}")
return "\n".join(lines)
@dataclass
class Histogram:
"""Prometheus histogram metric"""
name: str
help: str
labels: List[str] = field(default_factory=list)
buckets: List[float] = field(default_factory=lambda: [0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60])
observations: Dict[tuple, List[float]] = field(default_factory=dict)
_lock: threading.Lock = field(default_factory=threading.Lock)
def observe(self, value: float, labels: dict = None):
"""Record observation"""
key = tuple(labels.values()) if labels else ()
with self._lock:
if key not in self.observations:
self.observations[key] = []
self.observations[key].append(value)
def to_prometheus(self) -> str:
"""Format as Prometheus text"""
lines = [f"# HELP {self.name} {self.help}", f"# TYPE {self.name} histogram"]
for key, values in self.observations.items():
label_str = ",".join(f'{l}="{v}"' for l, v in zip(self.labels, key)) if self.labels and key else ""
prefix = f"{self.name}{{{label_str}," if label_str else f"{self.name}{{le="
# Bucket counts
for bucket in self.buckets:
count = sum(1 for v in values if v <= bucket)
if label_str:
lines.append(f'{self.name}_bucket{{{label_str},le="{bucket}"}} {count}')
else:
lines.append(f'{self.name}_bucket{{le="{bucket}"}} {count}')
# +Inf bucket
if label_str:
lines.append(f'{self.name}_bucket{{{label_str},le="+Inf"}} {len(values)}')
else:
lines.append(f'{self.name}_bucket{{le="+Inf"}} {len(values)}')
# Sum and count
total = sum(values)
if label_str:
lines.append(f"{self.name}_sum{{{label_str}}} {total}")
lines.append(f"{self.name}_count{{{label_str}}} {len(values)}")
else:
lines.append(f"{self.name}_sum {total}")
lines.append(f"{self.name}_count {len(values)}")
return "\n".join(lines)
# =============================================================================
# Metrics Registry
# =============================================================================
class MetricsRegistry:
"""Central registry for all metrics"""
def __init__(self):
# Agent execution metrics
self.agent_executions_total = Counter(
name="agent_executions_total",
help="Total number of agent executions",
labels=["tier", "action", "status"]
)
self.agent_execution_duration = Histogram(
name="agent_execution_duration_seconds",
help="Agent execution duration in seconds",
labels=["tier", "action"],
buckets=[0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300]
)
self.agent_active = Gauge(
name="agent_active_count",
help="Number of currently active agents",
labels=["tier"]
)
# Violation metrics
self.violations_total = Counter(
name="agent_violations_total",
help="Total number of violations",
labels=["type", "severity"]
)
# Promotion metrics
self.promotions_total = Counter(
name="agent_promotions_total",
help="Total number of promotions",
labels=["from_tier", "to_tier"]
)
# API metrics
self.api_requests_total = Counter(
name="api_requests_total",
help="Total API requests",
labels=["method", "endpoint", "status"]
)
self.api_request_duration = Histogram(
name="api_request_duration_seconds",
help="API request duration in seconds",
labels=["method", "endpoint"],
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
)
# Tenant/quota metrics
self.tenant_quota_usage = Gauge(
name="tenant_quota_usage_ratio",
help="Tenant quota usage as ratio (0-1)",
labels=["tenant_id", "resource"]
)
self.tenant_api_calls = Counter(
name="tenant_api_calls_total",
help="API calls by tenant",
labels=["tenant_id"]
)
# System health metrics
self.component_health = Gauge(
name="component_health",
help="Component health status (1=healthy, 0=unhealthy)",
labels=["component"]
)
self.db_connections = Gauge(
name="db_connections_active",
help="Active database connections"
)
# Marketplace metrics
self.template_downloads = Counter(
name="marketplace_template_downloads_total",
help="Template downloads",
labels=["template_id", "category"]
)
self.template_installs = Gauge(
name="marketplace_template_installs_active",
help="Active template installations",
labels=["category"]
)
# Orchestration metrics
self.orchestration_requests = Counter(
name="orchestration_requests_total",
help="Orchestration requests by model",
labels=["model", "status"]
)
self.orchestration_tokens = Counter(
name="orchestration_tokens_total",
help="Tokens used in orchestration",
labels=["model"]
)
# Uptime
self._start_time = time.time()
self.uptime = Gauge(
name="governance_uptime_seconds",
help="System uptime in seconds"
)
def collect_from_db(self):
"""Collect metrics from database"""
try:
conn = sqlite3.connect(DB_PATH)
conn.row_factory = sqlite3.Row
# Agent counts by tier
cursor = conn.execute("""
SELECT current_tier, COUNT(*) as count
FROM agent_metrics GROUP BY current_tier
""")
for row in cursor.fetchall():
self.agent_active.set(row["count"], {"tier": str(row["current_tier"])})
# Recent violations by type/severity
cursor = conn.execute("""
SELECT violation_type, severity, COUNT(*) as count
FROM violations
WHERE timestamp > datetime('now', '-1 hour')
GROUP BY violation_type, severity
""")
for row in cursor.fetchall():
self.violations_total.values[(row["violation_type"], row["severity"])] = row["count"]
# Promotions
cursor = conn.execute("""
SELECT from_tier, to_tier, COUNT(*) as count
FROM promotions GROUP BY from_tier, to_tier
""")
for row in cursor.fetchall():
self.promotions_total.values[(str(row["from_tier"]), str(row["to_tier"]))] = row["count"]
# Tenant usage
cursor = conn.execute("""
SELECT tu.tenant_id, tu.api_calls, tu.tokens_used,
tq.max_api_calls_per_day, tq.max_tokens_per_day
FROM tenant_usage tu
JOIN tenant_quotas tq ON tu.tenant_id = tq.tenant_id
WHERE tu.period_start = date('now')
""")
for row in cursor.fetchall():
if row["max_api_calls_per_day"] > 0:
ratio = row["api_calls"] / row["max_api_calls_per_day"]
self.tenant_quota_usage.set(ratio, {"tenant_id": row["tenant_id"], "resource": "api_calls"})
if row["max_tokens_per_day"] > 0:
ratio = row["tokens_used"] / row["max_tokens_per_day"]
self.tenant_quota_usage.set(ratio, {"tenant_id": row["tenant_id"], "resource": "tokens"})
# Marketplace stats
cursor = conn.execute("""
SELECT category, SUM(active_installs) as installs
FROM template_stats ts
JOIN agent_templates t ON ts.template_id = t.template_id
GROUP BY category
""")
for row in cursor.fetchall():
self.template_installs.set(row["installs"] or 0, {"category": row["category"]})
conn.close()
except Exception as e:
print(f"Error collecting DB metrics: {e}")
def check_health(self):
"""Check component health"""
import subprocess
# Database
try:
conn = sqlite3.connect(DB_PATH)
conn.execute("SELECT 1")
conn.close()
self.component_health.set(1, {"component": "database"})
except:
self.component_health.set(0, {"component": "database"})
# Vault
try:
result = subprocess.run(
["docker", "exec", "vault", "vault", "status", "-format=json"],
capture_output=True, timeout=5
)
self.component_health.set(1 if result.returncode == 0 else 0, {"component": "vault"})
except:
self.component_health.set(0, {"component": "vault"})
# DragonflyDB
try:
import redis
r = redis.Redis(host='127.0.0.1', port=6379, password='governance2026')
r.ping()
self.component_health.set(1, {"component": "dragonfly"})
except:
self.component_health.set(0, {"component": "dragonfly"})
# Uptime
self.uptime.set(time.time() - self._start_time)
def to_prometheus(self) -> str:
"""Export all metrics in Prometheus format"""
self.collect_from_db()
self.check_health()
metrics = [
self.agent_executions_total,
self.agent_execution_duration,
self.agent_active,
self.violations_total,
self.promotions_total,
self.api_requests_total,
self.api_request_duration,
self.tenant_quota_usage,
self.tenant_api_calls,
self.component_health,
self.db_connections,
self.template_downloads,
self.template_installs,
self.orchestration_requests,
self.orchestration_tokens,
self.uptime,
]
output = []
for metric in metrics:
if metric.values or isinstance(metric, Histogram) and metric.observations:
output.append(metric.to_prometheus())
elif not metric.values and not (isinstance(metric, Histogram) and metric.observations):
# Include empty metrics with help text
output.append(f"# HELP {metric.name} {metric.help}")
output.append(f"# TYPE {metric.name} {type(metric).__name__.lower()}")
return "\n\n".join(output)
# =============================================================================
# Global Registry
# =============================================================================
registry = MetricsRegistry()
# =============================================================================
# FastAPI Router
# =============================================================================
from fastapi import APIRouter, Response
router = APIRouter(tags=["Observability"])
@router.get("/metrics")
async def get_metrics():
"""Prometheus metrics endpoint"""
return Response(
content=registry.to_prometheus(),
media_type="text/plain; version=0.0.4; charset=utf-8"
)
@router.get("/health/detailed")
async def detailed_health():
"""Detailed health check with all components"""
import subprocess
health = {
"status": "healthy",
"timestamp": datetime.now(timezone.utc).isoformat(),
"uptime_seconds": time.time() - registry._start_time,
"components": {}
}
# Database
try:
conn = sqlite3.connect(DB_PATH)
cursor = conn.execute("SELECT COUNT(*) FROM agent_metrics")
count = cursor.fetchone()[0]
conn.close()
health["components"]["database"] = {"status": "healthy", "agent_count": count}
except Exception as e:
health["components"]["database"] = {"status": "unhealthy", "error": str(e)}
health["status"] = "degraded"
# Vault
try:
result = subprocess.run(
["docker", "exec", "vault", "vault", "status", "-format=json"],
capture_output=True, text=True, timeout=5
)
if result.returncode == 0:
import json
vault_status = json.loads(result.stdout)
health["components"]["vault"] = {
"status": "healthy",
"sealed": vault_status.get("sealed", True),
"version": vault_status.get("version", "unknown")
}
else:
health["components"]["vault"] = {"status": "unhealthy", "error": "non-zero exit"}
health["status"] = "degraded"
except Exception as e:
health["components"]["vault"] = {"status": "unhealthy", "error": str(e)}
health["status"] = "degraded"
# DragonflyDB
try:
import redis
r = redis.Redis(host='127.0.0.1', port=6379, password='governance2026')
info = r.info()
health["components"]["dragonfly"] = {
"status": "healthy",
"connected_clients": info.get("connected_clients", 0),
"used_memory_human": info.get("used_memory_human", "unknown")
}
except Exception as e:
health["components"]["dragonfly"] = {"status": "unhealthy", "error": str(e)}
health["status"] = "degraded"
return health
# =============================================================================
# Middleware for request metrics
# =============================================================================
class MetricsMiddleware:
"""Middleware to collect API request metrics"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] != "http":
await self.app(scope, receive, send)
return
start_time = time.time()
path = scope.get("path", "unknown")
method = scope.get("method", "unknown")
# Normalize path (remove IDs)
import re
normalized_path = re.sub(r'/[a-f0-9-]{8,}', '/{id}', path)
normalized_path = re.sub(r'/\d+', '/{id}', normalized_path)
status_code = 500
async def send_wrapper(message):
nonlocal status_code
if message["type"] == "http.response.start":
status_code = message["status"]
await send(message)
try:
await self.app(scope, receive, send_wrapper)
finally:
duration = time.time() - start_time
# Record metrics
registry.api_requests_total.inc({
"method": method,
"endpoint": normalized_path,
"status": str(status_code)
})
registry.api_request_duration.observe(duration, {
"method": method,
"endpoint": normalized_path
})
# =============================================================================
# Helper functions for use in other modules
# =============================================================================
def record_agent_execution(tier: int, action: str, success: bool, duration: float):
"""Record agent execution metrics"""
registry.agent_executions_total.inc({
"tier": str(tier),
"action": action,
"status": "success" if success else "error"
})
registry.agent_execution_duration.observe(duration, {
"tier": str(tier),
"action": action
})
def record_violation(violation_type: str, severity: str):
"""Record violation metric"""
registry.violations_total.inc({
"type": violation_type,
"severity": severity
})
def record_promotion(from_tier: int, to_tier: int):
"""Record promotion metric"""
registry.promotions_total.inc({
"from_tier": str(from_tier),
"to_tier": str(to_tier)
})
def record_orchestration(model: str, success: bool, tokens: int = 0):
"""Record orchestration metrics"""
registry.orchestration_requests.inc({
"model": model,
"status": "success" if success else "error"
})
if tokens > 0:
registry.orchestration_tokens.inc({"model": model}, tokens)
def record_template_download(template_id: str, category: str):
"""Record template download"""
registry.template_downloads.inc({
"template_id": template_id,
"category": category
})
observability/tracing.py
+490
View File
@@ -0,0 +1,490 @@
#!/usr/bin/env python3
"""
Distributed Tracing Module
==========================
Provides trace ID propagation and span tracking for agent operations.
Usage:
from observability.tracing import Tracer, get_current_trace
tracer = Tracer()
with tracer.span("agent_execution", agent_id="agent-001") as span:
# Do work
span.set_attribute("result", "success")
with tracer.span("sub_operation") as child:
# Child span automatically linked to parent
pass
"""
import json
import secrets
import sqlite3
import threading
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional, Dict, List, Any
# =============================================================================
# Configuration
# =============================================================================
DB_PATH = Path("/opt/agent-governance/ledger/governance.db")
# Thread-local storage for current trace context
_trace_context = threading.local()
# =============================================================================
# Data Classes
# =============================================================================
@dataclass
class Span:
"""A single span in a trace"""
trace_id: str
span_id: str
parent_span_id: Optional[str]
operation_name: str
service_name: str = "agent-governance"
start_time: float = field(default_factory=time.time)
end_time: Optional[float] = None
duration_ms: Optional[float] = None
status: str = "ok" # ok, error
attributes: Dict[str, Any] = field(default_factory=dict)
events: List[Dict[str, Any]] = field(default_factory=list)
def set_attribute(self, key: str, value: Any):
"""Set span attribute"""
self.attributes[key] = value
def add_event(self, name: str, attributes: Dict[str, Any] = None):
"""Add an event to the span"""
self.events.append({
"name": name,
"timestamp": time.time(),
"attributes": attributes or {}
})
def set_error(self, error: Exception):
"""Mark span as error with exception details"""
self.status = "error"
self.attributes["error.type"] = type(error).__name__
self.attributes["error.message"] = str(error)
def finish(self):
"""Finish the span"""
self.end_time = time.time()
self.duration_ms = (self.end_time - self.start_time) * 1000
def to_dict(self) -> dict:
"""Convert to dictionary"""
return {
"trace_id": self.trace_id,
"span_id": self.span_id,
"parent_span_id": self.parent_span_id,
"operation_name": self.operation_name,
"service_name": self.service_name,
"start_time": datetime.fromtimestamp(self.start_time, tz=timezone.utc).isoformat(),
"end_time": datetime.fromtimestamp(self.end_time, tz=timezone.utc).isoformat() if self.end_time else None,
"duration_ms": self.duration_ms,
"status": self.status,
"attributes": self.attributes,
"events": self.events
}
@dataclass
class Trace:
"""A complete trace consisting of multiple spans"""
trace_id: str
root_span_id: str
spans: List[Span] = field(default_factory=list)
start_time: float = field(default_factory=time.time)
end_time: Optional[float] = None
def add_span(self, span: Span):
"""Add a span to the trace"""
self.spans.append(span)
def finish(self):
"""Finish the trace"""
self.end_time = time.time()
def to_dict(self) -> dict:
"""Convert to dictionary"""
return {
"trace_id": self.trace_id,
"root_span_id": self.root_span_id,
"span_count": len(self.spans),
"start_time": datetime.fromtimestamp(self.start_time, tz=timezone.utc).isoformat(),
"end_time": datetime.fromtimestamp(self.end_time, tz=timezone.utc).isoformat() if self.end_time else None,
"duration_ms": (self.end_time - self.start_time) * 1000 if self.end_time else None,
"spans": [s.to_dict() for s in self.spans]
}
# =============================================================================
# Trace Storage
# =============================================================================
class TraceStorage:
"""Stores traces in SQLite and Redis for querying"""
def __init__(self):
self._ensure_table()
def _ensure_table(self):
"""Create traces table if not exists"""
try:
conn = sqlite3.connect(DB_PATH)
conn.execute("""
CREATE TABLE IF NOT EXISTS traces (
trace_id TEXT PRIMARY KEY,
root_operation TEXT,
service_name TEXT,
start_time TEXT,
end_time TEXT,
duration_ms REAL,
span_count INTEGER,
status TEXT,
spans TEXT, -- JSON
tenant_id TEXT DEFAULT 'default',
created_at TEXT DEFAULT CURRENT_TIMESTAMP
)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_traces_time ON traces(start_time)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_traces_tenant ON traces(tenant_id)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS idx_traces_operation ON traces(root_operation)
""")
conn.commit()
conn.close()
except Exception as e:
print(f"Error creating traces table: {e}")
def save(self, trace: Trace, tenant_id: str = "default"):
"""Save trace to storage"""
try:
conn = sqlite3.connect(DB_PATH)
# Find root span
root_span = next((s for s in trace.spans if s.span_id == trace.root_span_id), None)
root_operation = root_span.operation_name if root_span else "unknown"
status = "error" if any(s.status == "error" for s in trace.spans) else "ok"
conn.execute("""
INSERT OR REPLACE INTO traces
(trace_id, root_operation, service_name, start_time, end_time,
duration_ms, span_count, status, spans, tenant_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
trace.trace_id,
root_operation,
"agent-governance",
datetime.fromtimestamp(trace.start_time, tz=timezone.utc).isoformat(),
datetime.fromtimestamp(trace.end_time, tz=timezone.utc).isoformat() if trace.end_time else None,
(trace.end_time - trace.start_time) * 1000 if trace.end_time else None,
len(trace.spans),
status,
json.dumps([s.to_dict() for s in trace.spans]),
tenant_id
))
conn.commit()
conn.close()
except Exception as e:
print(f"Error saving trace: {e}")
def get(self, trace_id: str) -> Optional[dict]:
"""Get trace by ID"""
try:
conn = sqlite3.connect(DB_PATH)
conn.row_factory = sqlite3.Row
cursor = conn.execute("""
SELECT * FROM traces WHERE trace_id = ?
""", (trace_id,))
row = cursor.fetchone()
conn.close()
if row:
result = dict(row)
result["spans"] = json.loads(result["spans"]) if result.get("spans") else []
return result
return None
except Exception as e:
print(f"Error getting trace: {e}")
return None
def search(
self,
operation: str = None,
status: str = None,
min_duration_ms: float = None,
tenant_id: str = None,
start_after: str = None,
limit: int = 100
) -> List[dict]:
"""Search traces"""
try:
conn = sqlite3.connect(DB_PATH)
conn.row_factory = sqlite3.Row
query = "SELECT trace_id, root_operation, start_time, duration_ms, span_count, status FROM traces WHERE 1=1"
params = []
if operation:
query += " AND root_operation LIKE ?"
params.append(f"%{operation}%")
if status:
query += " AND status = ?"
params.append(status)
if min_duration_ms:
query += " AND duration_ms >= ?"
params.append(min_duration_ms)
if tenant_id:
query += " AND tenant_id = ?"
params.append(tenant_id)
if start_after:
query += " AND start_time > ?"
params.append(start_after)
query += " ORDER BY start_time DESC LIMIT ?"
params.append(limit)
cursor = conn.execute(query, params)
results = [dict(row) for row in cursor.fetchall()]
conn.close()
return results
except Exception as e:
print(f"Error searching traces: {e}")
return []
# =============================================================================
# Tracer
# =============================================================================
class Tracer:
"""Main tracer class for creating and managing traces"""
def __init__(self, service_name: str = "agent-governance", tenant_id: str = "default"):
self.service_name = service_name
self.tenant_id = tenant_id
self.storage = TraceStorage()
self._current_trace: Optional[Trace] = None
self._span_stack: List[Span] = []
def _generate_id(self, length: int = 16) -> str:
"""Generate random ID"""
return secrets.token_hex(length)
def start_trace(self, operation_name: str, **attributes) -> Trace:
"""Start a new trace"""
trace_id = self._generate_id(16)
span_id = self._generate_id(8)
root_span = Span(
trace_id=trace_id,
span_id=span_id,
parent_span_id=None,
operation_name=operation_name,
service_name=self.service_name,
attributes=attributes
)
trace = Trace(trace_id=trace_id, root_span_id=span_id)
trace.add_span(root_span)
self._current_trace = trace
self._span_stack = [root_span]
# Store in thread-local
_trace_context.trace = trace
_trace_context.current_span = root_span
return trace
def start_span(self, operation_name: str, **attributes) -> Span:
"""Start a new span within current trace"""
if not self._current_trace:
# Auto-start trace if none exists
self.start_trace(operation_name, **attributes)
return self._span_stack[-1]
parent_span = self._span_stack[-1] if self._span_stack else None
span = Span(
trace_id=self._current_trace.trace_id,
span_id=self._generate_id(8),
parent_span_id=parent_span.span_id if parent_span else None,
operation_name=operation_name,
service_name=self.service_name,
attributes=attributes
)
self._current_trace.add_span(span)
self._span_stack.append(span)
_trace_context.current_span = span
return span
def end_span(self, span: Span = None):
"""End a span"""
if span is None and self._span_stack:
span = self._span_stack[-1]
if span:
span.finish()
if self._span_stack and self._span_stack[-1] == span:
self._span_stack.pop()
# Update current span in context
if self._span_stack:
_trace_context.current_span = self._span_stack[-1]
def end_trace(self):
"""End the current trace and save it"""
if self._current_trace:
# End any remaining spans
while self._span_stack:
self._span_stack[-1].finish()
self._span_stack.pop()
self._current_trace.finish()
self.storage.save(self._current_trace, self.tenant_id)
self._current_trace = None
_trace_context.trace = None
_trace_context.current_span = None
@contextmanager
def span(self, operation_name: str, **attributes):
"""Context manager for spans"""
span = self.start_span(operation_name, **attributes)
try:
yield span
except Exception as e:
span.set_error(e)
raise
finally:
self.end_span(span)
@contextmanager
def trace(self, operation_name: str, **attributes):
"""Context manager for complete traces"""
self.start_trace(operation_name, **attributes)
root_span = self._span_stack[0] if self._span_stack else None
try:
yield root_span
except Exception as e:
if root_span:
root_span.set_error(e)
raise
finally:
self.end_trace()
def get_current_span(self) -> Optional[Span]:
"""Get current span"""
return getattr(_trace_context, 'current_span', None)
def get_current_trace_id(self) -> Optional[str]:
"""Get current trace ID"""
trace = getattr(_trace_context, 'trace', None)
return trace.trace_id if trace else None
# =============================================================================
# Global Tracer
# =============================================================================
_default_tracer: Optional[Tracer] = None
def get_tracer(service_name: str = "agent-governance", tenant_id: str = "default") -> Tracer:
"""Get or create default tracer"""
global _default_tracer
if _default_tracer is None:
_default_tracer = Tracer(service_name, tenant_id)
return _default_tracer
def get_current_trace_id() -> Optional[str]:
"""Get current trace ID from context"""
trace = getattr(_trace_context, 'trace', None)
return trace.trace_id if trace else None
def get_current_span() -> Optional[Span]:
"""Get current span from context"""
return getattr(_trace_context, 'current_span', None)
# =============================================================================
# FastAPI Router
# =============================================================================
from fastapi import APIRouter, Query, HTTPException
router = APIRouter(prefix="/traces", tags=["Tracing"])
@router.get("")
async def list_traces(
operation: Optional[str] = None,
status: Optional[str] = None,
min_duration_ms: Optional[float] = None,
limit: int = Query(50, le=200)
):
"""List recent traces"""
storage = TraceStorage()
traces = storage.search(
operation=operation,
status=status,
min_duration_ms=min_duration_ms,
limit=limit
)
return {"traces": traces, "count": len(traces)}
@router.get("/{trace_id}")
async def get_trace(trace_id: str):
"""Get trace details"""
storage = TraceStorage()
trace = storage.get(trace_id)
if not trace:
raise HTTPException(status_code=404, detail="Trace not found")
return trace
# =============================================================================
# Trace ID Propagation Headers
# =============================================================================
TRACE_ID_HEADER = "X-Trace-ID"
SPAN_ID_HEADER = "X-Span-ID"
PARENT_SPAN_HEADER = "X-Parent-Span-ID"
def extract_trace_context(headers: dict) -> dict:
"""Extract trace context from HTTP headers"""
return {
"trace_id": headers.get(TRACE_ID_HEADER),
"span_id": headers.get(SPAN_ID_HEADER),
"parent_span_id": headers.get(PARENT_SPAN_HEADER)
}
def inject_trace_context(headers: dict, span: Span):
"""Inject trace context into HTTP headers"""
headers[TRACE_ID_HEADER] = span.trace_id
headers[SPAN_ID_HEADER] = span.span_id
if span.parent_span_id:
headers[PARENT_SPAN_HEADER] = span.parent_span_id