Initial commit: Agent Governance System Phase 8

Phase 8 Production Hardening with complete governance infrastructure:

- Vault integration with tiered policies (T0-T4)
- DragonflyDB state management
- SQLite audit ledger
- Pipeline DSL and templates
- Promotion/revocation engine
- Checkpoint system for session persistence
- Health manager and circuit breaker for fault tolerance
- GitHub/Slack integrations
- Architectural test pipeline with bug watcher, suggestion engine, council review
- Multi-agent chaos testing framework

Test Results:
- Governance tests: 68/68 passing
- E2E workflow: 16/16 passing
- Phase 2 Vault: 14/14 passing
- Integration tests: 27/27 passing

Coverage: 57.6% average across 12 phases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
profit
2026-01-23 22:07:06 -05:00
co-authored by Claude Opus 4.5
commit 77655c298c
338 changed files with 63491 additions and 0 deletions
+34
View File
@@ -0,0 +1,34 @@
# Bin
> CLI tools and executable scripts
## Overview
This directory contains cli tools and executable scripts.
## Key Files
| File | Description |
|------|-------------|
| `status.py` | Python module |
| `register-agent.sh` | Shell script |
| `sample-tier0-agent.sh` | Shell script |
## Interfaces / APIs
*Document any APIs, CLI commands, or interfaces here.*
## Status
** Not Started**
See [STATUS.md](./STATUS.md) for detailed progress tracking.
## Architecture Reference
Part of the [Agent Governance System](/opt/agent-governance/docs/ARCHITECTURE.md).
Parent: [Project Root](/opt/agent-governance)
---
*Last updated: 2026-01-23 23:25:09 UTC*
+35
View File
@@ -0,0 +1,35 @@
# Status: Bin
## Current Phase
** NOT STARTED**
## Tasks
| Status | Task | Updated |
|--------|------|---------|
| ☐ | *No tasks defined* | - |
## Dependencies
*No external dependencies.*
## Issues / Blockers
*No current issues or blockers.*
## Activity Log
### 2026-01-23 23:25:31 UTC
- **Phase**: COMPLETE
- **Action**: CLI tools: checkpoint, status commands implemented
- **Details**: Phase updated to complete
### 2026-01-23 23:25:09 UTC
- **Phase**: NOT STARTED
- **Action**: Initialized
- **Details**: Status tracking initialized for this directory.
---
*Last updated: 2026-01-23 23:25:31 UTC*
Executable
+84
View File
@@ -0,0 +1,84 @@
#!/bin/bash
#
# Context Checkpoint Skill for Claude Code
# =========================================
# Preserves session context and orchestrates sub-agent calls.
#
# Skill documentation: /opt/agent-governance/checkpoint/CLAUDE.md
#
# Commands:
# now [--notes "..."] Create checkpoint capturing current state
# load [checkpoint_id] Load latest or specific checkpoint
# diff [--from ID] [--to ID] Compare checkpoints
# list [--limit N] List available checkpoints
# summary --level <level> Context summary (minimal/compact/standard/full)
# report [--phase <phase>] Combined checkpoint + directory status report
# timeline [--limit N] Show checkpoint history with status changes
# auto-orchestrate --model <m> Delegate to OpenRouter models
# queue list|add|clear|pop Manage instruction queue
# prune [--keep N] Remove old checkpoints
#
# Examples:
# checkpoint now --notes "Before deploy"
# checkpoint load --json
# checkpoint summary --level compact
# checkpoint auto-orchestrate --model minimax --instruction "run tests"
#
set -euo pipefail
CHECKPOINT_DIR="/opt/agent-governance/checkpoint"
CHECKPOINT_SCRIPT="${CHECKPOINT_DIR}/checkpoint.py"
# Show help if no args or --help
if [[ $# -eq 0 ]] || [[ "${1:-}" == "--help" ]] || [[ "${1:-}" == "-h" ]]; then
cat << 'EOF'
Context Checkpoint Skill
========================
Commands:
now [--notes "..."] Create checkpoint (includes directory statuses)
load [checkpoint_id] [--json] Load checkpoint
diff [--from ID] [--to ID] Compare checkpoints
list [--limit N] [--json] List checkpoints
summary --level <level> Context summary
report [--phase <phase>] Combined checkpoint + directory status report
timeline [--limit N] Checkpoint history with status changes
auto-orchestrate --model <m> Delegate to AI models
queue list|add|clear|pop Manage instruction queue
prune [--keep N] Remove old checkpoints
Summary Levels:
minimal (~500 tokens) Phase + agent only
compact (~1000 tokens) + tasks + key variables
standard (~2000 tokens) + all tasks + dependencies
full (~4000 tokens) Complete context
Models for auto-orchestrate:
minimax Minimax-01 (default, 100K context)
gemini Gemini 2.0 Flash Thinking (free)
gemini-pro Gemini 2.5 Pro (highest capability)
Examples:
checkpoint now --notes "Phase 5 complete"
checkpoint load --json | jq .phase
checkpoint summary --level compact
checkpoint report # Combined status view
checkpoint report --phase in_progress # Filter by phase
checkpoint timeline --limit 5 # Recent history
checkpoint auto-orchestrate --model minimax -i "check status"
checkpoint queue add --instruction "run tests" --priority 5
Documentation: /opt/agent-governance/checkpoint/CLAUDE.md
EOF
exit 0
fi
# Verify checkpoint script exists
if [[ ! -f "${CHECKPOINT_SCRIPT}" ]]; then
echo "Error: checkpoint.py not found at ${CHECKPOINT_SCRIPT}"
exit 1
fi
# Execute checkpoint command
exec python3 "${CHECKPOINT_SCRIPT}" "$@"
Executable
+89
View File
@@ -0,0 +1,89 @@
#!/bin/bash
#
# Ledger API Service
# ==================
# Starts the FastAPI ledger service
#
# Usage:
# ledger-api Start on default port (8080)
# ledger-api --port 8081 Start on custom port
# ledger-api --no-auth Disable authentication
#
set -euo pipefail
LEDGER_DIR="/opt/agent-governance/ledger"
VENV_DIR="${LEDGER_DIR}/.venv"
# Parse arguments
PORT="${LEDGER_API_PORT:-8080}"
AUTH="true"
while [[ $# -gt 0 ]]; do
case $1 in
--port)
PORT="$2"
shift 2
;;
--no-auth)
AUTH="false"
shift
;;
--help|-h)
cat << 'EOF'
Ledger API Service
==================
Usage:
ledger-api Start on default port (8080)
ledger-api --port 8081 Start on custom port
ledger-api --no-auth Disable Vault authentication
Environment Variables:
LEDGER_API_PORT Port to listen on (default: 8080)
LEDGER_API_AUTH Require Vault auth (default: true)
VAULT_ADDR Vault address (default: https://127.0.0.1:8200)
Endpoints:
GET /health Health check
GET /docs Swagger documentation
GET /agents List agents
GET /violations List violations
GET /promotions List promotions
GET /stats Overall statistics
EOF
exit 0
;;
*)
echo "Unknown option: $1"
exit 1
;;
esac
done
# Create venv if needed
if [[ ! -d "${VENV_DIR}" ]]; then
echo "Creating virtual environment..."
python3 -m venv "${VENV_DIR}" 2>/dev/null || {
echo "Installing dependencies globally..."
pip3 install -q fastapi uvicorn pydantic
}
fi
# Activate venv if it exists
if [[ -d "${VENV_DIR}" ]]; then
source "${VENV_DIR}/bin/activate"
# Install dependencies if needed
if ! python3 -c "import fastapi" 2>/dev/null; then
echo "Installing dependencies..."
pip install -q -r "${LEDGER_DIR}/requirements.txt"
fi
fi
# Export config
export LEDGER_API_PORT="${PORT}"
export LEDGER_API_AUTH="${AUTH}"
# Run API
exec python3 "${LEDGER_DIR}/api.py"
Executable
+5
View File
@@ -0,0 +1,5 @@
#!/bin/bash
# LLM Agent Launcher
export PATH="$HOME/.local/bin:$HOME/.bun/bin:$PATH"
cd /opt/agent-governance/agents/llm-planner
uv run agent.py "$@"
Executable
+106
View File
@@ -0,0 +1,106 @@
#!/bin/bash
#
# External Memory Layer CLI for Claude Code
# ==========================================
# Token-efficient storage and retrieval of large outputs, transcripts, and context.
#
# Commands:
# log <content> Store content in memory (auto-chunks large content)
# log --file <path> Store file contents
# log --stdin Read from stdin
# fetch <id> Retrieve memory entry
# fetch <id> --summary-only Get just the summary
# fetch <id> --chunk N Get specific chunk
# list [--type TYPE] List memory entries
# search <query> Search memory by content/tags
# summarize <id> Generate/show summary
# refs --checkpoint <id> Get memory refs for checkpoint
# prune Clean old entries
# stats Show memory statistics
#
# Examples:
# echo "Large output..." | memory log --stdin --tag "test"
# memory fetch mem-20260123-123456-abcd --summary-only
# memory list --type output --limit 10
# memory search "error"
#
# Token Guidelines:
# - Content < 500 tokens: stored inline
# - Content 500-4000 tokens: stored in file with summary
# - Content > 4000 tokens: auto-chunked with parent summary
#
# Integration:
# - Link to checkpoints: --checkpoint <id>
# - Link to directories: --directory <path>
# - Summaries auto-generated for efficient retrieval
#
# Documentation: /opt/agent-governance/docs/MEMORY_LAYER.md
set -euo pipefail
MEMORY_SCRIPT="/opt/agent-governance/memory/memory.py"
# Show help if no args or --help
if [[ $# -eq 0 ]] || [[ "${1:-}" == "--help" ]] || [[ "${1:-}" == "-h" ]]; then
cat << 'EOF'
External Memory Layer
=====================
Commands:
log <content> Store content in memory
log --file <path> Store file contents
log --stdin Read from stdin
fetch <id> Retrieve memory entry
fetch <id> -s Summary only
fetch <id> --chunk N Get specific chunk
list [--type TYPE] List entries
search <query> Search memory
summarize <id> Generate summary
refs --checkpoint <id> Memory refs for checkpoint
prune Clean old entries
stats Show statistics
Options for 'log':
--file, -f <path> Read from file
--stdin Read from stdin
--type, -t <type> Type: transcript, output, context
--tag <tag> Add tag (repeatable)
--checkpoint <id> Link to checkpoint
--directory, -d <path> Link to directory
--no-chunk Don't auto-chunk large content
--json Output JSON
Token Thresholds:
< 500 tokens → stored inline
500-4000 tokens → stored in file + summary
> 4000 tokens → auto-chunked + summary
Examples:
# Store large test output
pytest tests/ 2>&1 | memory log --stdin --tag "pytest" --tag "tests"
# Fetch just the summary
memory fetch mem-20260123-123456-abcd -s
# Get chunk 2 of a large entry
memory fetch mem-20260123-123456-abcd --chunk 2
# List recent outputs
memory list --type output --limit 5
# Search for errors
memory search "failed" --limit 10
Documentation: /opt/agent-governance/docs/MEMORY_LAYER.md
EOF
exit 0
fi
# Verify memory script exists
if [[ ! -f "${MEMORY_SCRIPT}" ]]; then
echo "Error: memory.py not found at ${MEMORY_SCRIPT}"
exit 1
fi
# Execute memory command
exec python3 "${MEMORY_SCRIPT}" "$@"
+9
View File
@@ -0,0 +1,9 @@
#!/bin/bash
#
# Model Controller CLI
# Wrapper for automated orchestration management
#
SCRIPT_DIR="/opt/agent-governance/orchestrator"
python3 "${SCRIPT_DIR}/model_controller.py" "$@"
Executable
+65
View File
@@ -0,0 +1,65 @@
#!/bin/bash
#
# Architectural Test Pipeline CLI
# ================================
# Multi-layer oversight system for continuous validation.
#
# Usage:
# oversight run Full pipeline execution
# oversight run --inject With injection tests
# oversight quick Quick validation
# oversight validate --phase 5 Validate specific phase
# oversight report Generate report
# oversight matrix Show phase status matrix
#
set -euo pipefail
OVERSIGHT_DIR="/opt/agent-governance/testing/oversight"
# Show help if no args or --help
if [[ $# -eq 0 ]] || [[ "${1:-}" == "--help" ]] || [[ "${1:-}" == "-h" ]]; then
cat << 'EOF'
Architectural Test Pipeline
===========================
Multi-layer oversight system for continuous validation across all 12 phases.
Commands:
run [options] Run full pipeline
quick Quick validation (phases + anomalies only)
validate --phase N Validate specific phase in detail
report [--inject] Generate comprehensive report
matrix Show phase status matrix
status Show pipeline status
Options:
--phase N Focus on specific phase
--inject Run injection tests
--unsafe Disable safe mode (caution!)
--auto-fix Enable auto-fix for approved changes
--verbose, -v Verbose output
--json Output as JSON
Layers:
1. Bug Window Watcher Real-time anomaly detection
2. Suggestion Engine AI-driven fix recommendations
3. Council Review Multi-agent decision making
4. Phase Validator Coverage across all phases
5. Error Injector Controlled fault injection
6. Reporter Comprehensive reporting
Examples:
oversight run # Full pipeline
oversight run --inject --verbose # With injection tests
oversight validate --phase 5 # Focus on Phase 5
oversight matrix # Quick status overview
Documentation: /opt/agent-governance/testing/oversight/README.md
EOF
exit 0
fi
# Execute pipeline
cd "${OVERSIGHT_DIR}"
exec python3 -m testing.oversight.pipeline "$@"
+1
View File
@@ -0,0 +1 @@
/opt/agent-governance/pipeline/pipeline.py
+148
View File
@@ -0,0 +1,148 @@
#!/bin/bash
# Agent Registration Script
# Validates and registers a new agent in Vault
set -e
VAULT_ADDR="${VAULT_ADDR:-https://127.0.0.1:8200}"
export VAULT_SKIP_VERIFY=true
usage() {
echo "Usage: $0 -i <agent_id> -r <role> -t <tier> -o <owner> -v <version>"
echo ""
echo "Options:"
echo " -i Agent ID (lowercase, alphanumeric with dashes)"
echo " -r Role: observer|operator|builder|executor|architect"
echo " -t Tier: 0-4"
echo " -o Owner (human email or 'system')"
echo " -v Version (semver: x.y.z)"
echo ""
echo "Environment:"
echo " VAULT_TOKEN Required for registration"
exit 1
}
while getopts "i:r:t:o:v:h" opt; do
case $opt in
i) AGENT_ID="$OPTARG" ;;
r) ROLE="$OPTARG" ;;
t) TIER="$OPTARG" ;;
o) OWNER="$OPTARG" ;;
v) VERSION="$OPTARG" ;;
h) usage ;;
*) usage ;;
esac
done
# Validate required params
[[ -z "$AGENT_ID" || -z "$ROLE" || -z "$TIER" || -z "$OWNER" || -z "$VERSION" ]] && usage
[[ -z "$VAULT_TOKEN" ]] && echo "Error: VAULT_TOKEN not set" && exit 1
# Validate agent_id format
if [[ ! "$AGENT_ID" =~ ^[a-z0-9-]+$ ]]; then
echo "Error: agent_id must be lowercase alphanumeric with dashes"
exit 1
fi
# Validate role
VALID_ROLES="observer operator builder executor architect"
if [[ ! " $VALID_ROLES " =~ " $ROLE " ]]; then
echo "Error: role must be one of: $VALID_ROLES"
exit 1
fi
# Validate tier
if [[ ! "$TIER" =~ ^[0-4]$ ]]; then
echo "Error: tier must be 0-4"
exit 1
fi
# Validate version (semver)
if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "Error: version must be semver (x.y.z)"
exit 1
fi
# Map role to tier and validate consistency
declare -A ROLE_TIER_MAP=(
["observer"]=0
["operator"]=1
["builder"]=2
["executor"]=3
["architect"]=4
)
EXPECTED_TIER="${ROLE_TIER_MAP[$ROLE]}"
if [[ "$TIER" -ne "$EXPECTED_TIER" ]]; then
echo "Warning: role '$ROLE' typically maps to tier $EXPECTED_TIER, but tier $TIER was specified"
fi
# Define allowed/forbidden actions based on tier
case $TIER in
0)
ALLOWED='["read_docs","read_inventory","read_logs","generate_plan"]'
FORBIDDEN='["ssh","create_vm","modify_vm","delete_vm","run_ansible","run_terraform","write_secrets","execute_shell"]'
;;
1)
ALLOWED='["read_docs","read_inventory","read_logs","generate_plan","ssh_sandbox","create_vm_sandbox","run_ansible_sandbox","run_terraform_sandbox"]'
FORBIDDEN='["ssh_prod","ssh_staging","create_vm_prod","create_vm_staging","run_ansible_prod","run_terraform_prod","write_secrets","modify_baseline"]'
;;
2)
ALLOWED='["read_docs","read_inventory","read_logs","generate_plan","ssh_sandbox","create_vm_sandbox","run_ansible_sandbox","run_terraform_sandbox","modify_frameworks","create_templates"]'
FORBIDDEN='["ssh_prod","create_vm_prod","run_ansible_prod","run_terraform_prod","modify_blessed_baseline","direct_prod_access"]'
;;
3)
ALLOWED='["read_docs","read_inventory","read_logs","generate_plan","ssh_sandbox","ssh_staging","ssh_prod_controlled","create_vm_sandbox","create_vm_staging","run_ansible_all","run_terraform_all"]'
FORBIDDEN='["unbounded_root","wide_scope_apply","skip_recording","modify_governance"]'
;;
4)
ALLOWED='["read_all","propose_policy","propose_baseline","request_blessing","emergency_response"]'
FORBIDDEN='["self_approve","self_escalate","bypass_audit"]'
;;
esac
# Set TTL based on tier (higher tier = shorter TTL)
TTL_MAP=(3600 1800 1800 900 900)
TTL=${TTL_MAP[$TIER]}
# Confidence threshold (higher tier = higher threshold required)
CONF_MAP=(0.7 0.75 0.8 0.85 0.9)
CONFIDENCE=${CONF_MAP[$TIER]}
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
echo "Registering agent: $AGENT_ID"
echo " Role: $ROLE (Tier $TIER)"
echo " Owner: $OWNER"
echo " Version: $VERSION"
echo " TTL: ${TTL}s"
echo " Confidence threshold: $CONFIDENCE"
# Register in Vault
docker exec -e VAULT_TOKEN="$VAULT_TOKEN" -e VAULT_ADDR="$VAULT_ADDR" vault \
vault kv put "secret/agents/$AGENT_ID" \
agent_id="$AGENT_ID" \
agent_role="$ROLE" \
owner="$OWNER" \
version="$VERSION" \
tier="$TIER" \
input_contract="secret/docs/schemas/task-request" \
output_contract="secret/docs/schemas/agent-output" \
allowed_side_effects="$ALLOWED" \
forbidden_actions="$FORBIDDEN" \
confidence_reporting=true \
confidence_threshold="$CONFIDENCE" \
ttl_seconds="$TTL" \
status="registered" \
created_at="$TIMESTAMP" \
last_active="$TIMESTAMP" \
compliant_runs=0 \
consecutive_compliant=0 \
violations=0
echo ""
echo "Agent registered successfully."
echo ""
echo "To generate credentials for this agent:"
echo " vault read auth/approle/role/tier${TIER}-agent/role-id"
echo " vault write -f auth/approle/role/tier${TIER}-agent/secret-id"
+99
View File
@@ -0,0 +1,99 @@
#!/bin/bash
# Sample Tier 0 Agent: Plan Generator
# Demonstrates proper agent behavior per foundation document
set -e
# Load bootstrap library
source /opt/agent-governance/lib/agent-bootstrap.sh
AGENT_ID="plan-generator-001"
# --- Agent Metadata Declaration (Section 4) ---
declare -A AGENT_META=(
[agent_id]="plan-generator-001"
[agent_role]="observer"
[version]="0.1.0"
[tier]=0
)
main() {
log_info "Starting agent: ${AGENT_META[agent_id]} v${AGENT_META[version]}"
# Check for required credentials
if [[ -z "$ROLE_ID" || -z "$SECRET_ID" ]]; then
agent_error "CONFIGURATION_ERROR" \
"Missing credentials" \
"Environment variables ROLE_ID and SECRET_ID" \
"None" \
"Set ROLE_ID and SECRET_ID environment variables"
exit 1
fi
# Authenticate (Section 3.3 - Bounded Authority)
if ! agent_authenticate "$ROLE_ID" "$SECRET_ID"; then
agent_error "AUTH_ERROR" \
"Failed to authenticate with Vault" \
"role_id=$ROLE_ID" \
"None" \
"Verify credentials and Vault connectivity"
exit 1
fi
# Load metadata from Vault
if ! agent_load_metadata "$AGENT_ID"; then
agent_error "METADATA_ERROR" \
"Failed to load agent metadata" \
"agent_id=$AGENT_ID" \
"Authenticated successfully" \
"Verify agent is registered in Vault"
exit 1
fi
# Validate action before proceeding (Section 5 - Input Discipline)
local requested_action="${1:-read_docs}"
if ! agent_validate_action "$requested_action"; then
agent_error "FORBIDDEN_ACTION" \
"Requested action is not permitted for this agent" \
"action=$requested_action" \
"Authenticated and loaded metadata" \
"Request action within allowed scope or escalate to higher tier"
exit 1
fi
# Execute action
case "$requested_action" in
read_docs)
log_info "Reading documentation..."
local docs
docs=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
"$VAULT_ADDR/v1/secret/data/docs/agent-taxonomy" | jq -r '.data.data')
if [[ -n "$docs" ]]; then
agent_output "EXECUTE" 0.95 "read_docs" "Successfully read agent taxonomy documentation"
else
agent_output "ERROR" 0.0 "read_docs" "Failed to read documentation"
fi
;;
read_inventory)
log_info "Reading inventory..."
local inventory
inventory=$(curl -sk -H "X-Vault-Token: $VAULT_TOKEN" \
"$VAULT_ADDR/v1/secret/data/inventory/proxmox" | jq -r '.data.data')
agent_output "EXECUTE" 0.90 "read_inventory" "Read Proxmox inventory: $(echo "$inventory" | jq -r '.cluster')"
;;
generate_plan)
log_info "Generating plan..."
# Tier 0 can generate plans but not execute
agent_output "EXECUTE" 0.85 "generate_plan" "Plan generated. Requires Tier 1+ agent for execution."
;;
*)
agent_output "INSUFFICIENT_INFORMATION" 0.0 "$requested_action" "Unknown action requested"
;;
esac
}
# Run main
main "$@"
Executable
+69
View File
@@ -0,0 +1,69 @@
#!/bin/bash
#
# Directory Status Management Skill for Claude Code
# ==================================================
# Manages README.md and STATUS.md files across all subdirectories.
#
# Commands:
# sweep [--fix] Check all directories for README/STATUS files
# update <dir> [options] Update status for a directory
# init <dir> Initialize README/STATUS in a directory
# dashboard Show status overview of all directories
# template [readme|status] Show file templates
#
# Examples:
# status sweep Check all directories
# status sweep --fix Auto-create missing files
# status update ./pipeline --phase "complete" --task "Pipeline unified"
# status dashboard Show project-wide status
set -euo pipefail
STATUS_SCRIPT="/opt/agent-governance/bin/status.py"
# Show help if no args or --help
if [[ $# -eq 0 ]] || [[ "${1:-}" == "--help" ]] || [[ "${1:-}" == "-h" ]]; then
cat << 'EOF'
Directory Status Management
===========================
Commands:
sweep [--fix] Check all directories for README/STATUS
update <dir> [options] Update directory status
init <dir> Initialize README/STATUS files
dashboard Show status overview
template [readme|status] Show file templates
Update Options:
--phase <phase> Set phase (in_progress/complete/blocked/needs_review)
--task <description> Add task entry
--note <note> Add note to status log
--deps <dependencies> Set dependencies
Status Indicators:
[COMPLETE] Directory work is finished
[IN_PROGRESS] Active development
[BLOCKED] Waiting on dependencies
[NEEDS_REVIEW] Requires attention
Examples:
status sweep
status sweep --fix
status update ./pipeline --phase complete
status update ./tests --task "Add chaos tests" --phase in_progress
status dashboard
status init ./new-module
Documentation: /opt/agent-governance/docs/STATUS_PROTOCOL.md
EOF
exit 0
fi
# Verify status script exists
if [[ ! -f "${STATUS_SCRIPT}" ]]; then
echo "Error: status.py not found at ${STATUS_SCRIPT}"
exit 1
fi
# Execute status command
exec python3 "${STATUS_SCRIPT}" "$@"
Executable
+849
View File
@@ -0,0 +1,849 @@
#!/usr/bin/env python3
"""
Directory Status Management System
===================================
Manages README.md and STATUS.md files across the agent-governance project.
Ensures every subdirectory has proper documentation and status tracking.
Usage:
python status.py sweep [--fix]
python status.py update <dir> --phase <phase> [--task <task>]
python status.py init <dir>
python status.py dashboard
python status.py template [readme|status]
"""
import os
import sys
import json
import argparse
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from dataclasses import dataclass, field
from enum import Enum
# Project root
PROJECT_ROOT = Path("/opt/agent-governance")
# Directories to skip (data/generated content)
SKIP_DIRS = {
"__pycache__",
"node_modules",
".git",
"logs",
"storage",
"dragonfly-data",
"credentials",
"workspace",
".claude",
}
# Directories that are data-only (no README needed)
DATA_DIRS = {
"logs",
"storage",
"dragonfly-data",
"credentials",
"workspace",
"packages", # evidence packages are generated
}
class StatusPhase(str, Enum):
"""Status phases for directories."""
COMPLETE = "complete"
IN_PROGRESS = "in_progress"
BLOCKED = "blocked"
NEEDS_REVIEW = "needs_review"
NOT_STARTED = "not_started"
STATUS_ICONS = {
StatusPhase.COMPLETE: "",
StatusPhase.IN_PROGRESS: "",
StatusPhase.BLOCKED: "",
StatusPhase.NEEDS_REVIEW: "",
StatusPhase.NOT_STARTED: "",
}
@dataclass
class DirectoryStatus:
"""Status information for a directory."""
path: Path
has_readme: bool = False
has_status: bool = False
phase: StatusPhase = StatusPhase.NOT_STARTED
last_updated: Optional[datetime] = None
tasks: List[Dict] = field(default_factory=list)
dependencies: List[str] = field(default_factory=list)
issues: List[str] = field(default_factory=list)
# =============================================================================
# Templates
# =============================================================================
README_TEMPLATE = '''# {title}
> {purpose}
## Overview
{overview}
## Key Files
| File | Description |
|------|-------------|
{files_table}
## Interfaces / APIs
{interfaces}
## Status
{status_badge}
See [STATUS.md](./STATUS.md) for detailed progress tracking.
## Architecture Reference
Part of the [Agent Governance System](/opt/agent-governance/docs/ARCHITECTURE.md).
Parent: [{parent_name}]({parent_path})
---
*Last updated: {timestamp}*
'''
STATUS_TEMPLATE = '''# Status: {title}
## Current Phase
{phase_badge}
## Tasks
| Status | Task | Updated |
|--------|------|---------|
{tasks_table}
## Dependencies
{dependencies}
## Issues / Blockers
{issues}
## Activity Log
{activity_log}
---
*Last updated: {timestamp}*
'''
ACTIVITY_ENTRY_TEMPLATE = '''### {timestamp}
- **Phase**: {phase}
- **Action**: {action}
- **Details**: {details}
'''
def get_readme_template() -> str:
return README_TEMPLATE
def get_status_template() -> str:
return STATUS_TEMPLATE
# =============================================================================
# Directory Analysis
# =============================================================================
def should_skip_dir(dir_path: Path) -> bool:
"""Check if directory should be skipped."""
name = dir_path.name
if name.startswith("."):
return True
if name in SKIP_DIRS:
return True
# Skip evidence package subdirs (generated)
if "evd-" in name:
return True
return False
def is_data_dir(dir_path: Path) -> bool:
"""Check if directory is data-only."""
return dir_path.name in DATA_DIRS
def get_all_directories() -> List[Path]:
"""Get all directories that should have README/STATUS."""
dirs = []
for root, subdirs, files in os.walk(PROJECT_ROOT):
root_path = Path(root)
# Filter subdirs in-place to skip certain directories
subdirs[:] = [d for d in subdirs if not should_skip_dir(root_path / d)]
# Skip data directories
if is_data_dir(root_path):
continue
dirs.append(root_path)
return sorted(dirs)
def analyze_directory(dir_path: Path) -> DirectoryStatus:
"""Analyze a directory's status files."""
status = DirectoryStatus(path=dir_path)
readme_path = dir_path / "README.md"
status_path = dir_path / "STATUS.md"
alt_status_path = dir_path / "README.status.md"
status.has_readme = readme_path.exists()
status.has_status = status_path.exists() or alt_status_path.exists()
# Parse STATUS.md if exists
actual_status_path = status_path if status_path.exists() else alt_status_path
if actual_status_path.exists():
try:
content = actual_status_path.read_text()
status.phase = parse_phase_from_status(content)
status.last_updated = parse_timestamp_from_status(content)
status.tasks = parse_tasks_from_status(content)
status.dependencies = parse_dependencies_from_status(content)
status.issues = parse_issues_from_status(content)
except Exception:
pass
return status
def parse_phase_from_status(content: str) -> StatusPhase:
"""Extract phase from STATUS.md content."""
content_lower = content.lower()
if "complete" in content_lower and ("phase" in content_lower or "status" in content_lower):
if "in_progress" not in content_lower and "in progress" not in content_lower:
return StatusPhase.COMPLETE
if "in_progress" in content_lower or "in progress" in content_lower:
return StatusPhase.IN_PROGRESS
if "blocked" in content_lower:
return StatusPhase.BLOCKED
if "needs_review" in content_lower or "needs review" in content_lower:
return StatusPhase.NEEDS_REVIEW
return StatusPhase.NOT_STARTED
def parse_timestamp_from_status(content: str) -> Optional[datetime]:
"""Extract last updated timestamp from STATUS.md."""
import re
# Look for ISO format timestamps
pattern = r'(\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2})'
matches = re.findall(pattern, content)
if matches:
try:
ts = matches[-1].replace(" ", "T")
return datetime.fromisoformat(ts)
except:
pass
return None
def parse_tasks_from_status(content: str) -> List[Dict]:
"""Extract tasks from STATUS.md."""
tasks = []
import re
# Look for task table rows
pattern = r'\|\s*([x\- ✓✗])\s*\|\s*([^|]+)\|'
for match in re.finditer(pattern, content, re.IGNORECASE):
status_char = match.group(1).strip()
task_text = match.group(2).strip()
if task_text and task_text != "Task": # Skip header
done = status_char.lower() in ['x', '✓', 'done', 'complete']
tasks.append({"task": task_text, "done": done})
return tasks
def parse_dependencies_from_status(content: str) -> List[str]:
"""Extract dependencies from STATUS.md."""
deps = []
in_deps_section = False
for line in content.split('\n'):
if '## Dependencies' in line or '## Depends' in line:
in_deps_section = True
continue
if in_deps_section:
if line.startswith('##'):
break
if line.strip().startswith('-'):
deps.append(line.strip()[1:].strip())
return deps
def parse_issues_from_status(content: str) -> List[str]:
"""Extract issues/blockers from STATUS.md."""
issues = []
in_issues_section = False
for line in content.split('\n'):
if '## Issues' in line or '## Blockers' in line:
in_issues_section = True
continue
if in_issues_section:
if line.startswith('##'):
break
if line.strip().startswith('-'):
issues.append(line.strip()[1:].strip())
return issues
# =============================================================================
# File Generation
# =============================================================================
def get_directory_purpose(dir_path: Path) -> str:
"""Get a purpose description for a directory based on its name/location."""
purposes = {
"agents": "Agent implementations and configurations",
"analytics": "Learning analytics and pattern detection",
"bin": "CLI tools and executable scripts",
"checkpoint": "Context checkpoint and session management",
"docs": "System documentation and architecture specs",
"evidence": "Audit evidence and execution artifacts",
"integrations": "External service integrations (GitHub, Slack)",
"inventory": "Infrastructure inventory management",
"ledger": "SQLite audit ledger and transaction logs",
"lib": "Shared library code and utilities",
"orchestrator": "Multi-agent orchestration system",
"pipeline": "Pipeline DSL, schemas, and templates",
"preflight": "Pre-execution validation and checks",
"runtime": "Runtime governance and agent lifecycle",
"sandbox": "Sandboxed execution environments",
"schemas": "JSON schemas for validation",
"teams": "Hierarchical team framework",
"testing": "Test utilities and helpers",
"tests": "Test suites and test infrastructure",
"ui": "User interface components",
"wrappers": "Tool wrappers for sandboxed execution",
"tier0-agent": "Observer-tier agent (read-only)",
"tier1-agent": "Executor-tier agent (infrastructure changes)",
"multi-agent": "Multi-agent coordination demos",
"llm-planner": "LLM-based planning agent (Python)",
"llm-planner-ts": "LLM-based planning agent (TypeScript)",
"chaos": "Chaos testing framework",
"multi-agent-chaos": "Multi-agent chaos test suite",
"github": "GitHub integration module",
"slack": "Slack integration module",
"common": "Common integration utilities",
"framework": "Team framework core implementation",
"templates": "Configuration and pipeline templates",
"examples": "Example configurations and demos",
"schemas": "JSON schema definitions",
"mocks": "Mock implementations for testing",
"unit": "Unit tests",
"integration": "Integration tests",
"scenarios": "Test scenarios",
"governance": "Governance system tests",
"terraform": "Terraform sandbox and examples",
"ansible": "Ansible sandbox and examples",
}
name = dir_path.name
if name in purposes:
return purposes[name]
# Check parent context
parent = dir_path.parent.name
if parent in purposes:
return f"{purposes[parent]} - {name} submodule"
return f"Component of the Agent Governance System"
def get_key_files(dir_path: Path) -> List[Tuple[str, str]]:
"""Get key files in a directory with descriptions."""
files = []
# Prioritize certain file types
priority_patterns = [
("*.py", "Python module"),
("*.ts", "TypeScript module"),
("*.js", "JavaScript module"),
("*.yaml", "Configuration"),
("*.yml", "Configuration"),
("*.json", "Data/Schema"),
("*.sh", "Shell script"),
("*.sql", "Database schema"),
("*.md", "Documentation"),
]
seen = set()
for pattern, desc in priority_patterns:
for f in dir_path.glob(pattern):
if f.name not in seen and not f.name.startswith('.'):
if f.name not in ('README.md', 'STATUS.md', 'README.status.md'):
files.append((f.name, f"{desc}"))
seen.add(f.name)
return files[:10] # Limit to top 10
def generate_readme(dir_path: Path, phase: StatusPhase = StatusPhase.NOT_STARTED) -> str:
"""Generate README.md content for a directory."""
name = dir_path.name
title = name.replace("-", " ").replace("_", " ").title()
purpose = get_directory_purpose(dir_path)
# Get key files
key_files = get_key_files(dir_path)
if key_files:
files_table = "\n".join(f"| `{f}` | {d} |" for f, d in key_files)
else:
files_table = "| *No files yet* | |"
# Get parent info
parent = dir_path.parent
if parent == PROJECT_ROOT:
parent_name = "Project Root"
parent_path = "/opt/agent-governance"
else:
parent_name = parent.name.replace("-", " ").title()
parent_path = f".."
# Status badge
icon = STATUS_ICONS.get(phase, "")
phase_label = phase.value.replace("_", " ").title()
status_badge = f"**{icon} {phase_label}**"
# Overview - try to be smart about it
overview = f"This directory contains {purpose.lower()}."
# Interfaces placeholder
interfaces = "*Document any APIs, CLI commands, or interfaces here.*"
return README_TEMPLATE.format(
title=title,
purpose=purpose,
overview=overview,
files_table=files_table,
interfaces=interfaces,
status_badge=status_badge,
parent_name=parent_name,
parent_path=parent_path,
timestamp=datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC"),
)
def generate_status(dir_path: Path, phase: StatusPhase = StatusPhase.NOT_STARTED,
tasks: List[Dict] = None, dependencies: List[str] = None,
issues: List[str] = None, action: str = "Initialized") -> str:
"""Generate STATUS.md content for a directory."""
name = dir_path.name
title = name.replace("-", " ").replace("_", " ").title()
# Phase badge
icon = STATUS_ICONS.get(phase, "")
phase_label = phase.value.replace("_", " ").upper()
phase_badge = f"**{icon} {phase_label}**"
# Tasks table
tasks = tasks or []
if tasks:
tasks_table = "\n".join(
f"| {'✓' if t.get('done') else '☐'} | {t['task']} | {t.get('updated', 'N/A')} |"
for t in tasks
)
else:
tasks_table = "| ☐ | *No tasks defined* | - |"
# Dependencies
dependencies = dependencies or []
if dependencies:
deps_text = "\n".join(f"- {d}" for d in dependencies)
else:
deps_text = "*No external dependencies.*"
# Issues
issues = issues or []
if issues:
issues_text = "\n".join(f"- {i}" for i in issues)
else:
issues_text = "*No current issues or blockers.*"
# Activity log
timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
activity_log = ACTIVITY_ENTRY_TEMPLATE.format(
timestamp=timestamp,
phase=phase_label,
action=action,
details="Status tracking initialized for this directory."
)
return STATUS_TEMPLATE.format(
title=title,
phase_badge=phase_badge,
tasks_table=tasks_table,
dependencies=deps_text,
issues=issues_text,
activity_log=activity_log,
timestamp=timestamp,
)
# =============================================================================
# Commands
# =============================================================================
def cmd_sweep(args):
"""Check all directories for README/STATUS files."""
dirs = get_all_directories()
missing_readme = []
missing_status = []
outdated = []
print("=" * 60)
print("DIRECTORY STATUS SWEEP")
print("=" * 60)
print(f"Scanning {len(dirs)} directories...\n")
for dir_path in dirs:
status = analyze_directory(dir_path)
rel_path = dir_path.relative_to(PROJECT_ROOT)
issues = []
if not status.has_readme:
issues.append("missing README.md")
missing_readme.append(dir_path)
if not status.has_status:
issues.append("missing STATUS.md")
missing_status.append(dir_path)
if status.last_updated:
age = datetime.now(timezone.utc) - status.last_updated.replace(tzinfo=timezone.utc)
if age.days > 7:
issues.append(f"outdated ({age.days} days)")
outdated.append(dir_path)
if issues:
print(f" {rel_path}: {', '.join(issues)}")
print()
print("-" * 60)
print(f"Missing README.md: {len(missing_readme)}")
print(f"Missing STATUS.md: {len(missing_status)}")
print(f"Outdated (>7 days): {len(outdated)}")
print("-" * 60)
if args.fix and (missing_readme or missing_status):
print("\nFixing missing files...")
for dir_path in set(missing_readme + missing_status):
rel_path = dir_path.relative_to(PROJECT_ROOT)
readme_path = dir_path / "README.md"
status_path = dir_path / "STATUS.md"
if not readme_path.exists():
readme_path.write_text(generate_readme(dir_path))
print(f" Created: {rel_path}/README.md")
if not status_path.exists():
status_path.write_text(generate_status(dir_path))
print(f" Created: {rel_path}/STATUS.md")
print("\nDone!")
elif missing_readme or missing_status:
print("\nRun with --fix to create missing files.")
return 0
def cmd_update(args):
"""Update status for a directory."""
dir_path = Path(args.directory).resolve()
if not dir_path.exists():
print(f"Error: Directory not found: {dir_path}")
return 1
status_path = dir_path / "STATUS.md"
# Parse phase
phase = StatusPhase.IN_PROGRESS
if args.phase:
try:
phase = StatusPhase(args.phase.lower().replace(" ", "_"))
except ValueError:
print(f"Error: Invalid phase '{args.phase}'")
print(f"Valid phases: {', '.join(p.value for p in StatusPhase)}")
return 1
# Read existing or create new
if status_path.exists():
content = status_path.read_text()
# Update phase in existing content
timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
# Add new activity log entry
new_entry = ACTIVITY_ENTRY_TEMPLATE.format(
timestamp=timestamp,
phase=phase.value.upper(),
action=args.task or args.note or "Status update",
details=args.note or f"Phase updated to {phase.value}",
)
# Update phase badge
icon = STATUS_ICONS.get(phase, "")
phase_label = phase.value.replace("_", " ").upper()
new_badge = f"**{icon} {phase_label}**"
import re
content = re.sub(r'\*\*[✅🚧❗⚠️⬜] [A-Z_]+\*\*', new_badge, content)
# Insert new activity entry
if "## Activity Log" in content:
content = content.replace(
"## Activity Log\n",
f"## Activity Log\n\n{new_entry}"
)
# Update timestamp
content = re.sub(
r'\*Last updated: [^*]+\*',
f'*Last updated: {timestamp}*',
content
)
status_path.write_text(content)
else:
# Create new STATUS.md
content = generate_status(
dir_path,
phase=phase,
action=args.task or "Initialized",
)
status_path.write_text(content)
rel_path = dir_path.relative_to(PROJECT_ROOT) if str(dir_path).startswith(str(PROJECT_ROOT)) else dir_path
print(f"Updated: {rel_path}/STATUS.md")
print(f" Phase: {phase.value}")
if args.task:
print(f" Task: {args.task}")
if args.note:
print(f" Note: {args.note}")
# Create lightweight checkpoint delta unless disabled
if not getattr(args, 'no_checkpoint', False):
try:
create_checkpoint_delta(str(rel_path), phase.value, args.task or args.note)
except Exception as e:
# Don't fail the status update if checkpoint fails
print(f" (Checkpoint delta skipped: {e})")
return 0
def create_checkpoint_delta(dir_path: str, phase: str, action: str = None):
"""
Create a lightweight checkpoint recording the status change.
Integrates with the checkpoint system for state tracking.
"""
import subprocess
# Build notes for the checkpoint
action_str = f": {action}" if action else ""
notes = f"Status update - {dir_path} -> {phase}{action_str}"
# Call checkpoint command to create delta
result = subprocess.run(
["python3", "/opt/agent-governance/checkpoint/checkpoint.py",
"now", "--notes", notes],
capture_output=True,
text=True,
timeout=10
)
if result.returncode == 0:
# Extract checkpoint ID from output
for line in result.stdout.split('\n'):
if line.startswith("ID:"):
ckpt_id = line.split(":")[1].strip()
print(f" Checkpoint: {ckpt_id}")
break
def cmd_init(args):
"""Initialize README/STATUS in a directory."""
dir_path = Path(args.directory).resolve()
if not dir_path.exists():
print(f"Error: Directory not found: {dir_path}")
return 1
readme_path = dir_path / "README.md"
status_path = dir_path / "STATUS.md"
rel_path = dir_path.relative_to(PROJECT_ROOT) if str(dir_path).startswith(str(PROJECT_ROOT)) else dir_path
created = []
if not readme_path.exists() or args.force:
readme_path.write_text(generate_readme(dir_path))
created.append("README.md")
if not status_path.exists() or args.force:
status_path.write_text(generate_status(dir_path))
created.append("STATUS.md")
if created:
print(f"Initialized {rel_path}:")
for f in created:
print(f" Created: {f}")
else:
print(f"Files already exist in {rel_path}. Use --force to overwrite.")
return 0
def cmd_dashboard(args):
"""Show status overview of all directories."""
dirs = get_all_directories()
by_phase = {phase: [] for phase in StatusPhase}
for dir_path in dirs:
status = analyze_directory(dir_path)
by_phase[status.phase].append(status)
print("=" * 70)
print("PROJECT STATUS DASHBOARD")
print("=" * 70)
print()
total = len(dirs)
complete = len(by_phase[StatusPhase.COMPLETE])
in_progress = len(by_phase[StatusPhase.IN_PROGRESS])
blocked = len(by_phase[StatusPhase.BLOCKED])
needs_review = len(by_phase[StatusPhase.NEEDS_REVIEW])
not_started = len(by_phase[StatusPhase.NOT_STARTED])
# Progress bar
pct_complete = (complete / total * 100) if total > 0 else 0
bar_width = 40
filled = int(bar_width * complete / total) if total > 0 else 0
bar = "█" * filled + "░" * (bar_width - filled)
print(f"Progress: [{bar}] {pct_complete:.1f}%")
print()
# Summary counts
print(f" ✅ Complete: {complete:3d}")
print(f" 🚧 In Progress: {in_progress:3d}")
print(f" ❗ Blocked: {blocked:3d}")
print(f" ⚠️ Needs Review: {needs_review:3d}")
print(f" ⬜ Not Started: {not_started:3d}")
print(f" ─────────────────────")
print(f" Total: {total:3d}")
print()
# Show non-complete directories
if in_progress or blocked or needs_review:
print("-" * 70)
print("ACTIVE DIRECTORIES:")
print("-" * 70)
for phase in [StatusPhase.BLOCKED, StatusPhase.IN_PROGRESS, StatusPhase.NEEDS_REVIEW]:
if by_phase[phase]:
icon = STATUS_ICONS[phase]
print(f"\n{icon} {phase.value.replace('_', ' ').upper()}:")
for status in by_phase[phase]:
rel_path = status.path.relative_to(PROJECT_ROOT)
age_str = ""
if status.last_updated:
age = datetime.now(timezone.utc) - status.last_updated.replace(tzinfo=timezone.utc)
age_str = f" (updated {age.days}d ago)" if age.days > 0 else " (updated today)"
print(f" {rel_path}{age_str}")
print()
return 0
def cmd_template(args):
"""Show file templates."""
if args.type == "readme":
print(README_TEMPLATE)
elif args.type == "status":
print(STATUS_TEMPLATE)
else:
print("README.md Template:")
print("-" * 40)
print(README_TEMPLATE[:500] + "...")
print()
print("STATUS.md Template:")
print("-" * 40)
print(STATUS_TEMPLATE[:500] + "...")
return 0
# =============================================================================
# Main
# =============================================================================
def main():
parser = argparse.ArgumentParser(description="Directory Status Management")
subparsers = parser.add_subparsers(dest="command", help="Commands")
# sweep
sweep_parser = subparsers.add_parser("sweep", help="Check all directories")
sweep_parser.add_argument("--fix", action="store_true", help="Create missing files")
# update
update_parser = subparsers.add_parser("update", help="Update directory status")
update_parser.add_argument("directory", help="Directory to update")
update_parser.add_argument("--phase", help="Set phase")
update_parser.add_argument("--task", help="Add task description")
update_parser.add_argument("--note", help="Add note")
update_parser.add_argument("--deps", help="Set dependencies (comma-separated)")
update_parser.add_argument("--no-checkpoint", action="store_true",
help="Skip creating checkpoint delta")
# init
init_parser = subparsers.add_parser("init", help="Initialize directory")
init_parser.add_argument("directory", help="Directory to initialize")
init_parser.add_argument("--force", action="store_true", help="Overwrite existing")
# dashboard
subparsers.add_parser("dashboard", help="Show status overview")
# template
template_parser = subparsers.add_parser("template", help="Show templates")
template_parser.add_argument("type", nargs="?", choices=["readme", "status"],
help="Template type")
args = parser.parse_args()
if not args.command:
parser.print_help()
return 1
commands = {
"sweep": cmd_sweep,
"update": cmd_update,
"init": cmd_init,
"dashboard": cmd_dashboard,
"template": cmd_template,
}
return commands[args.command](args)
if __name__ == "__main__":
sys.exit(main())