Initial commit: Agent Governance System Phase 8
Phase 8 Production Hardening with complete governance infrastructure: - Vault integration with tiered policies (T0-T4) - DragonflyDB state management - SQLite audit ledger - Pipeline DSL and templates - Promotion/revocation engine - Checkpoint system for session persistence - Health manager and circuit breaker for fault tolerance - GitHub/Slack integrations - Architectural test pipeline with bug watcher, suggestion engine, council review - Multi-agent chaos testing framework Test Results: - Governance tests: 68/68 passing - E2E workflow: 16/16 passing - Phase 2 Vault: 14/14 passing - Integration tests: 27/27 passing Coverage: 57.6% average across 12 phases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
profit
@@ -0,0 +1,219 @@
|
||||
# Agent Governance Pipeline System
|
||||
|
||||
This directory contains the **authoritative pipeline implementation** for the AI Agent Governance System.
|
||||
|
||||
## Architecture Reference
|
||||
|
||||
See `/opt/agent-governance/docs/ARCHITECTURE.md` for the full system design.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
pipeline/
|
||||
├── core.py # AUTHORITATIVE: Core definitions, enums, constants
|
||||
├── pipeline.py # Pipeline DSL parser and executor
|
||||
├── README.md # This file
|
||||
├── schemas/
|
||||
│ └── pipeline.schema.json # JSON Schema for pipeline validation
|
||||
├── templates/
|
||||
│ ├── default.yaml # Generic observer-tier agent
|
||||
│ ├── terraform.yaml # Infrastructure specialist (T1)
|
||||
│ ├── ansible.yaml # Configuration management (T1)
|
||||
│ └── code-review.yaml # Code review specialist (T0)
|
||||
└── examples/
|
||||
├── infrastructure-deploy.yaml
|
||||
└── multi-agent-analysis.yaml
|
||||
```
|
||||
|
||||
## Core Module (`core.py`)
|
||||
|
||||
**All code should import pipeline definitions from `pipeline.core`** to ensure consistency.
|
||||
|
||||
### Agent Lifecycle Phases
|
||||
|
||||
The official agent lifecycle follows these phases in order:
|
||||
|
||||
```
|
||||
BOOTSTRAP → PREFLIGHT → PLAN → EXECUTE → VERIFY → PACKAGE → REPORT → EXIT
|
||||
```
|
||||
|
||||
| Phase | Description | Output Type |
|
||||
|-------|-------------|-------------|
|
||||
| BOOTSTRAP | Agent initialization and authentication | Alpha |
|
||||
| PREFLIGHT | Pre-execution validation (sandbox, inventory, deps) | Alpha |
|
||||
| PLAN | Generate and validate execution plan | Beta |
|
||||
| EXECUTE | Perform the planned actions | Beta |
|
||||
| VERIFY | Validate execution results | Gamma |
|
||||
| PACKAGE | Bundle artifacts and evidence | Gamma |
|
||||
| REPORT | Generate completion report | Gamma |
|
||||
| EXIT | Clean shutdown and resource release | Gamma |
|
||||
| REVOKED | Agent was revoked (terminal state) | - |
|
||||
|
||||
### Importing Core Definitions
|
||||
|
||||
```python
|
||||
from pipeline.core import (
|
||||
# Enums
|
||||
AgentPhase,
|
||||
AgentStatus,
|
||||
OutputType,
|
||||
ChaosCondition,
|
||||
StageType,
|
||||
StageStatus,
|
||||
|
||||
# Data classes
|
||||
AgentOutput,
|
||||
ClarifiedPlan,
|
||||
ErrorBudget,
|
||||
StageResult,
|
||||
PipelineContext,
|
||||
|
||||
# Constants
|
||||
AGENT_PHASE_NAMES,
|
||||
AGENT_PHASES_ORDERED,
|
||||
PHASE_OUTPUT_TYPES,
|
||||
DEFAULT_REDIS_HOST,
|
||||
DEFAULT_REDIS_PORT,
|
||||
DEFAULT_REDIS_PASSWORD,
|
||||
DEFAULT_LEDGER_PATH,
|
||||
|
||||
# Key patterns
|
||||
RedisKeys,
|
||||
|
||||
# Utilities
|
||||
get_output_type_for_phase,
|
||||
is_terminal_phase,
|
||||
next_phase,
|
||||
)
|
||||
```
|
||||
|
||||
### Output Types (Alpha/Beta/Gamma)
|
||||
|
||||
Agents produce outputs at checkpoints, classified as:
|
||||
|
||||
- **Alpha**: Initial/draft outputs (plans, analysis)
|
||||
- **Beta**: Refined outputs (validated plans, partial results)
|
||||
- **Gamma**: Final outputs (completed work, verified results)
|
||||
|
||||
### DragonflyDB Key Patterns
|
||||
|
||||
Use `RedisKeys` class for consistent key naming:
|
||||
|
||||
```python
|
||||
from pipeline.core import RedisKeys
|
||||
|
||||
# Agent keys
|
||||
agent_state = RedisKeys.agent_state("agent-001") # "agent:agent-001:state"
|
||||
agent_lock = RedisKeys.agent_lock("agent-001") # "agent:agent-001:lock"
|
||||
|
||||
# Project keys
|
||||
project_agents = RedisKeys.project_agents("proj-001") # "project:proj-001:agents"
|
||||
```
|
||||
|
||||
## Pipeline DSL (`pipeline.py`)
|
||||
|
||||
The pipeline DSL supports four stage types:
|
||||
|
||||
### Stage Types
|
||||
|
||||
1. **agent**: Execute an agent task
|
||||
2. **gate**: Approval/consensus checkpoint
|
||||
3. **parallel**: Concurrent execution of branches
|
||||
4. **condition**: Conditional branching (if/then/else)
|
||||
|
||||
### Example Pipeline
|
||||
|
||||
```yaml
|
||||
name: example-pipeline
|
||||
version: "1.0"
|
||||
timeout: 30m
|
||||
|
||||
stages:
|
||||
- name: plan
|
||||
type: agent
|
||||
template: default
|
||||
config:
|
||||
tier: 1
|
||||
timeout: 10m
|
||||
|
||||
- name: review
|
||||
type: gate
|
||||
requires: [plan]
|
||||
config:
|
||||
gate_type: approval
|
||||
approvers: ["team-lead"]
|
||||
timeout: 30m
|
||||
|
||||
- name: execute
|
||||
type: agent
|
||||
requires: [review]
|
||||
template: terraform
|
||||
config:
|
||||
tier: 2
|
||||
```
|
||||
|
||||
### Running a Pipeline
|
||||
|
||||
```bash
|
||||
# Validate a pipeline
|
||||
python pipeline/pipeline.py validate pipeline/examples/infrastructure-deploy.yaml
|
||||
|
||||
# Run a pipeline
|
||||
python pipeline/pipeline.py run pipeline/examples/infrastructure-deploy.yaml \
|
||||
--input environment=staging
|
||||
|
||||
# List available templates
|
||||
python pipeline/pipeline.py list
|
||||
```
|
||||
|
||||
## Chaos Testing Integration
|
||||
|
||||
The chaos test framework (`tests/multi-agent-chaos/`) imports from `pipeline.core` to ensure consistency:
|
||||
|
||||
```python
|
||||
# In tests/multi-agent-chaos/orchestrator.py
|
||||
from pipeline.core import (
|
||||
AgentPhase,
|
||||
OutputType,
|
||||
ChaosCondition,
|
||||
AGENT_PHASE_NAMES,
|
||||
RedisKeys,
|
||||
)
|
||||
```
|
||||
|
||||
### Running Chaos Tests
|
||||
|
||||
```bash
|
||||
python tests/multi-agent-chaos/orchestrator.py
|
||||
```
|
||||
|
||||
The chaos test:
|
||||
1. Spawns multiple real agents (Python, Bun, Diagnostic)
|
||||
2. Injects chaos conditions (lock loss, error spikes, etc.)
|
||||
3. Tracks Alpha/Beta/Gamma outputs
|
||||
4. Triggers plan clarification when error threshold crossed
|
||||
5. Verifies unified objective reached via DragonflyDB readiness checks
|
||||
|
||||
## Files Changed for Unification
|
||||
|
||||
The following files were aligned with the architecture spec:
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `pipeline/core.py` | NEW: Authoritative definitions |
|
||||
| `tests/multi-agent-chaos/orchestrator.py` | Updated to import from `pipeline.core` |
|
||||
| `pipeline/README.md` | NEW: This documentation |
|
||||
|
||||
## Consistency Checklist
|
||||
|
||||
When adding new pipeline-related code:
|
||||
|
||||
1. **Import from `pipeline.core`** - never define enums/constants locally
|
||||
2. **Use official phase names** - `AGENT_PHASE_NAMES` or `AGENT_PHASES_ORDERED`
|
||||
3. **Use RedisKeys class** - for consistent DragonflyDB key naming
|
||||
4. **Follow output types** - `PHASE_OUTPUT_TYPES` maps phases to Alpha/Beta/Gamma
|
||||
5. **Include PACKAGE phase** - often forgotten, but required for artifact bundling
|
||||
|
||||
## Version History
|
||||
|
||||
- **v1.0** (2026-01-23): Initial unified pipeline consolidation
|
||||
Reference in New Issue
Block a user