profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Sprint 4 deployment artifacts: 11 systemd units + REPLICATION.md + env templates

Browse Source 
Builds on ADR-006 to ship the operator-facing bits Sprint 4 was
blocked on. Single-host deploy is now a documented procedure.

deploy/systemd/ (12 files):
- 11 .service units, one per daemon. Each follows the same template:
  Type=simple, User=lakehouse, hardening (NoNewPrivileges,
  ProtectSystem=strict, ProtectHome, PrivateTmp, ReadWritePaths
  scoped to /var/lib/lakehouse + /var/log/lakehouse), JSON to
  journald with per-daemon SyslogIdentifier, EnvironmentFile=- on
  /etc/lakehouse/auth.env.
- Dependency graph baked in via After=/Requires=:
    storaged → standalone (only network-online)
    catalogd → Requires storaged
    ingestd → Requires storaged + catalogd
    queryd → Requires catalogd
    matrixd → Requires embedd + vectord
    gateway → Wants every other daemon (Wants= not Requires=
              so a single upstream restart doesn't cascade-restart
              the gateway)
    pathwayd / observerd / vectord / embedd / chatd → standalone
- chatd unit reads 4 cloud-provider EnvironmentFile=s
  (ollama_cloud / openrouter / opencode / kimi) — each is its own
  file so per-provider key rotation doesn't restart the others.
- lakehouse-go.target: convenience aggregator. Operators
  systemctl start/stop/enable lakehouse-go.target instead of
  managing 11 daemons individually. Per-daemon WantedBy=
  this target.

deploy/etc-lakehouse/ (2 templates):
- auth.env.example: AUTH_TOKEN per ADR-006 6.2 + rotation playbook
  comments. The committed file is empty — operators copy + fill in.
- secrets-go.toml.example: [s3.primary] template with
  REPLACE_ME placeholders. Multi-bucket G2 example commented.

REPLICATION.md (top-level):
- Operator runbook from fresh box → 11 daemons running.
- Prereqs (Go 1.25+, gcc, MinIO, Ollama, optionally Langfuse +
  Postgres for Langfuse) with reachability checks.
- Bind ports table (3110–3220, shifted by 10 from Rust legacy).
- Bootstrap: useradd → build → install → config → secrets →
  systemd → validation.
- Auth posture matrix (loopback / non-loopback / multi-host / TLS).
- Token rotation procedure inline (ADR-006 Decision 6.5).
- Logs (journalctl), backup paths, troubleshooting matrix.

Validation: systemd-analyze verify passed on all 11 .service files
(only "not executable" warnings, expected since binaries don't live
at /usr/local/bin/lakehouse/ until step 2 of bootstrap runs).

Sprint 4 is now operator-ready. Next: Dockerfile + multi-stage
build for container deploys (separate concern; deploy targets
either systemd OR docker, not both).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 18:54:49 -05:00
co-authored by Claude Opus 4.7
parent 814197cfd3
commit a59ef5b930
15 changed files with 625 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/etc-lakehouse/auth.env.example
+23
View File
@@ -0,0 +1,23 @@
# /etc/lakehouse/auth.env — inter-service auth token per ADR-006.
#
# Mode 0600, owned by the lakehouse user (or root if systemd reads it
# before dropping privileges via User=). Loaded by every daemon's
# systemd unit via EnvironmentFile=-/etc/lakehouse/auth.env (the `-`
# prefix means "missing file is OK" so loopback-only deploys can skip
# this entirely).
#
# When the daemon binds non-loopback (anything other than 127.0.0.0/8
# or ::1), AUTH_TOKEN MUST be set — otherwise shared.Run refuses to
# start (R-001 + R-007 mechanical gate). Loopback-only deploys can
# leave this empty.
#
# Token rotation (ADR-006 Decision 6.5):
# 1. Generate new secret
# 2. Set AUTH_SECONDARY_TOKEN to new secret while AUTH_TOKEN stays
# on old (lakehouse.toml [auth].secondary_tokens reads this)
# 3. Update every caller to use new secret
# 4. Promote: AUTH_TOKEN=<new>, clear AUTH_SECONDARY_TOKEN
# 5. Restart daemons (or SIGHUP once hot-reload lands)
AUTH_TOKEN=
# AUTH_SECONDARY_TOKEN= # only set during rotation windows
deploy/etc-lakehouse/secrets-go.toml.example
+25
View File
@@ -0,0 +1,25 @@
# /etc/lakehouse/secrets-go.toml — per-bucket S3 credentials.
#
# Mode 0600, root-owned (storaged + queryd both need to read it; they
# run as the lakehouse user, so chgrp lakehouse + 0640 is fine if the
# group is restricted).
#
# Schema is one [s3.<bucket-logical-name>] block per bucket the
# storaged BucketRegistry should serve. G0 is single-bucket
# ("primary" — see cmd/storaged/main.go primaryBucket const). G2
# multi-bucket federation will add more entries here.
#
# This file is DELIBERATELY NOT in version control. Operators copy
# this template, fill in real credentials, and place at
# /etc/lakehouse/secrets-go.toml. The committed lakehouse.toml [s3]
# block has bucket=<logical-name> + endpoint=<url> + region=<region>
# — only the credentials live here.
[s3.primary]
access_key_id = "REPLACE_ME"
secret_access_key = "REPLACE_ME"
# Future G2 example — multiple buckets:
# [s3.archive]
# access_key_id = "..."
# secret_access_key = "..."
deploy/systemd/lakehouse-catalogd.service
+30
View File
@@ -0,0 +1,30 @@
[Unit]
Description=Lakehouse-Go catalogd — Parquet manifest registry
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target lakehouse-storaged.service
Wants=network-online.target
Requires=lakehouse-storaged.service
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/catalogd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-catalogd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-chatd.service
+41
View File
@@ -0,0 +1,41 @@
[Unit]
Description=Lakehouse-Go chatd — multi-provider LLM dispatcher
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target
Wants=network-online.target
# Operator prereq: Ollama on localhost:11434 for the bare/ollama/
# providers; cloud providers (ollama_cloud, openrouter, opencode,
# kimi) read keys from /etc/lakehouse/<provider>.env per chatd
# config. Missing key files leave that provider unregistered (404
# at first call, never 503).
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/chatd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
# chatd reads provider key files via paths in lakehouse.toml [chatd]
# (ollama_cloud_key_file etc.) — each is its own EnvironmentFile so
# operators can rotate one provider without restarting others.
EnvironmentFile=-/etc/lakehouse/ollama_cloud.env
EnvironmentFile=-/etc/lakehouse/openrouter.env
EnvironmentFile=-/etc/lakehouse/opencode.env
EnvironmentFile=-/etc/lakehouse/kimi.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-chatd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-embedd.service
+33
View File
@@ -0,0 +1,33 @@
[Unit]
Description=Lakehouse-Go embedd — text→vector via Ollama
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target
Wants=network-online.target
# Operator prereq: Ollama running at the URL in lakehouse.toml
# [embedd].provider_url, with default_model loaded (e.g.
# `ollama pull nomic-embed-text-v2-moe`). Not a systemd unit we
# control; embedd surfaces unreachable-Ollama as 502 at request time.
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/embedd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-embedd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-gateway.service
+32
View File
@@ -0,0 +1,32 @@
[Unit]
Description=Lakehouse-Go gateway — single OpenAI-compat-shaped edge proxy
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target lakehouse-storaged.service lakehouse-catalogd.service lakehouse-ingestd.service lakehouse-queryd.service lakehouse-vectord.service lakehouse-embedd.service lakehouse-pathwayd.service lakehouse-observerd.service lakehouse-matrixd.service lakehouse-chatd.service
Wants=network-online.target lakehouse-storaged.service lakehouse-catalogd.service lakehouse-ingestd.service lakehouse-queryd.service lakehouse-vectord.service lakehouse-embedd.service lakehouse-pathwayd.service lakehouse-observerd.service lakehouse-matrixd.service lakehouse-chatd.service
# gateway is the public-facing edge — if any upstream is down, the
# proxy returns 502 at request time. Wants= (not Requires=) so a
# single upstream restart doesn't cascade-restart the gateway.
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/gateway -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-gateway
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-go.target
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Lakehouse-Go — all 11 daemons (gateway + 10 backing services)
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
# A single target operators can `systemctl start` / `stop` / `enable`
# instead of touching each daemon individually. Per-daemon units list
# this target in their [Install].WantedBy=, so `systemctl enable
# lakehouse-go.target` enables every daemon's unit, and a target
# start cascades through Wants=.
[Install]
WantedBy=multi-user.target
deploy/systemd/lakehouse-ingestd.service
+30
View File
@@ -0,0 +1,30 @@
[Unit]
Description=Lakehouse-Go ingestd — CSV → Parquet → catalogd registration
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target lakehouse-storaged.service lakehouse-catalogd.service
Wants=network-online.target
Requires=lakehouse-storaged.service lakehouse-catalogd.service
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/ingestd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-ingestd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-matrixd.service
+30
View File
@@ -0,0 +1,30 @@
[Unit]
Description=Lakehouse-Go matrixd — multi-corpus retrieve+merge with Shape B playbook
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target lakehouse-embedd.service lakehouse-vectord.service
Wants=network-online.target
Requires=lakehouse-embedd.service lakehouse-vectord.service
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/matrixd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-matrixd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-observerd.service
+34
View File
@@ -0,0 +1,34 @@
[Unit]
Description=Lakehouse-Go observerd — witness ring + workflow runner + inbox
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target
Wants=network-online.target
# observerd CAN call matrixd (workflow modes that hit matrix.search)
# but doesn't strictly require it — modes that fail at startup are
# logged and the daemon keeps running. So no Requires= here.
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/observerd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
# observerd's [observerd].persist_path defaults under
# /var/lib/lakehouse/observer/ for ops.jsonl persistence.
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-observerd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-pathwayd.service
+31
View File
@@ -0,0 +1,31 @@
[Unit]
Description=Lakehouse-Go pathwayd — Mem0-style versioned trace store
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/pathwayd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
# pathwayd's [pathwayd].persist_path defaults under /var/lib/lakehouse/pathway/
# in production for JSONL append-only persistence.
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-pathwayd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-queryd.service
+33
View File
@@ -0,0 +1,33 @@
[Unit]
Description=Lakehouse-Go queryd — DuckDB-over-S3 SQL surface
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target lakehouse-catalogd.service
Wants=network-online.target
Requires=lakehouse-catalogd.service
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
# queryd's DuckDB httpfs path needs S3 credentials; secrets_path
# in lakehouse.toml [queryd] points at /etc/lakehouse/secrets-go.toml
# which storaged also reads.
ExecStart=/usr/local/bin/lakehouse/queryd -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-queryd
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-storaged.service
+39
View File
@@ -0,0 +1,39 @@
[Unit]
Description=Lakehouse-Go storaged — S3-backed object store gateway
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target
Wants=network-online.target
# Operator prereq: MinIO (or AWS S3) reachable at the URL in
# /etc/lakehouse/secrets-go.toml [s3.primary]. Not a systemd unit
# we control, so we just wait for network and let the bind+probe
# in storaged surface unreachable-bucket errors.
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/storaged -config /etc/lakehouse/lakehouse.toml -secrets /etc/lakehouse/secrets-go.toml
Restart=on-failure
RestartSec=5
# Per ADR-006 Decision 6.2: auth token from env, not committed TOML.
# Empty AUTH_TOKEN is fine for loopback-only deploys (matches
# requireAuthOnNonLoopback gate at startup).
EnvironmentFile=-/etc/lakehouse/auth.env
# Hardening — minimum needed for the daemon to read its config
# + write its log + open its bind port.
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
# Log routing — JSON to journald, structured per slogRequest middleware.
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-storaged
[Install]
WantedBy=lakehouse-go.target
deploy/systemd/lakehouse-vectord.service
+32
View File
@@ -0,0 +1,32 @@
[Unit]
Description=Lakehouse-Go vectord — HNSW vector index registry
Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
After=network-online.target
Wants=network-online.target
# storaged is OPTIONAL for vectord — when [vectord].storaged_url is
# empty, indexes are in-memory-only and don't survive restart. When
# set, vectord persists LHV1 files via storaged. Operator's call.
[Service]
Type=simple
User=lakehouse
Group=lakehouse
WorkingDirectory=/var/lib/lakehouse
ExecStart=/usr/local/bin/lakehouse/vectord -config /etc/lakehouse/lakehouse.toml
Restart=on-failure
RestartSec=5
EnvironmentFile=-/etc/lakehouse/auth.env
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
StandardOutput=journal
StandardError=journal
SyslogIdentifier=lakehouse-vectord
[Install]
WantedBy=lakehouse-go.target