Operator Guide¶

This guide is intended for system administrators and LLM operators managing a Ductile instance. It covers day-to-day operations, monitoring, and administrative safety.

1. System Operations¶

Starting the Service¶

The primary way to run Ductile is in the foreground:

./ductile system start

For production environments, we recommend using a systemd unit. See Architecture for an example configuration.

Reloading Configuration¶

You can reload the configuration without restarting the service by sending a SIGHUP signal or using the CLI:

./ductile system reload

Backups¶

ductile system backup writes a point-in-time snapshot to a single tar.gz archive. The DB snapshot uses SQLite VACUUM INTO, so the gateway can stay running.

ductile system backup --to /backups/ductile-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \
  --scope config

Scope is a nested ladder; each level adds to the previous: - db — DB snapshot only - config (default) — db + ductile config dir - plugins — config + every directory under plugin_roots - all — plugins + every file under environment_vars.include

Each archive embeds a BACKUP_MANIFEST.txt recording ductile version, commit, hostname, source paths, source DB sha256, included items, excluded items with reasons, plugin-root mappings, and any boundary warnings (e.g. api.yaml appearing at scope config, env files appearing at scope all). Inspect with tar -xzOf <archive> BACKUP_MANIFEST.txt without re-extracting the rest.

The command refuses to overwrite an existing destination — operator owns the naming pattern and retention. For a scheduled-backup setup (systemd timer or launchd), see docs/DEPLOYMENT.md §10.

Self-check¶

ductile system selfcheck runs four read-only invariants against the local state DB: - PID lock check (refuses to run while the gateway holds the lock — WAL safety) - PRAGMA integrity_check on the SQLite file - Schema validation (ValidateSQLiteSchema) against the embedded baseline - queue_terminal_freshness — terminal-state job_queue rows older than the retention window (24h default) should not exist

ductile system selfcheck --json

Exit code 0 = healthy, 1 = at least one check failed. Use as a deploy gate between binary swap and re-enabling the service.

2. Monitoring & Observability¶

Real-Time Dashboard (TUI)¶

Ductile includes a built-in terminal UI for real-time visibility:

./ductile system watch --api-key "your-admin-token"

Ductile system watch TUI

The watch view shows: - Service health, uptime, queue depth, and plugin count. - Metadata header (config path, binary path, version). - Pipelines with live status and last activity. - An event stream of recent activity.

Logging¶

Ductile emits structured JSON logs to stdout. These are ideal for consumption by Logstash, Fluentd, or simple jq queries.

./ductile system start | jq 'select(.level == "ERROR")'

SSE Event Stream¶

For custom monitoring tools, subscribe to the live event stream:

curl -N -H "Authorization: Bearer <token>" http://localhost:8080/events

3. Configuration Management¶

Ductile loads config.yaml from the config directory (typically ~/.config/ductile/) and merges any files listed under include:.

Administrative Commands¶

Use the config noun for surgical administration: - Show resolved config: ductile config show (includes all defaults and merges). - Get a specific value: ductile config get plugins.echo.enabled. - Set a value safely: ductile config set plugins.echo.enabled=false --apply.

Operational Integrity (Lock & Check)¶

To prevent unauthorized modifications to sensitive files (like tokens.yaml or webhooks.yaml), Ductile uses BLAKE3 hash verification. For webhook setup and signing examples, see WEBHOOKS.md.

Authorize changes: After editing config files, update the hashes:
```
ductile config lock
```
Validate state: Ductile runs an automatic check at startup. You can run it manually with:
```
ductile config check
```

Strict Mode¶

For hardened environments, enable service.strict_mode: true in your config.yaml. In strict mode: - The system will not start if any file fails integrity verification (no warnings). - The system will not start if any configuration check fails (e.g., missing dependencies). - The system requires at least one API token to be defined if the API is enabled.

Managing Scoped Tokens¶

Create scoped API tokens by passing scopes directly or by providing a scopes JSON file:

./ductile config token create --name "my-service" --scopes "jobs:ro,plugin:rw"

4. API Reference¶

Ductile provides a REST API for programmatic control. By default, it listens on localhost:8080.

Manual Triggering¶

You can manually enqueue any plugin command via the API:

curl -X POST http://localhost:8080/plugin/echo/poll 
  -H "Authorization: Bearer <token>" 
  -H "Content-Type: application/json" 
  -d '{"payload": {"message": "Hello from API"}}'

Job Inspection¶

Retrieve the status and results of any job:

curl http://localhost:8080/job/<job_id> -H "Authorization: Bearer <token>"

For a full list of endpoints and schemas, see the API Reference.

5. Troubleshooting¶

Failed to acquire PID lock: Another instance is running. Check ps aux | grep ductile.
Plugin not running: Ensure it is enabled: true in config.yaml and has a valid schedule.
Database is locked: SQLite concurrency limit. Ductile uses WAL mode to mitigate this, but very high API volume may still trigger it.
Tampering detected: Configuration file was modified without running config lock. Run ductile config lock if the change was intentional.
Plugin directory ignored: If a subdirectory in your plugin_roots contains an entrypoint (like run.py) but no manifest.yaml, Ductile will log a warning and ignore it. Add a manifest to enable discovery.