Skip to content

Operator Runbook

Document: docs/product/operator-runbook.md
Status: Canonical
Last updated: 2026-04-06
Authority: Tim Rignold, RTOpacks Pty Ltd

Who This Is For

The person who runs RTOpacks day-to-day. Currently Tim Rignold. This document tells you how to do every operational task without reading architecture docs.

Add a User to the Allowlist

  1. Go to admin.rtopacks.com.au/access
  2. Select type: Email (specific person) or Domain (entire organisation)
  3. Enter the value and optional label
  4. Click Add

The user can now request a magic link at my.rtopacks.com.au/auth. They will not be told they were previously blocked — the experience is seamless.

To remove access: Click Deactivate next to the entry. This blocks new magic link requests but does not invalidate existing sessions.

Add an RTO Client Organisation

Currently manual via D1 console:

CLOUDFLARE_API_TOKEN=<token> npx wrangler d1 execute ops-db --remote \
  --command="INSERT INTO rto_clients (rto_code, is_client, client_since, client_status, plan) VALUES ('<RTO_CODE>', 1, datetime('now'), 'active', 'essential')"

The RTO code must match a record in rtopacks-db rtos table. When the user logs in with an email linked to this org (via workspace-db users.org_id), they will resolve as L4.

Read the Anomaly Log

CLOUDFLARE_API_TOKEN=<token> npx wrangler d1 execute ops-db --remote \
  --command="SELECT * FROM anomaly_log ORDER BY created_at DESC LIMIT 20"

Or query via the admin panel (route to be built — ADMIN-UI-01).

Columns: session_id, user_id, org_id, ucca_layer, zone (amber/red), trigger_reason (velocity/sequential/breadth/timing/escalation), detail (JSON), action (latency_injected/session_revoked), created_at.

SEC-02 Red Zone Alert Email

Subject: [SEC-02 RED] Session revoked — {user_id}
From: noreply@rtopacks.com.au
To: admin@rtopacks.com.au

What it means: A session exceeded red zone thresholds. The session has been automatically revoked. The user will see a normal "session expired" message — no indication of the revocation reason.

What to do: 1. Check anomaly_log for the session_id in the email 2. Check api_access_log for the full request history of that session 3. Determine if this was a legitimate user (e.g. a power user with many tabs) or an automated agent 4. If legitimate: add a note, no further action needed — the user can re-authenticate 5. If automated: check if the user_id is associated with an org. Consider deactivating the allowlist entry for that email/domain.

Check API Access Log

CLOUDFLARE_API_TOKEN=<token> npx wrangler d1 execute ops-db --remote \
  --command="SELECT endpoint, method, response_status, timestamp FROM api_access_log WHERE user_id = '<USER_ID>' ORDER BY timestamp DESC LIMIT 50"

Suspicious patterns: - Hundreds of sequential /rto/{code} or /units/{code} requests - Sub-second timing between requests - Requests across many endpoint groups with no repeat pattern

Worker Deployment

Always verify account before deploying: 

npx wrangler whoami

Must show Admin@ucca.online's Account (e5a9830215a8d88961dc6c80a8c7442a). If it shows a different account, run CLOUDFLARE_API_TOKEN=<token> prefix.

Deploy order (when all surfaces touched): 1. internal-api — always first (other surfaces depend on it) 2. workspace — auth surface 3. admin — operator surface 4. site — public surface (lowest risk)

Commands: 

cd workers/internal-api && npx wrangler deploy
cd apps/workspace && npm run deploy
cd apps/admin && npm run deploy
cd apps/site && npm run deploy

Key Workers and Their Roles

Worker Domain Purpose
rtopacks-internal-api internal-api.rtopacks.com.au Sole access point for rtopacks-db. Auth + logging + anomaly detection.
rtopacks-workspace my.rtopacks.com.au Workspace app — auth, AppGrid, studio, documents, people
rtopacks-admin admin.rtopacks.com.au Admin panel — behind CF Access. Org management, CRM, allowlist.
rtopacks-site rtopacks.com.au Public site — marketing, search, claim flow
rtopacks-prelaunch rtopacks.com.au (front) Prelaunch blocker — remove at go-live

Key Database Bindings

Database ID Purpose Access
rtopacks-db 334ac8fb NRT corpus — qualifications, units, RTOs, scope READ ONLY via internal-api
ops-db 00daba3d Commercial — clients, access control, audit logs, anomaly logs Read/Write
workspace-db fb6ddc43 Identity — users, teams, groups, sessions, documents Read/Write

Amendment Log

Date Change Authority
2026-04-06 Initial document — Session 44 Tim Rignold