Skip to content

RTOpacks System Map — v1 (10,000-foot)

Status: v1. First concrete cartography per ADR-018. Describes the substrate as it actually is today, not as it will be after ADR-016/017 consolidation lands. Authored: 2026-05-27 by Alex per CARTOGRAPHY-PRELIM-01 brief. Resolution: System level. Module-level and component-level drilldowns are subsequent briefs. Naming convention: working hypothesis per ADR-018 — <scope>-<module-or-domain>-<function>-<artefact-type>. Awkwardness noted in § Notes below; convention is refinable.

The map

flowchart TB

    USER([Users<br/>RTO operators, admins,<br/>prospective signups])

    %% ===================== EXTERNAL BOUNDARY =====================
    subgraph external["External providers — outside RTOpacks scope"]
        direction LR
        REG["Regulators<br/>training.gov.au (Swagger),<br/>yourcareer.gov.au,<br/>ASQA notifications (not yet wired)"]
        LM["Labour-market data<br/>ABS via JSA proxy<br/>(OSL, IVI feeds)"]
        FUND["Funding + tender data<br/>AusTender, Magda discovery,<br/>TEQSA, CRICOS"]
        BILL["Billing providers<br/>Stripe, QuickBooks"]
        COMMS["Comms providers<br/>Resend / CF Email (in use),<br/>Twilio (scaffold; superseded),<br/>Cellcast (planned per ADR-014)"]
    end

    %% ===================== EDGE / AUTH =====================
    subgraph edge["Edge + auth layer"]
        CFA["cf-access-gateway<br/>(Cloudflare Zero Trust;<br/>gates admin surface today,<br/>credential decision per ADR-006 pending)"]
    end

    %% ===================== CUSTOMER-FACING SURFACES =====================
    subgraph surfaces["Customer-facing surfaces"]
        direction LR
        SITE["rtopacks-site-surface<br/>(rtopacks.com.au — public,<br/>waitlist, checkout, qual search)"]
        ADMIN["rtopacks-admin-surface<br/>(admin.rtopacks.com.au —<br/>operator console, CF Access gated)"]
        WS["rtopacks-workspace-surface<br/>(my.rtopacks.com.au —<br/>client workspace; Studio, Radar,<br/>People, Documents)"]
        PRE["rtopacks-prelaunch-surface<br/>(waitlist + magic-link verify)"]
    end

    %% ===================== INTERNAL API MEDIATION =====================
    subgraph mediation["Internal API mediation"]
        direction LR
        IAPI["rtopacks-internal-api<br/>(service-bound by all apps;<br/>holds catalogue + billing +<br/>identity routes)"]
        APUB["rtopacks-api-public<br/>(api.rtopacks.com.au —<br/>Bearer auth, public catalogue API)"]
        MCP["rtopacks-mcp<br/>(MCP tools API)"]
    end

    %% ===================== SUBSTRATE INGESTION =====================
    subgraph sync["Sync substrate (Layer 1 ingestion)"]
        SUB_W[("Ingest workers<br/>tga-sync + tga-ingest, cricos-sync,<br/>enrich-sync, teqsa-sync, cal-sync,<br/>jsa-ingest, osl-ingest, ivi-ingest,<br/>magda-monitor, radar-crawl,<br/>rto-tender-sync, stats-cache")]
        WATCH[("Sync Watchtower<br/>8 _ingest_runs tables (IRSL pattern,<br/>commit 3877313f) + tga_sync_steps<br/>(canonical sync-status surface)")]
        PROXIES[("Outbound proxies<br/>jsa-proxy (POST→GET converter),<br/>rtopacks-geocoder,<br/>rto-docs-proxy, rto-trust-proxy")]
        QUEUES[("CF Queues<br/>rtopacks-ingest-queue,<br/>enrich-sync-queue (self-chain)")]
    end

    %% ===================== OPERATIONAL MACHINERY =====================
    subgraph ops_mech["Operational machinery"]
        OPS_W[("Operational workers<br/>traffic-logger (daily prune),<br/>rto-traffic-snapshot (zone aggregate),<br/>d1-warmer (keep-warm),<br/>session-archiver (D1→R2),<br/>qb-reconcile (billing→QB)")]
    end

    %% ===================== STORAGE LAYER =====================
    subgraph storage["Storage layer"]
        subgraph d1_l1["Layer 1 substrate D1s"]
            NRT["rto-nrt-db<br/>(apex; TGA mirror corpus +<br/>4 IRSL tables + tga_snapshots +<br/>stats_cache_ingest_runs)"]
            RADAR["rto-radar-db"]
            ABS_DB["rto-abs-db"]
            MICRO["rto-micro-db (micro-credentials)"]
            LIC["rto-licensing-db"]
            LAND["rto-landscape-db"]
            CAL_DB["rto-calendar-db"]
            INTAKE["rto-intake-db"]
        end
        subgraph d1_l23["Layer 2/3 (ops + workspace) D1s — mixed today"]
            OPS["rto-ops-db<br/>(62 tables; 24 customer-touching<br/>per OPS-DB-CONTENT-AUDIT-01;<br/>split pending per ADR-008/009/016)"]
            ENGINE["engine-db-oc<br/>(workspace-db: users (canonical),<br/>sessions, documents — vestigial<br/>name from pre-RTOpacks era)"]
        end
        subgraph kvs["KV namespaces"]
            KV["KV: STATS_CACHE, RTOPACKS_SESSIONS,<br/>RTOPACKS_OPERATIONAL_KEYS,<br/>LEADS, MCP_API_KEYS, SEARCH_CACHE,<br/>RADAR_CURSOR, ANON_THREAT_KV"]
        end
        subgraph r2["R2 buckets"]
            R2_ARCH["rtopacks-session-archive<br/>(AES-256 encrypted; 12mo retention)"]
        end
    end

    %% ===================== PRINCIPAL FLOWS =====================

    %% Inbound user
    USER --> SITE
    USER --> WS
    USER --> ADMIN
    USER --> PRE
    USER -.-> APUB
    USER -.-> MCP

    %% CF Access gates admin
    ADMIN -.->|outer gate| CFA

    %% Customer-facing → Internal mediation
    SITE -->|service binding| IAPI
    ADMIN -->|service binding| IAPI
    WS -->|service binding| IAPI
    SITE -->|via public API| APUB

    %% Internal-api → Storage
    IAPI -->|read catalogue| NRT
    IAPI -->|read/write ops| OPS
    IAPI -->|read users + sessions| ENGINE
    IAPI -->|cache| KV
    APUB -->|read catalogue| NRT

    %% Internal-api → External outbound
    IAPI -->|billing API| BILL
    IAPI -->|outbound mail| COMMS
    PRE -->|magic-link mail| COMMS

    %% Substrate ingestion flows
    SUB_W -->|reads| REG
    SUB_W -->|reads| LM
    SUB_W -->|reads| FUND
    SUB_W -->|via| PROXIES
    PROXIES -->|mediates| LM
    SUB_W -->|writes Layer 1| NRT
    SUB_W -->|writes Layer 1| RADAR
    SUB_W -->|writes Layer 1| ABS_DB
    SUB_W -->|writes Layer 1| INTAKE
    SUB_W -->|writes status| WATCH
    SUB_W -->|messages| QUEUES
    QUEUES -->|consumes| SUB_W

    %% Operational machinery
    OPS_W -->|prune/snapshot| OPS
    OPS_W -->|archive| R2_ARCH
    OPS_W -->|read sessions| ENGINE
    OPS_W -->|reconcile| BILL
    OPS_W -->|writes status| WATCH

    %% Style
    classDef external_style fill:#fff4e6,stroke:#ff8c00,color:#333
    classDef edge_style fill:#e8f0ff,stroke:#4a6cf7,color:#333
    classDef surface_style fill:#e6f7ed,stroke:#22a06b,color:#333
    classDef storage_style fill:#f0e6ff,stroke:#8a55cc,color:#333
    classDef pending fill:#fafafa,stroke:#999,stroke-dasharray:4 3,color:#666

    class REG,LM,FUND,BILL,COMMS external_style
    class CFA edge_style
    class SITE,ADMIN,WS,PRE surface_style
    class NRT,RADAR,ABS_DB,MICRO,LIC,LAND,CAL_DB,INTAKE,OPS,ENGINE,KV,R2_ARCH storage_style

How to read this map

This is a system-level map. Each node represents a system (a collection of components serving a coherent function), not a specific worker, table, or endpoint. Module-level and component-level maps drill into individual systems and live in their own files when those briefs land.

The map shows seven layers organised top-to-bottom:

  1. Users — entry point. RTO operators, admins, prospective signups.
  2. External providers — the boundary line. Anything above this is outside RTOpacks; anything below is inside.
  3. Edge + auth — what gates inbound user traffic. Currently just CF Access on the admin surface; the credential model for site + workspace is pending per ADR-006.
  4. Customer-facing surfaces — the four worker apps users actually touch.
  5. Internal API mediation — the service-bound layer between apps and storage. internal-api is the heaviest; api and mcp are narrower public surfaces.
  6. Sync substrate, operational machinery — the back-end. Ingestion writes Layer 1; operational machinery touches Layer 2/3 + R2 + external services.
  7. Storage layer — D1s (split by Layer-1 / Layer-2-3 mixing), KV, R2.

Flows are directional. Solid arrows are active flows in production. Dashed arrows are gates (CF Access) or service-routing structures.

The external subgraph is rendered top because data from upstream flows downward into the substrate; internal flows then propagate upward to surface to users.

Naming-convention notes

ADR-018's proposed convention is <scope>-<module-or-domain>-<function>-<artefact-type> with artefact-type vocabulary module, bus, wrapper, db, kv, r2, worker, surface, connector.

Where the convention worked cleanly:

  • D1 databases — rto-nrt-db, rto-ops-db, rto-radar-db, etc. — already named <scope>-<purpose>-<db>. Convention fits as 3-part <scope>-<domain>-<artefact-type> (no separate function dimension needed).
  • Workers I would name fresh today: rto-tender-sync (scope=rto, domain=tender, function=sync, artefact-type=worker — 4-part). tga-mirror-ingest-worker would fit the 4-part shape if we were renaming rtopacks-tga-ingest.
  • Customer-facing apps re-cast as surfaces: rtopacks-site-surface, rtopacks-admin-surface, rtopacks-workspace-surface work cleanly.

Where the convention felt awkward — list of working-name reservations:

  1. rtopacks-internal-api doesn't fit the 4-part shape; it's <scope>-<function>-<artefact-type> with no domain dimension. The "internal API" IS the function; there's no separate "module or domain" to name. Options: (a) accept 3-part shape for cross-cutting infrastructure; (b) treat it as rtopacks-mediation-internal-api (forced); (c) split internal-api into per-domain mediators (e.g. rtopacks-catalogue-mediator-worker, rtopacks-billing-mediator-worker) and retire the monolith — that's an actual architectural decision, not just a naming question.

  2. engine-db-oc breaks the convention twice: engine is a UCCA-engine reference from the pre-RTOpacks era, not an RTOpacks scope; -oc is a legacy suffix (per existing memory entry, "pre-RTOpacks era pattern"). A spine-conforming name would be rto-workspace-db. The rename is its own brief and is non-trivial (touches 5+ workers).

  3. Outbound proxies — jsa-proxy, rto-docs-proxy, rto-trust-proxy, rtopacks-geocoder. None of these use the -wrapper or -connector artefact-type from ADR-018's vocabulary, even though they ARE wrappers/connectors. Working hypothesis: proxy is a legitimate sibling artefact-type to wrapper — a wrapper exposes the full provider API surface (per ADR-015); a proxy translates contracts (e.g. POST→GET converter) without wrapping the full surface. If that distinction holds, the artefact-type vocabulary should extend to include proxy.

  4. KV namespaces — STATS_CACHE, RTOPACKS_SESSIONS, RTOPACKS_OPERATIONAL_KEYS use UPPER_SNAKE_CASE per the CF binding-name convention. The map shows them with their current names; the spine convention would have them as rtopacks-stats-cache-kv, rtopacks-sessions-kv, etc. Renaming KV namespaces is destructive (requires data migration). Working approach: KV namespaces keep their current names but acquire ADR-018-compliant aliases in the map and in code (const statsCacheKv = env.STATS_CACHE).

  5. Worker names containing rtopacks- prefix — rtopacks-tga-ingest, rtopacks-internal-api, rtopacks-site. The prefix is the <scope> part of the convention but it's redundant for everything in the RTOpacks CF account. Some workers omit it (tga-sync, cricos-sync, enrich-sync); some include it. No consistent rule today. Working hypothesis: omit the prefix for components that have a single-scope home; include it for components that might have cross-account siblings (none today; UCCA-side components, when they exist, would live in a different CF account anyway).

  6. stats-cache — does double duty as worker + KV namespace + (now) stats_cache_ingest_runs D1 table. Three artefacts share one stem. The convention says one name per component; that's OK because stats-cache-worker, stats-cache-kv, stats-cache-runs-table are all clear. But the existing names don't carry the suffix.

  7. rto- vs rtopacks- scope prefix — the substrate is inconsistent. D1s use rto- (rto-nrt-db, rto-ops-db). Workers use rtopacks- (rtopacks-tga-ingest, rtopacks-internal-api). The reasoning is not documented; my guess is rto is the domain (Registered Training Organisations) and rtopacks is the product, but the convention doesn't cleanly separate them. Both are reasonable; consistency is the missing piece.

Substrate features that didn't fit cleanly into the system-level taxonomy

  1. The stats-cache worker. It's a sync worker (ingests from ABS + TGA), it's a cache (writes to KV), and it writes a Layer 1 artefact (tga_snapshots rows in rto-nrt-db). At system level it belongs in three places: sync substrate, internal API mediation (because apps/site reads from its KV), and storage layer. The map places it in sync substrate as the most-load-bearing role.

  2. Outbound API call observability. ADR-017's bus pattern doesn't exist yet. Every outbound call (Stripe, QB, TGA, ABS via proxy, Resend, Cellcast-future) is made inline from whichever worker needs it. The map shows these as direct edges from worker → external provider. When the outbound-bus structure lands, the map gains a rtopacks-outbound-bus node that mediates these flows.

  3. Audit-trail writing. Spine § 2 Phase 5 names the Record module as the audit-evidence surface. Today the Record module is not wired; audit-trail data is scattered across interaction_log, ops_log, _ingest_runs tables, tga_sync_steps, and worker-specific log tables in rto-ops-db. The map omits a unified audit-trail node because there isn't one to draw — the data exists, the surface does not. Worth noting in a Record module spec brief.

  4. Inbound webhooks. Stripe and CF Email webhooks land somewhere (probably handled inline by internal-api and apps/admin, but not verified at the system level for this map). ADR-017's inbound-bus would consolidate. Today there's no canonical webhook handler to draw on the map. Flagged.

  5. The proxy workers' purpose isn't immediately legible. jsa-proxy exists because the JSA API requires a POST→GET conversion (TLS fingerprint issues). rto-docs-proxy and rto-trust-proxy mediate access to specific external content surfaces. geocoder wraps Google Maps. At the system-level view I lumped them together as "outbound proxies"; module-level cartography should separate them since they're not interchangeable.

  6. The cross-DB read pattern (e.g. apps/admin's bridge in administrators/route.ts from workspace-db.users.id → ops-db.passkey_credentials.user_id, surfaced in OPS-DB-CONTENT-AUDIT-01). At system level the bridge isn't visible; the map shows both DBs as separate nodes. At module level the bridge becomes a named flow.

  7. Stripe + QB partial wrappers (already in code, do not conform to ADR-015 full-API-wrapper discipline). The map shows them as flows; the wrapper-conformance question is a Phase-3 instance of ADR-016's recurrence trigger (Stripe + QB + future Cellcast = three external services; the second instance was the warning, the third instance is now). Worth flagging.

Ad-hoc-attachment flags (per ADR-016)

Things visible at system-level that the map suggests are ad-hoc attachments worth consolidating:

  1. Sync workers write to mixed substrate. Most write to rto-nrt-db (correct for Layer 1 substrate). But rto-tender-sync writes to rto-ops-db (gov_tenders, gov_contracts). The substrate Layer 1 should live in rto-nrt-db (or a sibling Layer-1 DB); ops-db is Layer 2/3 by spine § 4. This is the kind of placement-by-history that ADR-008 / ADR-016 would untangle. Surfaced by OPS-DB-CONTENT-AUDIT-01 already; flagged here at system level too.

  2. engine-db-oc is a Layer 3 DB whose name doesn't reflect it. The -oc suffix is pre-RTOpacks era. ADR-018's naming convention would rename to rto-workspace-db. The rename is its own brief. The map uses the current name and notes the awkwardness here.

  3. External service integrations are partial wrappers (Stripe, QuickBooks). ADR-015 says full-API wrapper. The current partial wrappers are pre-ADR; consolidation is downstream.

  4. No outbound bus (ADR-017). Every outbound call is inline. The map shows direct edges; the bus would mediate.

  5. No inbound webhook bus (ADR-017). Same shape: webhooks land per-worker rather than via a canonical handler.

  6. No canonical naming for KV bindings. STATS_CACHE, RTOPACKS_SESSIONS, etc. — UPPER_SNAKE_CASE per CF convention, but no scope or function dimension named in the binding identifier.

  7. The translation layer in apps/admin/app/api/admin/administrators/route.ts (workspace-db.users.id → ops-db.passkey_credentials.user_id) is the kind of identity-bridge that IDENTITY-MODEL-RATIONALISATION-01 + the cluster-C migration shape will absorb. Visible as a cross-DB read in the map; will disappear when identity consolidates.

What this map is NOT

  • A complete worker inventory (use docs/docs/workers/inventory.md for that)
  • A complete database schema (use docs/docs/architecture/database.md)
  • A complete external API inventory (use individual API reference docs in docs/docs/ops/)
  • A roadmap for refactors (use the downstream consolidation briefs)
  • An authoritative source for which worker binds which DB (use the wrangler configs themselves)

This map is the strategic orientation artefact. It shows what the substrate looks like at the highest resolution, with named components carrying working-hypothesis names per ADR-018. Module-level and component-level maps drill into specific systems; this map orients to the whole.

Carry-forwards for v2 of this map

  • Resolve the 7 naming-convention reservations above into either (a) the convention extends to accommodate them, or (b) the components rename. Per ADR-018 "Naming convention" — "refinement happens against concrete examples when the prelim cartography lands."
  • Re-render after ADR-016/017 consolidation lands (outbound bus, inbound bus, full-API wrappers for Stripe + QB). Map will gain bus nodes and lose direct edges.
  • Re-render after OPS-DB-CONTENT-AUDIT-01's split execution lands (the chosen migration shape — A / B / C — will reshape the Layer 2/3 portion of the storage layer).
  • Re-render after IDENTITY-MODEL-RATIONALISATION-01 — the workspace-db ↔ ops-db identity bridge disappears.
  • Add module-level drilldowns as briefs land. Each module spec includes its own module-level cartography file at docs/docs/ops/cartography/<module>-module-map.md.

v1. Filed alongside the spine + ADR rewrites at commit landing 2026-05-27.