User guide

1. Overview

Nanodems Connector exposes IP cameras (and any TCP/HTTP/RTSP device on a private LAN) to authenticated remote operators without VPNs, port-forwarding, or static public IPs. A small agent installed on a site PC dials out to the cloud over mTLS+gRPC; the cloud rents back signed proxy URLs and bearer tokens that operators use to view streams or hit vendor SDK ports as if they were on the same LAN.

Three operator-facing access paths share that one agent:

• Browser proxy at /c/{handle}/ — point-and-click for any HTTP/HTTPS camera UI.
• RTSP gateway at :9554/{handle}/ — paste into VLC, NVRs, VMS software that expects an rtsp:// URL.
• Raw WebSocket tunnel at /api/tunnel/{handle} and the nanodems-tunnel CLI — covers vendor SDKs, RTSP from non-URL-driven tools, and any other protocol a browser can't speak. The same WebSocket endpoint is what external operator consoles (OpCon) embed for m2m access via a license GUID.

The system is multi-tenant — each customer organization (tenant) sees only its own sites, devices, agents, and operators. Every tunnel request is bound to its tenant at the cloud: the same handle string presented from a different tenant's session is rejected with 403, not silently routed. Superadmins cross tenants for support; normal users are pinned to a single tenant context.

2. Architecture

Read the spine across the middle. A browser request to /c/{handle}/ hits the cloud over HTTPS, gets forwarded through the agent's mTLS tunnel (the bright orange link), then the agent dials the camera on its LAN. CLI tunnel and OpCon WebSocket follow the same path — only the operator-side protocol changes. The site itself never accepts an inbound connection.

Bytes flow either direction. A browser request to /c/{handle}/ is proxied through cloud → agent → camera; an RTSP player hitting rtsp://gateway:9554/{handle}/ goes the same way; a raw WebSocket to /api/tunnel/{handle} is the fallback for vendor SDKs and protocols the proxy can't rewrite. All three resolve to the same (agent, host, port) tuple via the handle.

Trust model. Each agent has its own X.509 cert (mTLS to cloud); each operator has their own login; CLI sessions carry sliding-30d tokens capped at 180d. The browser proxy speaks TLS 1.2 / 1.3 only to LAN cameras (1.0 / 1.1 dropped for BEAST/POODLE/SLOTH), and the first TLS handshake per handle logs the camera cert subject for forensic context. No static shared secrets in any flow.

3. Core concepts

Tenant — the customer

One organization. Owns sites, agents, devices, users, audit log, billing. A user signs in and is placed in exactly one tenant context (superadmins can switch). Tenants are isolated: tenant A's user cannot see tenant B's devices, even with the URL.

Site — a physical location with a shared LAN

One office, store, building floor, or any networked location where the cameras can talk to each other and to a single PC that runs the agent. A tenant has many sites. A site has one or more agents and many devices.

Agent — the bridge process

A small Go binary running as a system service (Windows Service / systemd unit) on a PC inside the site. On boot it dials the cloud over mTLS+gRPC and keeps a long-lived bidirectional stream open. The cloud sends commands ("open a tunnel to camera X port 80"), the agent answers by dialing the LAN target and piping bytes back. No inbound ports needed at the site.

Each agent has a unique cert (issued at enrollment via a site-bound token). Cert CN is matched against the registered agent list on every Hello — unknown CNs are rejected, not auto-grafted into a site. Revoking the cert kicks the agent off immediately.

Two deployment modes.

• Sidecar agent — the common case. One Windows / Linux PC at the site runs the agent and dials each camera over the LAN. The camera doesn't know about Nanodems; only the agent sees the cloud.
• Embedded agent — for cameras that ship the runtime baked in (Hikvision HEOP, Dahua DHOP). The device calls /claim with its manufacturer serial, cloud finds the matching pre-registered Device row, binds it to the site, and issues a cert. No sidecar PC needed; the agent IS the device.

Device — a single addressable target

Usually a camera, sometimes an NVR / encoder / generic IP device. A device record holds its LAN host (IP or hostname), its allowed ports, optional metadata (manufacturer, model, label, notes), and a globally-unique loopback IP (127.0.0.X) used by the CLI tunnel.

Adding a device bumps the site's AllowlistVersion and pushes the new (host, port) tuples to every connected agent in that site. Without an allowlist entry, the agent refuses to dial — preventing a compromised cloud from sweeping the site's LAN.

Handle — an opaque routing token

A string like h-RkN3Lzm9TgD8e2Bp7Q1xWA minted per (device, port) pair. 128 bits of entropy (22 chars of URL-safe base64 after the h- prefix); older 48-bit handles (h-86c2037a51fb) still resolve until they expire. URLs use the handle instead of raw IP+port:

• https://<cloud>/c/{handle}/ — HTTP/HTTPS browser proxy
• rtsp://<cloud>:9554/{handle}/ — RTSP gateway
• wss://<cloud>/api/tunnel/{handle} — raw WebSocket relay (OpCon, custom clients)

Handles let URLs survive when the camera's LAN IP changes (DHCP, hardware swap). Operators bookmark the URL once; replacing the camera updates the handle's target without breaking the bookmark. Revoking a handle expires the URL instantly.

Tenant-bound. Each handle records the tenant that owns it. A handle minted in tenant A is rejected with 403 if presented through tenant B's session — possession of the string is not sufficient for access on its own.

License — per-tenant capability for external consoles

A GUID issued by the external licensing application and registered against a tenant via the admin API. External operator consoles (OpCon) embed the license GUID as their WebSocket bearer at /api/tunnel/{handle}; cloud resolves the GUID to a tenant, then enforces tenant-scope on the requested handle. The login-user id travels alongside as audit metadata only — the license is the capability, not the user.

Lifecycle is owned externally (issue, rotate, revoke through /api/admin/tenants*). The cloud UI surfaces licenses read-only for forensic context; superadmins can inspect, but mint/revoke goes through the licensing app.

4. Multi-agent sites

A site can have many agents (redundancy + capacity). Common setups:

• Redundant pair — two PCs running the agent. If one goes offline, the other takes over.
• Load distribution — large camera count spread across several mid-sized PCs.
• Network segments — different agents have line-of-sight to different VLAN segments of the same site.

The cloud picks an agent for each tunnel request using sticky-with-failover:

1. If the request carries a handle, the handle's pinned AgentId is preferred.
2. If that agent is offline, the cloud picks the least-loaded online agent in the same site (OrderBy(ActiveStreams) ThenBy(AgentId)).
3. Pinning is sticky for the lifetime of the handle so a camera doesn't bounce between agents mid-session.

Allowlist updates are pushed to all connected agents in the site simultaneously, so any agent can serve any device. Memory + CPU per agent stays low because the bridge is byte-pumping, not parsing protocols.

5. High availability (multi-node setup)

Connector runs as two identical pods in production — every pod is both the API/UI and the agent relay. There is no load balancer, no pfSense, no Hetzner Cloud LB. Failover is handled at the DNS layer + by the client's connection-racing logic, and cross-pod state is kept in sync via a shared MongoDB Atlas cluster.

5.1 The big picture

The Agents page shows the current relay assignment in the "Relay" column. If a pod restarts, its agents reconnect within ~15 s (Happy Eyeballs picks the surviving IP), and the column updates automatically on the next 5 s poll.

5.2 mTLS — what it is and why we use two CAs

TLS (vanilla HTTPS) authenticates the server to the client — the browser verifies google.com's cert is signed by a trusted CA, but Google never sees a cert from the browser. Mutual TLS (mTLS) goes both ways: both ends present a cert, both verify the other belongs to the right CA. Replaces shared-secret bearer tokens with cryptographic proof of identity. Used in two distinct trust domains here:

We run two independent CAs with no shared trust:

The boundary is intentional: compromising one CA does not grant lateral movement to the other listener. The CA root keys live on disk under /opt/nanodems/ca/ (agent CA) and /opt/nanodems/data/interrelay-certs/ (internal CA), 600 perms, owned by the service user.

5.3 Cross-pod tunnel routing (BridgeTunnel)

Because the agent picks ONE pod at boot, a request landing on the OTHER pod has to be forwarded. That's BridgeTunnel — a gRPC RPC over the internal mTLS channel that re-issues the request to the pod that owns the agent. The sequence:

1. Operator's browser request → DNS picks node-2.
2. node-2's HandleProxyEndpoint resolves the handle, gets (agentId, host, port).
3. node-2 calls TunnelManager.OpenStreamWithFailoverAsync. The agent isn't in node-2's local registry.
4. node-2's InterRelayRouter reads agent_presence from Mongo, finds relay = "ubuntu-4gb-fsn1-1" and endpoint = "10.0.0.2:7443".
5. node-2 opens a BridgeTunnel gRPC stream to 10.0.0.2:7443 with mTLS handshake (internal CA).
6. node-1's InterRelayService accepts, opens a local tunnel to the agent via its own TunnelManager, and bridges bytes between the gRPC stream and the agent stream.
7. node-2 hands the resulting Stream back to HttpClient.ConnectCallback. HTTP request flows: browser → node-2 → node-1 → agent → camera, and the response retraces the same path.

The "Relay" column on the Agents page, and the connector_interrelay_* Prometheus metrics (bridges_open, bridge_opens_total, bridge_bytes_total), surface this hop so you can see how much traffic crosses the bridge at any moment.

5.4 Distributed lock leader-elect

Some jobs must run on exactly one pod at a time — otherwise they double-count. The hourly stats rollup is the canonical example: it reads agent_presence, computes per-agent byte deltas, and $incs into stats_hourly. If two pods do this concurrently on the same tick, every counter ends up roughly 2× the real value.

We solve this with a tiny distributed lock stored in Mongo's locks collection. Each lock is one document with a TTL, looking like:

{
  _id:       "lock:hourly-rollup",
  owner:     "ubuntu-4gb-fsn1-1/395167/2614662698",  // host / PID / Ticks
  expiresAt: ISODate("2026-05-26T03:00:18Z")          // TTL — auto-deletes
}

On every tick, each pod runs a single findAndModify with an OR filter ("the lock is mine OR has expired") and an upsert that sets owner = ME and renews expiresAt. One pod's write succeeds, the other gets a "no document matched" and skips this tick.

Three properties fall out for free:

• Crash recovery. If node-1 dies mid-tick, its TTL on expiresAt means Mongo auto-deletes the lock after 90 s. node-2's next tick sees no lock and acquires it. No human intervention.
• Renewal not transfer. The same pod can acquire the lock again on its next tick because the $or filter matches its own owner. Sticky leadership while healthy; instant rebalance on failure.
• No master election protocol. We don't need ZooKeeper / Consul / Raft. The lock is just a Mongo document; the database handles the atomicity.

Different jobs use different lock ids (lock:hourly-rollup, lock:daily-rollup, lock:stats-janitor) so they can be held by different pods concurrently — you might see node-1 doing hourly stats while node-2 is doing the daily reconcile. That's normal and intentional.

5.5 Restart / failover behaviour

6. Common workflows

Onboard a new site

1. Sign in. Click Sites → + New site. Enter a name (e.g. "Istanbul HQ").
2. The site is created with a usable enrollment token. Click into the site row.
3. On the site page, click ⬇ Windows installer (.cmd) (or Linux .sh from Download) to grab a per-site installer. Token is embedded — no manual paste.
4. Run the installer on a site PC as Administrator / root. The agent installs as a service, dials the cloud, and shows up online within ~10 s.
5. Add the second redundancy agent the same way if needed.

Discover and add cameras

1. Go to Discover with the site selected.
2. Click Run discovery (ONVIF WS-Discovery, runs on the agent across its LAN). Cameras appear in seconds.
3. Tick the cameras to add and click + Add selected. Each becomes a Device row with default ports for its protocol mix.
4. If discovery misses cameras (multicast blocked, vendor no-ONVIF, or you already know the IPs), use the Bulk add form below — single IP, IP range, or CIDR.

Edit a device's ports

1. Open the site detail page; in the Devices table click the pencil ✏ next to a row.
2. The popup shows the current ports as a comma list. Add or remove ports, then Save changes.
3. The cloud bumps allowlist version, pushes the new list to connected agents in the site, and re-mints handles for any newly-added port. The new ports are usable immediately.

Open a camera

• Browser (HTTP/HTTPS cameras) — click the port pill on the site detail row. The cloud mints a handle if there isn't one and redirects to /c/{handle}/ in a new tab.
• RTSP — click the RTSP port pill to copy the rtsp://gateway:9554/{handle}/ URL; paste into VLC / your NVR / your VMS.
• Vendor SDK / raw TCP — see the CLI tunnel section.

7. CLI tunnel for vendor SDKs / RTSP / non-browser

The CLI tunnel (nanodems-tunnel) is a small cross-platform binary that runs on the operator's machine and exposes one or more cameras as local TCP listeners on 127.0.0.X. Existing tools (Hikvision Webs login, vendor SDKs, RTSP players) connect as if to a LAN camera.

1. Open the site detail page and click 🛠 Open CLI session. Cloud mints a CLI session token (ops_…) scoped to the site(s) you picked and shows a one-shot dialog with the ready-to-paste command: nanodems-tunnel run --token ops_…. Copy it now — the token plaintext is never shown again (only its hash is stored).
2. Download nanodems-tunnel from Download for your OS, or grab the per-platform link directly under the CLI tunnel card on the site page.
3. Run the copied command. The CLI fetches its bind plan from /api/cli/sessions/plan, opens listeners on 0.0.0.0:<port> for every (loopback IP, port) tuple in scope, and stays alive.
4. Point your tool at 127.0.0.X:<port> as shown in the CLI's startup log. Bytes flow: tool → CLI → cloud → agent → camera over a WebSocket at /api/cli/raw-tunnel/{ip}/{port}.

The CLI refreshes its plan every 60 seconds. Adding / removing cameras at the cloud, or rotating the site's loopback IPs, propagates without restarting the tunnel; new ports are bound on the fly and disappeared ports stop accepting until they reappear in the plan.

Session lifetime. CLI sessions slide 30 days on every successful connect, capped at 180 days from the issue date. Operators stay logged in across a working week without re-clicking; after six months a fresh session must be issued (hard security boundary). Revoke any time from the Active sessions list on the same site page — the running CLI exits on its next request.

Bind failures don't kill the process. If a port is already held by a stale tunnel on the operator's machine, the CLI logs once and keeps retrying on each refresh tick; clean up the stale process and the bind heals without a restart.

8. Programmatic access (API tokens, OpCon license, NDIS keys)

The interactive "Open CLI session" flow above covers an operator at a keyboard. There are three other ways to reach a tunnel that don't require clicking through a web UI — pick the one that matches who's calling.

Per-user API token nt_… (account → CLI Tokens)

Long-lived bearer for your own user, minted at /account/tokens. Use it as Authorization: Bearer nt_… against wss://<cloud>/api/tunnel/{handle} for scripts and personal tooling that opens raw WebSocket tunnels to specific handles. Tenant is fixed at the active tenant when the token was created; revoke any time from the same page. Tokens are shown once on creation — store them somewhere safe.

OpCon license GUID (m2m, external operator console)

External operator consoles (OpCon) embed the tenant's license GUID as their WebSocket auth, no per-user onboarding:

• Browser: new WebSocket(url, "license.{guid}.user.{uid}") or the two-subprotocol form ["license.{guid}", "user.{uid}"].
• Server WS: X-License: {guid} header (and optional X-User-Id for audit).

Cloud resolves the GUID → tenant, enforces tenant-scope on the requested handle, and accepts the connection. The userId slot is audit metadata only — never used for authorization (the license is the capability).

Licenses are issued and rotated by the external licensing application via the admin API (POST /api/admin/tenants on first onboarding, POST /api/admin/tenants/{tenantId}/licenses for rotation, DELETE …/licenses/{licenseId} for revoke). The cloud UI is read-only.

NDIS / partner CLI API key nk_…

Server-to-server credential for tooling that mints per-operator CLI sessions on someone's behalf (NDIS production, OPCON build scripts, partner integrations). Distinct from a CLI session itself: an API key is a long-lived service credential issued by a cloud admin, scoped to one tenant; the sessions it mints are short-lived per-operator tokens.

• POST /api/v1/sessions with X-API-Key: nk_… and a JSON body containing external_operator_id + operator_display_name — returns a fresh ops_… token plus its expiry / hard cap.
• GET /api/v1/sessions — list sessions in the key's tenant.
• DELETE /api/v1/sessions/{tokenHash} — revoke a session (cascades to the running CLI on its next request).

API keys are pinnable to a caller CIDR (allowed_ip_cidr) — strongly recommended for service accounts with a known address. Revoking the key invalidates every session it issued via the Issuer = NdisApi + IssuerRef cascade.

Which one do I use?

9. Stats & retention

The cloud captures per-agent counters every heartbeat and persists hourly + daily rollups:

StatsJanitor runs once per day: it reconciles each day's AgentDailyStat row from its underlying hourly rows (hourly is the source of truth — agent counters survive restart via stats.json), then drops hourly rows older than 30 days and daily rows older than 365 days.

Agent-side: counters are flushed to %ProgramData%\Nanodems\Agent\state\stats.json (Windows) / ~/.cache/nanodems-agent/stats.json (Linux/macOS) every 30 seconds, so a service restart doesn't reset "today's bytes".

10. Troubleshooting

Agent shows offline on /agents

• Check the agent host can reach connector.nddevteam.com:443 (cloud HTTP) and :8443 (mTLS gRPC). Firewall? Outbound proxy?
• On the agent host, sc query NanodemsAgent (Windows) or systemctl status nanodems-agent (Linux). Check the journal for "control stream" lines.
• If the cert is revoked (Audit log → "agent.revoke"), re-enroll with a fresh installer.

"Target not in allowlist"

• Means the agent's local allowlist hasn't picked up a recently-added device or port. Cloud auto-pushes on every edit, but if the agent reconnected mid-edit it might be a tick behind.
• Click Edit on any device in the site to trigger a fresh push, or restart the agent service.

Browser camera page shows 502 / 504

• The agent reached the camera but the camera didn't respond in time. Common with quirky firmware doing CGI scripts — try the same URL with ?t=<random> appended (some Hikvision pages reject duplicate cookies).
• Check if the camera is actually up — direct ping from the agent host (RDP / SSH in) confirms.

CLI tunnel: "rejecting unknown loopback 127.0.0.X:Y"

• The CLI's bind plan is older than the device list. CLI auto-refreshes every 60 s; wait or restart the CLI.
• If the message persists, the (IP, port) tuple isn't in the operator's session scope. Re-issue a session that includes the right sites.

WebSocket tunnel: 403 "handle does not belong to your tenant"

• The handle string was minted in a different tenant than the session / token / license presenting it. Switch to the owning tenant (top-right tenant picker) and re-open the site to get a handle in your current tenant — or use a token issued in the right tenant.
• For OpCon: the license GUID's tenant doesn't match the handle's tenant. Check that the license registered in the external licensing app maps to the same tenant that owns the cameras you're trying to reach.

WebSocket tunnel: 401 "authentication required"

• The relay rejects anonymous access. Pick one of: cookie session (sign in to the cloud UI), Authorization: Bearer nt_… from /account/tokens, Authorization: Bearer ops_… from a CLI session, X-License header for OpCon, or X-Admin-Api-Key for platform-ops triage.
• If a token used to work and now 401s, it may be revoked or expired — check its row on /account/tokens or the Active sessions list on the site page.

Agent rejected on connect: "unknown CN"

• The agent's cert CN is not in the registered agents table for any tenant. Cloud no longer auto-grafts unknown CNs into a default site (privilege-escalation prevention) — re-enroll the agent with a fresh installer to mint a new cert against the right site.
• Look in the Audit log for the matching rejection entry (action agent.reject_unregistered_cn) for forensic context.

"This page isn't working — HTTP 400" on form submit

• Antiforgery token mismatch. Hard-refresh the page (Ctrl+Shift+R) and try again.
• If repeated, sign out + sign in to refresh the cookie.

11. Glossary

12. Version history

See CHANGELOG.md for per-version notes (cloud + agent + CLI versioned independently). The cloud's current version is shown in the page footer.