Building a self-hosted, multi-tenant tunnel service in four days.

21tunnel started the week with a working transport and no dashboard. Four days later it had signup, email verify, hybrid approval, org-scoped RBAC, MFA, session rotation with theft detection, Stripe Checkout, and a superadmin console. This is a build-log post — what shipped each day and the three or four decisions that made the velocity possible.

What we built

The shape of the MVP, from the design doc:

• Hybrid self-serve signup → org pends → superadmin approves.
• JWT access (15 min) + opaque refresh cookie (30 day, rotating, HttpOnly, SameSite=Lax).
• Optional TOTP MFA + 10 one-shot recovery codes, hashed with argon2id.
• Owner / admin / member / viewer RBAC per org; a single is_superadmin bit that bypasses org checks.
• Stripe Checkout + Customer Portal + HMAC-verified webhook.
• Every mutating action writes an audit-log row, partitioned by month.

Week 1 — auth foundations

Day one was the bottom of the stack: argon2id password hashing, opaque-token generation, SHA-256 at-rest storage, JWT issue/verify, and repositories for sessions + refresh_tokens + email_tokens.

crates/qnt-core/src/password.rspub fn hash_password(password: &str) -> Result<String, PasswordError> {
    let salt = SaltString::generate(&mut OsRng);
    Argon2::default()
        .hash_password(password.as_bytes(), &salt)
        .map(|h| h.to_string())
        .map_err(|e| PasswordError::Argon2(e.to_string()))
}

pub fn generate_opaque_token() -> String {
    let mut bytes = [0u8; 32];   // 256 bits
    OsRng.fill_bytes(&mut bytes);
    URL_SAFE_NO_PAD.encode(bytes) // 43-char base64url
}

The JWT module is short on purpose — jsonwebtoken handles HS256 and expiry validation, we just carry the minimum needed in claims:

crates/qnt-server/src/web_auth.rspub struct AccessClaims {
    pub sub:  Uuid,     // user
    pub org:  Uuid,     // active org at login
    pub role: String,   // owner | admin | member | viewer
    pub su:   bool,     // is_superadmin
    pub jti:  Uuid,     // session_id — the middleware revokes by this
    pub exp:  u64,
    pub iat:  u64,
}

Refresh-token rotation

The refresh-token table carries a rotated_to foreign key that chains to its replacement. Presenting an already-rotated token revokes the whole session — free theft detection, no heartbeat needed:

// session.rs — rotate_refresh_token
let (old_id, session_id): (Uuid, Uuid) = sqlx::query_as(
    r"SELECT id, session_id FROM refresh_tokens
       WHERE id = $1
         AND rotated_to IS NULL
         AND revoked_at IS NULL
       FOR UPDATE",
)
.bind(old_token_id)
.fetch_optional(&mut *tx)
.await?
.ok_or_else(|| DbError::NotFound("refresh_token(rotation)".into()))?;

// insert new token, mark old one rotated_to the new id
// if this query returned zero rows, the handler calls revoke_session(...)
// — the thief and the legit user both lose access.

The cookie itself is HttpOnly, SameSite=Lax, scoped to /auth/refresh so it only flies on the one endpoint that needs it. Secure is config-gated: on by default, off only in HTTP dev.

Week 2 — RBAC + superadmin

Migration 0004 added the five core tables (sessions, refresh_tokens, email_tokens, mfa_recovery_codes, billing_events, usage_rollups) plus two columns: users.is_superadmin and organizations.approval_status. Intentionally flat — no separate superadmin table, no separate admin app. Superadmin is a boolean bit on the existing user row.

Why a bit? Two reasons:

1. Same login form. A superadmin types the same email + password + TOTP as any other user. No second authentication surface to harden separately.
2. The JWT carries it. The middleware's AuthContext has is_superadmin: bool; a single if ctx.is_superadmin { ... } branch replaces every org-membership check.

The RBAC matrix is enforced by one extractor (RequirePermission) and one ordinal helper (role_level). No per-route hand-written checks, which means the permission matrix in the design doc is also the enforcement surface. Change the matrix, update one helper.

Week 3 — the dashboard

The dashboard is a static Next.js export served from nginx. Auth state lives in a Zustand store; the access token is in-memory only (not persisted), while user + activeOrg + isSuperadmin persist to localStorage — enough to render the shell optimistically on page reload without a flash of unauthenticated UI.

The fetch wrapper does one thing well:

frontend/src/lib/auth.ts// If a request returns 401 AND we aren't already on /auth/refresh,
// try refresh once. Dedupe concurrent refreshes.
if (resp.status === 401 &&
    path !== "/auth/refresh" &&
    path !== "/auth/login") {
  const fresh = await refreshAccessToken();
  if (fresh) resp = await doFetch(fresh);
  else {
    useAuth.getState().clear();
    const returnTo = encodeURIComponent(location.pathname + location.search);
    location.href = `/login?return_to=${returnTo}`;
  }
}

Concurrent 401s share one in-flight refresh Promise (refreshInFlight) so a dashboard mount with five parallel queries doesn't rotate the token five times and race the theft-detection path. Small detail, real bug avoided.

Week 4 — dual_auth, MFA, Stripe

Two problems on day four: (a) the existing tunnel/token/event routes were still gated by a static admin bearer — nginx was no longer injecting it, so logged-in tenants got 401 on every dashboard call. (b) we wanted MFA and Stripe, but couldn't break the ops scripts that still use the static bearer.

One middleware solved both:

api/middleware_jwt.rspub async fn dual_auth(/* ... */) -> Response {
    // Fast-path: matches the static admin bearer → synthetic superadmin ctx
    if presented_matches_static_admin_bearer {
        req.extensions_mut().insert(AuthContext {
            user_id:       defaults::default_user_id(),
            active_org:    defaults::default_org_id(),
            role:          "owner".into(),
            is_superadmin: true,
            session_id:    Uuid::nil(),
        });
        return next.run(req).await;
    }
    // Otherwise fall through to JWT verification.
    require_user_auth(Extension(state), req, next).await
}

Handlers read ctx.active_org uniformly. Admin bearer → default org's view. Tenant JWT → tenant's view. One middleware, no branching in handlers.

MFA without external deps

TOTP via the totp-rs crate. Recovery codes are 10-byte random, formatted xxxxx-xxxxx-xxxxx, argon2-hashed before storage, shown once. The login flow short-circuits on mfa_enabled:

// /auth/login
if user.mfa_enabled {
    // Stateless challenge: a short JWT with role="mfa_challenge", 5-min TTL.
    return Json(json!({
        "mfa_required": true,
        "challenge_token": issue_mfa_challenge(user.id, &enc)?
    }));
}
// Otherwise: session + access_token + refresh cookie as usual.

The challenge itself is a signed JWT — no DB row for the in-flight login. /auth/login/mfa verifies it, accepts either a TOTP code or a recovery code, and issues the real session. Servers stay stateless between the two legs.

The TOTP secret is sealed at rest with an XOR-SHA256 stream keyed on the master key — not defense-grade AEAD yet, but enough that a DB leak doesn't directly hand out live TOTP seeds. AEAD upgrade is on the follow-up list.

What's next

The MVP ships as-is for self-host users. The outstanding items for a public launch:

• HTTPS + app.21tunnel.com — DNS record, Certbot, secure_cookies = true. Runbook is already in the repo.
• Resend sending-domain verification — signup emails currently log tokens at INFO when Resend isn't configured; fine for dev, not for real users.
• Stripe test-mode → live-mode cutover — the code is done, waiting on the account.
• Nightly metered-bandwidth reporting — schema (usage_rollups) is ready; the job that reports to Stripe's metered-billing endpoint is the last missing piece.
• AEAD for the MFA secret — swap the stream cipher for chacha20poly1305.

If you want to watch the build in real time, the repo is github.com/vikasswaminh/21tunnel. Every commit in this post is in master with the SHA. Pull it tonight.

---