Design: TLS on the 1.0 core

← Viltrum

Status: implemented (1.2) · Tracks: ROADMAP.md Phase 1 · Date: 2026-10-21

Goal

Serve https:// from the same epoll workers that serve cleartext, next to a cleartext listener in the same process, without adding allocation per request. A reverse proxy in front stays a first-class deploy.

Done when:

• app.listen_tls(addr, tls) serves HTTPS/1.1, and app.serve(listeners) runs cleartext and TLS listeners together
• TLS 1.3 is negotiated with clients that offer it; TLS 1.2 is the floor; SSLv3, TLS 1.0 and 1.1 fail the handshake
• A certificate replaced on disk (rename-swapped, no signal) is served on new handshakes within about a second, and a half-written or mismatched pair never replaces the one in use
• The zero-allocation gate passes on the TLS request path
• close_notify is sent before the socket closes

Library

mbedtls 3.6, the copy V ships (thirdparty/mbedtls, used by net.mbedtls). No system dependency.

The build that matters, read from V 0.5.2's mbedtls_config.h:

We cannot change this config from viltrum (it is V's), so the per-connection buffer size is fixed. Under tcc (__TINYC__) V's config turns AES-NI off; -prod builds use gcc or clang and keep it.

Public API

pub struct TlsOptions {
pub:
	cert_file string // PEM chain, leaf first
	key_file  string // PEM private key
}

pub struct Listener {
pub:
	addr string
	tls  ?TlsOptions // none = cleartext
}

pub fn (app &App) listen(addr string) !                       // unchanged
pub fn (app &App) listen_tls(addr string, tls TlsOptions) !   // one TLS listener
pub fn (app &App) serve(listeners []Listener) !               // several, one process

All three block until stop, as listen does today. Certificates are loaded before any listener opens, so a bad path fails at startup. Reload is always on: there is no option to bind a certificate once (see Reload).

Engine

Listeners

engine.serve takes []Listener. Each worker opens one SO_REUSEPORT socket per listener and adds all of them to its epoll set. A Loop keeps lfds []int and, per TLS listener, a worker-local TlsServer (below). The accept path tags a new Conn with its listener index.

Per-worker TLS state

TlsServer (one per worker per TLS listener, never shared across threads)
  conf   mbedtls_ssl_config    policy below; f_cert_cb = pick_cert
  drbg   mbedtls_ctr_drbg      seeded from its own mbedtls_entropy
  pair   &CertPair             current certificate + key
  old    []&CertPair           replaced pairs still referenced by a handshake
  watch  file identity (mtime ns, size, inode) of cert and key, checked once a second

Per-worker conf and drbg mean no locks on the record path. Only the handshake's PSA calls take mbedtls' global mutex.

Conn

Conn gains tls &TlsConn (nil for cleartext) and a handshaking flag. TlsConn holds the mbedtls_ssl_context. It is created the first time a pooled Conn serves a TLS connection and is kept with the Conn. Later connections call mbedtls_ssl_session_reset, which reuses its buffers, so a warm worker does not allocate per connection either. A Conn that served TLS and is reused for cleartext keeps its TlsConn idle.

The BIO callbacks are ours, not mbedtls_net_send/recv: send(fd, …, MSG_NOSIGNAL) (no SIGPIPE) and recv, mapping EAGAIN to MBEDTLS_ERR_SSL_WANT_WRITE / WANT_READ.

The receive side reads ahead. mbedtls asks for a record as a 5-byte header and then the body, which is two recv calls per record and a third to learn the socket is empty. The BIO reads up to 16 KB into a buffer kept with the TlsConn and serves mbedtls from it. Once a recv comes back short, it answers WANT_READ without asking the kernel again; epoll is level-triggered, so data that arrives later still wakes the worker. Because that buffer and mbedtls' own are invisible to epoll, "input pending" means either of them is non-empty, and the worker reads them before it waits (after a flush and after a deferred response).

I/O

Two functions replace the direct recv/send calls in worker_linux.c.v: conn_read(c, buf, n) and conn_write(c, buf, n). Each is a plain syscall for cleartext and mbedtls_ssl_read / mbedtls_ssl_write for TLS. process.v and the HTTP layer do not change.

Timeouts: the handshake runs under read_timeout, which is the same clock as a partial request. A client that connects and never speaks TLS is closed by the sweep, as a slowloris is today.

Policy

• Versions: mbedtls_ssl_conf_min_tls_version(TLS1_2), max TLS1_3
• Suites: TLS 1.3 AES_128_GCM_SHA256, AES_256_GCM_SHA384, CHACHA20_POLY1305_SHA256; TLS 1.2 ECDHE-{ECDSA,RSA}-WITH-{AES-128-GCM,AES-256-GCM,CHACHA20-POLY1305}. No CBC, RC4, 3DES, NULL or export
• Groups: x25519, secp256r1, secp384r1
• ALPN: http/1.1 only (we do not speak h2)
• SNI: not required. A client that sends none gets the listener's certificate
• Session tickets: off at first. A resumed session is a nice-to-have, and without one there is no ticket key to rotate. Revisit with numbers

Reload

f_cert_cb runs on every handshake, with or without SNI, in both TLS 1.2 and 1.3 (ssl_tls12_server.c:1451, ssl_tls13_server.c:1902 in V's tree). It calls mbedtls_ssl_set_hs_own_cert(ssl, &pair.crt, &pair.pk) with the worker's current pair. No lookup, no allocation, no lock.

Once a second, from sweep, each worker stats both paths. If the file identity changed, it parses the files again (mbedtls_x509_crt_parse_file, mbedtls_pk_parse_keyfile) and checks them with mbedtls_pk_check_pair:

• Pair matches: it becomes pair. The old one moves to old and is freed when no handshake that picked it is still running (a count on CertPair, taken in f_cert_cb, released when the handshake ends or the conn closes).
• Pair fails: keep serving the current one and log once per file change. A rename-swap writes the cert and key in two steps, so the first check can see a new cert with the old key. The next second sees both.

Paths are re-opened every time, so a rename onto the path is picked up (we never hold the old inode). This also replaces 0.x's reload_on_sighup. Nothing to opt into, no signal.

Zero allocation

The gate measures V's GC heap. mbedtls allocates through libc: the context buffers once per pooled Conn, and handshake state once per handshake. That is per connection, not per request, and never on the GC heap. The record path (ssl_read/ssl_write) does not allocate. The gate gets a TLS case: N keep-alive requests over one TLS connection, allocated_bytes() unchanged.

Tests

On Linux, in engine/ against a real listener, with a cert from scripts/dev-cert.sh:

1. Handshake and one request with TLS 1.3, and with TLS 1.2 forced
2. Pipelined burst over TLS; a response larger than 16 KB (several records) through a slow reader (partial writes)
3. Plain HTTP sent to the TLS port is closed without a response, not answered in cleartext
4. Client sends nothing after connect: closed at read_timeout
5. close_notify arrives before EOF on Connection: close
6. Rename-swap the pair while serving: new handshakes get the new certificate, existing connections keep working, a half-swapped pair is never served
7. TLS 1.0/1.1 and a CBC-only client fail the handshake
8. Zero-allocation gate over TLS
9. Cleartext and TLS listeners in one serve

Outside CI, before release: openssl s_client posture checks and the HttpArena validate.sh TLS probes.

Measure

hostlab (8-vCPU VMware guest, Xeon E5-2690 v2, kernel-heavy under load), -prod gcc build, server on 4 cores with 4 workers, wrk 4.2 on the other 4, 128 to 256 keep-alive connections. Medians of interleaved runs:

AES-NI and PCLMUL are in use (aesenc and pclmulqdq in the binary; GCM multiplies through mbedtls_aesni_gcm_mult). gprof of a single worker shows no lock and nothing per request beyond record processing: the cost is mbedtls handling GCM one 16-byte block per call (AES and GHASH calls separately), where OpenSSL runs a stitched loop over many blocks. That works out to roughly 250 MB/s per core, which is the limit for large bodies. Handshakes are an RSA-2048 signature each, about 1.2 ms.

Before the read-ahead, strace -c showed 2.2 recv per request over TLS against 1 in cleartext; after it, 1.

Risks

• mbedtls speed. Its AES-GCM is slower than OpenSSL's. If the gap matters, this choice gets revisited with numbers, not before
• Memory. About 33 KB per TLS connection for mbedtls' record buffers (fixed by V's config) plus 16 KB of read-ahead
• PSA global lock. Handshakes on all workers share one mutex. That is fine with keep-alive and could show under a handshake storm. Measure with short-lived TLS connections
• Benchmark rules. HttpArena's standard mode lists "OpenSSL, BoringSSL, rustls, … etc."; mbedtls is the library the language ships, which its "framework bundles a library" exception covers. Say so in the entry