Encrypting AI Memory at Rest: A Practical Walkthrough

Every encrypted memory row in Unimatrix carries exactly 60 bytes of cryptographic overhead: a 32-byte salt, a 12-byte initialization vector, and a 16-byte GCM authentication tag, concatenated in front of the ciphertext. That is the whole envelope format, and it is worth understanding byte by byte, because the interesting problems in encrypting AI memory are not in the cipher. They are in everything the cipher does not cover.

The layout

[ salt: 32 bytes ][ IV: 12 bytes ][ auth tag: 16 bytes ][ ciphertext: N bytes ]
  0            31  32          43  44             59  60             60+N

The salt is fresh random bytes per ciphertext. It is fed with the master key into scrypt to derive a 32-byte AES key that is used exactly once. The IV is fresh random bytes per ciphertext as well. The tag is produced by GCM at encryption time and verified at decryption time before a single byte of plaintext is returned.

import { randomBytes, scryptSync, createCipheriv, createDecipheriv } from 'node:crypto';

const KDF = { N: 16384, r: 8, p: 1, maxmem: 64 * 1024 * 1024 };

export function seal(plaintext, masterKey) {
  const salt = randomBytes(32);
  const iv = randomBytes(12);
  const key = scryptSync(masterKey, salt, 32, KDF);

  const cipher = createCipheriv('aes-256-gcm', key, iv);
  const ct = Buffer.concat([cipher.update(plaintext, 'utf8'), cipher.final()]);

  return Buffer.concat([salt, iv, cipher.getAuthTag(), ct]);
}

export function open(blob, masterKey) {
  const salt = blob.subarray(0, 32);
  const iv   = blob.subarray(32, 44);
  const tag  = blob.subarray(44, 60);
  const ct   = blob.subarray(60);

  const decipher = createDecipheriv('aes-256-gcm', scryptSync(masterKey, salt, 32, KDF), iv);
  decipher.setAuthTag(tag);
  return Buffer.concat([decipher.update(ct), decipher.final()]).toString('utf8');
}

The decipher.final() call is the one that matters. If the tag does not verify, it throws, and you get no plaintext at all. That is the difference between authenticated encryption and raw AES-CTR, where a flipped bit in storage silently becomes a flipped bit in the decrypted memory and your model happily reads corrupted context as fact. GCM makes tampering a hard error. The construction and its security bounds are specified in NIST SP 800-38D, which is also where the IV rules come from.

Why the per-ciphertext key is not paranoia

GCM with random 96-bit IVs has a birthday bound. Under a single key, the probability of an IV collision reaches roughly 2^-33 at around 2^32 encryptions, and an IV collision under the same key is catastrophic for GCM: it leaks the XOR of two plaintexts and, worse, allows forgery of the authentication key. Four billion memories is not an absurd number for a long-lived multi-tenant store.

Deriving a fresh key per ciphertext from a fresh 32-byte salt makes that bound irrelevant. Each key encrypts exactly one message, so the number of messages per key is one, and the collision analysis collapses. The cost is a scrypt call per read and per write. With N=16384, r=8, p=1 the memory footprint is about 128 * N * r bytes, so roughly 16 MiB per derivation, and on typical server hardware it lands in the tens of milliseconds. That is significant. It means you batch decryptions, you do not decrypt inside a loop that fans out per row without bounding concurrency, and you keep an eye on the event loop.

The part nobody puts in the launch post

Here is the uncomfortable structural fact about encrypting a memory store that also does semantic search. The memory text is ciphertext. The embedding vector next to it is not.

You cannot embed ciphertext. AES output is indistinguishable from random, so an embedding of it carries zero semantic signal, which means vector search over it returns nothing useful. The vector has to be computed from plaintext, before sealing, and then stored in a pgvector column in the clear so the ANN index can traverse it.

A 1024-dimensional float32 vector is not the plaintext, but it is a lossy projection of it, and lossy is not the same as safe. An attacker holding a stolen database dump without the master key can still do a great deal:

• Query-by-guess.Embed candidate strings with the same public embedding model and rank them against the stored vectors. High cosine similarity to "quarterly revenue projections for the acquisition" tells you what a row is about without ever decrypting it.
• Clustering.Group vectors and you recover the topic structure of a user's entire memory store, including how many distinct projects they have and which rows belong together.
• Inversion. Published work on embedding inversion reconstructs substantial portions of short input text from its vector alone, given access to the same encoder. Short texts, which is exactly what memory entries are, are the easy case.

So the honest threat-model statement is narrower than "encrypted at rest." It is: encryption at rest protects memory content against disk theft, backup exfiltration, and snapshot leakage. It does not make the row opaque, because the vector beside it is a semantic side channel, and it does nothing at all against a compromise of the application tier, which by construction holds the master key in memory. Anyone claiming otherwise is selling you the cipher and not the system.

Partial mitigations exist and all of them cost something. You can encrypt the vector too and accept that semantic search now requires decrypting the whole column per query, which is fine at ten thousand rows and hopeless at ten million. You can quantize aggressively (binary or int8) to reduce the information content of each stored vector, which degrades both the attack and your recall. You can shard the ANN index per tenant so a partial dump only leaks one tenant's topic structure. None of these makes the leak zero.

Rotation is a schema decision, not a crypto decision

The scheme above derives each row key directly from the master key. That is clean and it has one sharp edge: rotating the master key means re-encrypting every row. Decrypt with the old master, derive a new salt, re-seal with the new master. At 30 ms of scrypt per direction, one million rows is roughly 17 hours of pure key derivation before you count I/O. You can parallelize it, but you are still running a migration that reads and rewrites your entire content column, and you need a key-version column so that both masters are live during the rollout.

Envelope encryption avoids that. Generate a random 32-byte data encryption key per row, use it for the AES-GCM operation, and store that DEK wrapped under a key encryption key. Now rotation means unwrapping and re-wrapping a 32-byte value per row, which is two AES operations and no KDF, and the large ciphertext column is never touched. The trade is an extra column and one more indirection on every read.

The rule of thumb: derive-from-master is the right call when your rotation cadence is measured in years or when the operator controls the whole deployment and can schedule downtime. Envelope with wrapped DEKs is the right call the moment rotation becomes routine, compliance-driven, or per-tenant. Both need a version byte in the stored blob from day one, because retrofitting versioning onto an unversioned format is a much worse migration than either rotation.

Unimatrix self-hosters supply their own MASTER_ENCRYPTION_KEY, which means the operator, not us, decides the rotation policy and owns the consequences. The full breakdown of what the ciphertext covers and what it does not is on the security page, including the vector side channel, because a threat model you cannot read is not a threat model.