Fluxer PostgreSQL→Cassandra migration tool

Type-aware migration from Fluxer's PostgreSQL fluxer_kv backend to the
native Apache Cassandra schema, plus scylla-feature image builds and a
full cutover runbook.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -0,0 +1,113 @@
+# fluxer-pg-to-cassandra
+
+Migrate a self-hosted [Fluxer](https://github.com/fluxerapp/fluxer) instance from the
+**PostgreSQL** key-value backend to the native **Apache Cassandra** schema, and switch
+the running stack over to it.
+
+Fluxer stores everything in PostgreSQL as one `fluxer_kv` table of `__fluxer_type`-tagged
+JSON. Cassandra mode uses ~200 native CQL tables. This tool copies the data across,
+coercing each value to its destination column's real CQL type.
+
+> Cassandra vs ScyllaDB: Fluxer's Rust services use the `scylla` crate, which speaks the
+> Cassandra wire protocol. Everything here works against **Apache Cassandra** (what the
+> official instance runs) or ScyllaDB unchanged.
+
+## Prerequisites
+
+1. A running Cassandra node, with a keyspace and login role created:
+   ```sql
+   CREATE ROLE fluxer WITH PASSWORD = '…' AND LOGIN = true;
+   CREATE KEYSPACE fluxer WITH replication =
+     {'class': 'NetworkTopologyStrategy', '<your-dc>': 1} AND durable_writes = true;
+   GRANT ALL PERMISSIONS ON KEYSPACE fluxer TO fluxer;
+   ALTER KEYSPACE system_auth WITH replication =
+     {'class': 'NetworkTopologyStrategy', '<your-dc>': 1};
+   ```
+   The datacenter name (`dc=…` in `cassandra-rackdc.properties`) must match
+   `FLUXER_CASSANDRA_LOCAL_DC` / `SCYLLA_DC` everywhere. Mismatch → the driver throws
+   `localDataCenter was configured as 'datacenter1', but only found hosts in [<dc>]`.
+
+2. **The Cassandra schema applied.** Fluxer does *not* create it on api startup. Use the
+   `fluxer-dev` tool from a source checkout:
+   ```bash
+   docker run --rm -it --network host -v ~/fluxer-src:/app -w /app \
+     -e FLUXER_CASSANDRA_HOSTS=<host> -e FLUXER_CASSANDRA_PORT=9042 \
+     -e FLUXER_CASSANDRA_KEYSPACE=fluxer -e FLUXER_CASSANDRA_LOCAL_DC=<dc> \
+     -e FLUXER_CASSANDRA_USERNAME=fluxer -e FLUXER_CASSANDRA_PASSWORD='…' \
+     rust:1-bookworm cargo run --release -p fluxer-dev -- cassandra apply
+   ```
+   On a slow/low-RAM node the apply may hit a 30s driver timeout part-way; it's
+   idempotent (`IF NOT EXISTS`), so just re-run until `cassandra verify` is clean.
+
+3. Docker on a host that can reach **both** PostgreSQL and Cassandra.
+
+## Running the migration
+
+```bash
+cp .env.example migrate.env      # fill in PG + Cassandra creds (literal values, no quotes)
+
+# dry run — reads Postgres, writes nothing, reports per-table counts + any errors
+docker run --rm -it --network host -v "$PWD":/app -w /app --env-file migrate.env \
+  node:20 sh -c "npm install && npm run dry-run"
+
+# real run — same, with migrate
+docker run --rm -it --network host -v "$PWD":/app -w /app --env-file migrate.env \
+  node:20 sh -c "npm install && npm run migrate"
+```
+
+`--network host` lets the container reach LAN IPs directly. The env-file passes
+passwords literally, sidestepping URL-encoding and shell `!`/`$` expansion headaches.
+
+### How it handles types
+
+The generic JSON transform can't know a column's real CQL type, so the script reads
+`system_schema.columns` at startup and coerces per column (`coerceForColumn`):
+
+- **`varint`** ← bigint `Long` is rejected by the driver; sent as a string.
+- **`map<…>`** ← Postgres stores some maps as `[[k,v],…]` arrays; rebuilt into a `Map`.
+- **`timeuuid`** ← Postgres has no timeuuid (it stored a plain timestamp); reconstructed
+  with `TimeUuid.fromDate(date, 0, fixedNode, fixedClock)` — **deterministic**, so
+  re-runs upsert the same row instead of duplicating the clustering key.
+
+### Idempotency & the delete caveat
+
+Every run does a **full re-copy**; INSERTs upsert by primary key, so re-running is safe
+and converges on the source state. It syncs inserts and updates but **not deletes** — a
+row deleted in Postgres after a prior run stays in Cassandra. For a clean cutover, run
+the final pass during a maintenance window (Postgres quiet), or `TRUNCATE` the Cassandra
+tables first.
+
+## Cutover
+
+1. Build the scylla-enabled shard images (the public images are Postgres-only — the
+   Cassandra code is behind `#[cfg(feature = "scylla")]`):
+   ```bash
+   ./images/build-cassandra-images.sh v1 ~/fluxer-src
+   ```
+2. Final migration run (delta) during a maintenance window.
+3. In `docker-compose.yml`:
+   - `x-fluxer-env`: set `FLUXER_DATABASE_BACKEND: cassandra` and add the
+     `FLUXER_CASSANDRA_*` block (incl. `FLUXER_CASSANDRA_LOCAL_DC`).
+   - `messages-shard` / `users-shard`: point `image:` at
+     `fluxer-messages-cassandra:<tag>` / `fluxer-users-cassandra:<tag>`.
+4. `docker compose config -q` to validate, then `docker compose up -d`.
+
+Rollback is a one-word change: set `FLUXER_DATABASE_BACKEND: postgres` (leave the
+postgres vars in place) and restart.
+
+### Upgrades
+
+The custom shard images are pinned to the commit you built from — they do **not** update
+with `docker compose pull`. On every Fluxer upgrade, re-run
+`./images/build-cassandra-images.sh <new-tag> ~/fluxer-src` so they track the same
+version as the stock images and avoid protocol skew. (Ideal long-term fix: ask upstream
+to publish Cassandra-enabled image variants.)
+
+## Files
+
+| Path | Purpose |
+|---|---|
+| `migrate.mjs` | The migration script (type-aware PG→Cassandra copy). |
+| `.env.example` | Template for `migrate.env` (gitignored). |
+| `images/Dockerfile.{messages,users}.cassandra` | Stock Dockerfile + `--features scylla`. |
+| `images/build-cassandra-images.sh` | One-command pinned build of both shard images. |