brendenandClaude Opus 4.8
abae7f9f41
Type-aware migration from Fluxer's PostgreSQL fluxer_kv backend to the native Apache Cassandra schema, plus scylla-feature image builds and a full cutover runbook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
114 lines
5.3 KiB
Markdown
114 lines
5.3 KiB
Markdown
# fluxer-pg-to-cassandra
|
|
|
|
Migrate a self-hosted [Fluxer](https://github.com/fluxerapp/fluxer) instance from the
|
|
**PostgreSQL** key-value backend to the native **Apache Cassandra** schema, and switch
|
|
the running stack over to it.
|
|
|
|
Fluxer stores everything in PostgreSQL as one `fluxer_kv` table of `__fluxer_type`-tagged
|
|
JSON. Cassandra mode uses ~200 native CQL tables. This tool copies the data across,
|
|
coercing each value to its destination column's real CQL type.
|
|
|
|
> Cassandra vs ScyllaDB: Fluxer's Rust services use the `scylla` crate, which speaks the
|
|
> Cassandra wire protocol. Everything here works against **Apache Cassandra** (what the
|
|
> official instance runs) or ScyllaDB unchanged.
|
|
|
|
## Prerequisites
|
|
|
|
1. A running Cassandra node, with a keyspace and login role created:
|
|
```sql
|
|
CREATE ROLE fluxer WITH PASSWORD = '…' AND LOGIN = true;
|
|
CREATE KEYSPACE fluxer WITH replication =
|
|
{'class': 'NetworkTopologyStrategy', '<your-dc>': 1} AND durable_writes = true;
|
|
GRANT ALL PERMISSIONS ON KEYSPACE fluxer TO fluxer;
|
|
ALTER KEYSPACE system_auth WITH replication =
|
|
{'class': 'NetworkTopologyStrategy', '<your-dc>': 1};
|
|
```
|
|
The datacenter name (`dc=…` in `cassandra-rackdc.properties`) must match
|
|
`FLUXER_CASSANDRA_LOCAL_DC` / `SCYLLA_DC` everywhere. Mismatch → the driver throws
|
|
`localDataCenter was configured as 'datacenter1', but only found hosts in [<dc>]`.
|
|
|
|
2. **The Cassandra schema applied.** Fluxer does *not* create it on api startup. Use the
|
|
`fluxer-dev` tool from a source checkout:
|
|
```bash
|
|
docker run --rm -it --network host -v ~/fluxer-src:/app -w /app \
|
|
-e FLUXER_CASSANDRA_HOSTS=<host> -e FLUXER_CASSANDRA_PORT=9042 \
|
|
-e FLUXER_CASSANDRA_KEYSPACE=fluxer -e FLUXER_CASSANDRA_LOCAL_DC=<dc> \
|
|
-e FLUXER_CASSANDRA_USERNAME=fluxer -e FLUXER_CASSANDRA_PASSWORD='…' \
|
|
rust:1-bookworm cargo run --release -p fluxer-dev -- cassandra apply
|
|
```
|
|
On a slow/low-RAM node the apply may hit a 30s driver timeout part-way; it's
|
|
idempotent (`IF NOT EXISTS`), so just re-run until `cassandra verify` is clean.
|
|
|
|
3. Docker on a host that can reach **both** PostgreSQL and Cassandra.
|
|
|
|
## Running the migration
|
|
|
|
```bash
|
|
cp .env.example migrate.env # fill in PG + Cassandra creds (literal values, no quotes)
|
|
|
|
# dry run — reads Postgres, writes nothing, reports per-table counts + any errors
|
|
docker run --rm -it --network host -v "$PWD":/app -w /app --env-file migrate.env \
|
|
node:20 sh -c "npm install && npm run dry-run"
|
|
|
|
# real run — same, with migrate
|
|
docker run --rm -it --network host -v "$PWD":/app -w /app --env-file migrate.env \
|
|
node:20 sh -c "npm install && npm run migrate"
|
|
```
|
|
|
|
`--network host` lets the container reach LAN IPs directly. The env-file passes
|
|
passwords literally, sidestepping URL-encoding and shell `!`/`$` expansion headaches.
|
|
|
|
### How it handles types
|
|
|
|
The generic JSON transform can't know a column's real CQL type, so the script reads
|
|
`system_schema.columns` at startup and coerces per column (`coerceForColumn`):
|
|
|
|
- **`varint`** ← bigint `Long` is rejected by the driver; sent as a string.
|
|
- **`map<…>`** ← Postgres stores some maps as `[[k,v],…]` arrays; rebuilt into a `Map`.
|
|
- **`timeuuid`** ← Postgres has no timeuuid (it stored a plain timestamp); reconstructed
|
|
with `TimeUuid.fromDate(date, 0, fixedNode, fixedClock)` — **deterministic**, so
|
|
re-runs upsert the same row instead of duplicating the clustering key.
|
|
|
|
### Idempotency & the delete caveat
|
|
|
|
Every run does a **full re-copy**; INSERTs upsert by primary key, so re-running is safe
|
|
and converges on the source state. It syncs inserts and updates but **not deletes** — a
|
|
row deleted in Postgres after a prior run stays in Cassandra. For a clean cutover, run
|
|
the final pass during a maintenance window (Postgres quiet), or `TRUNCATE` the Cassandra
|
|
tables first.
|
|
|
|
## Cutover
|
|
|
|
1. Build the scylla-enabled shard images (the public images are Postgres-only — the
|
|
Cassandra code is behind `#[cfg(feature = "scylla")]`):
|
|
```bash
|
|
./images/build-cassandra-images.sh v1 ~/fluxer-src
|
|
```
|
|
2. Final migration run (delta) during a maintenance window.
|
|
3. In `docker-compose.yml`:
|
|
- `x-fluxer-env`: set `FLUXER_DATABASE_BACKEND: cassandra` and add the
|
|
`FLUXER_CASSANDRA_*` block (incl. `FLUXER_CASSANDRA_LOCAL_DC`).
|
|
- `messages-shard` / `users-shard`: point `image:` at
|
|
`fluxer-messages-cassandra:<tag>` / `fluxer-users-cassandra:<tag>`.
|
|
4. `docker compose config -q` to validate, then `docker compose up -d`.
|
|
|
|
Rollback is a one-word change: set `FLUXER_DATABASE_BACKEND: postgres` (leave the
|
|
postgres vars in place) and restart.
|
|
|
|
### Upgrades
|
|
|
|
The custom shard images are pinned to the commit you built from — they do **not** update
|
|
with `docker compose pull`. On every Fluxer upgrade, re-run
|
|
`./images/build-cassandra-images.sh <new-tag> ~/fluxer-src` so they track the same
|
|
version as the stock images and avoid protocol skew. (Ideal long-term fix: ask upstream
|
|
to publish Cassandra-enabled image variants.)
|
|
|
|
## Files
|
|
|
|
| Path | Purpose |
|
|
|---|---|
|
|
| `migrate.mjs` | The migration script (type-aware PG→Cassandra copy). |
|
|
| `.env.example` | Template for `migrate.env` (gitignored). |
|
|
| `images/Dockerfile.{messages,users}.cassandra` | Stock Dockerfile + `--features scylla`. |
|
|
| `images/build-cassandra-images.sh` | One-command pinned build of both shard images. |
|