Fluxer PostgreSQL→Cassandra migration tool

Type-aware migration from Fluxer's PostgreSQL fluxer_kv backend to the
native Apache Cassandra schema, plus scylla-feature image builds and a
full cutover runbook.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-23 11:47:06 -04:00
co-authored by Claude Opus 4.8
commit abae7f9f41
8 changed files with 565 additions and 0 deletions
+21
View File
@@ -0,0 +1,21 @@
# Copy to `migrate.env` and fill in. Values are read literally by `docker run --env-file`
# (no quotes, no shell expansion) — paste passwords with special chars as-is.
# ── Source: PostgreSQL (the existing Fluxer fluxer_kv backend) ────────────────
PG_HOST=10.1.10.36
PG_PORT=5432
PG_USER=fluxer
PG_PASSWORD=CHANGE_ME
PG_DATABASE=fluxer
# ── Destination: Apache Cassandra ────────────────────────────────────────────
SCYLLA_HOSTS=10.1.10.38:9042
SCYLLA_DC=atl
CASSANDRA_USERNAME=fluxer
CASSANDRA_PASSWORD=CHANGE_ME
# ── Tuning (optional) ────────────────────────────────────────────────────────
# KV_TABLE=fluxer_kv
# CASSANDRA_KEYSPACE=fluxer
# BATCH_SIZE=200
# CONCURRENCY=16
+11
View File
@@ -0,0 +1,11 @@
# Secrets — never commit. Holds DB passwords in plaintext.
.env
*.env
!.env.example
# Node
node_modules/
npm-debug.log*
# Migration scratch
last-diff.cql
+113
View File
@@ -0,0 +1,113 @@
# fluxer-pg-to-cassandra
Migrate a self-hosted [Fluxer](https://github.com/fluxerapp/fluxer) instance from the
**PostgreSQL** key-value backend to the native **Apache Cassandra** schema, and switch
the running stack over to it.
Fluxer stores everything in PostgreSQL as one `fluxer_kv` table of `__fluxer_type`-tagged
JSON. Cassandra mode uses ~200 native CQL tables. This tool copies the data across,
coercing each value to its destination column's real CQL type.
> Cassandra vs ScyllaDB: Fluxer's Rust services use the `scylla` crate, which speaks the
> Cassandra wire protocol. Everything here works against **Apache Cassandra** (what the
> official instance runs) or ScyllaDB unchanged.
## Prerequisites
1. A running Cassandra node, with a keyspace and login role created:
```sql
CREATE ROLE fluxer WITH PASSWORD = '…' AND LOGIN = true;
CREATE KEYSPACE fluxer WITH replication =
{'class': 'NetworkTopologyStrategy', '<your-dc>': 1} AND durable_writes = true;
GRANT ALL PERMISSIONS ON KEYSPACE fluxer TO fluxer;
ALTER KEYSPACE system_auth WITH replication =
{'class': 'NetworkTopologyStrategy', '<your-dc>': 1};
```
The datacenter name (`dc=…` in `cassandra-rackdc.properties`) must match
`FLUXER_CASSANDRA_LOCAL_DC` / `SCYLLA_DC` everywhere. Mismatch → the driver throws
`localDataCenter was configured as 'datacenter1', but only found hosts in [<dc>]`.
2. **The Cassandra schema applied.** Fluxer does *not* create it on api startup. Use the
`fluxer-dev` tool from a source checkout:
```bash
docker run --rm -it --network host -v ~/fluxer-src:/app -w /app \
-e FLUXER_CASSANDRA_HOSTS=<host> -e FLUXER_CASSANDRA_PORT=9042 \
-e FLUXER_CASSANDRA_KEYSPACE=fluxer -e FLUXER_CASSANDRA_LOCAL_DC=<dc> \
-e FLUXER_CASSANDRA_USERNAME=fluxer -e FLUXER_CASSANDRA_PASSWORD='…' \
rust:1-bookworm cargo run --release -p fluxer-dev -- cassandra apply
```
On a slow/low-RAM node the apply may hit a 30s driver timeout part-way; it's
idempotent (`IF NOT EXISTS`), so just re-run until `cassandra verify` is clean.
3. Docker on a host that can reach **both** PostgreSQL and Cassandra.
## Running the migration
```bash
cp .env.example migrate.env # fill in PG + Cassandra creds (literal values, no quotes)
# dry run — reads Postgres, writes nothing, reports per-table counts + any errors
docker run --rm -it --network host -v "$PWD":/app -w /app --env-file migrate.env \
node:20 sh -c "npm install && npm run dry-run"
# real run — same, with migrate
docker run --rm -it --network host -v "$PWD":/app -w /app --env-file migrate.env \
node:20 sh -c "npm install && npm run migrate"
```
`--network host` lets the container reach LAN IPs directly. The env-file passes
passwords literally, sidestepping URL-encoding and shell `!`/`$` expansion headaches.
### How it handles types
The generic JSON transform can't know a column's real CQL type, so the script reads
`system_schema.columns` at startup and coerces per column (`coerceForColumn`):
- **`varint`** ← bigint `Long` is rejected by the driver; sent as a string.
- **`map<…>`** ← Postgres stores some maps as `[[k,v],…]` arrays; rebuilt into a `Map`.
- **`timeuuid`** ← Postgres has no timeuuid (it stored a plain timestamp); reconstructed
with `TimeUuid.fromDate(date, 0, fixedNode, fixedClock)` — **deterministic**, so
re-runs upsert the same row instead of duplicating the clustering key.
### Idempotency & the delete caveat
Every run does a **full re-copy**; INSERTs upsert by primary key, so re-running is safe
and converges on the source state. It syncs inserts and updates but **not deletes** — a
row deleted in Postgres after a prior run stays in Cassandra. For a clean cutover, run
the final pass during a maintenance window (Postgres quiet), or `TRUNCATE` the Cassandra
tables first.
## Cutover
1. Build the scylla-enabled shard images (the public images are Postgres-only — the
Cassandra code is behind `#[cfg(feature = "scylla")]`):
```bash
./images/build-cassandra-images.sh v1 ~/fluxer-src
```
2. Final migration run (delta) during a maintenance window.
3. In `docker-compose.yml`:
- `x-fluxer-env`: set `FLUXER_DATABASE_BACKEND: cassandra` and add the
`FLUXER_CASSANDRA_*` block (incl. `FLUXER_CASSANDRA_LOCAL_DC`).
- `messages-shard` / `users-shard`: point `image:` at
`fluxer-messages-cassandra:<tag>` / `fluxer-users-cassandra:<tag>`.
4. `docker compose config -q` to validate, then `docker compose up -d`.
Rollback is a one-word change: set `FLUXER_DATABASE_BACKEND: postgres` (leave the
postgres vars in place) and restart.
### Upgrades
The custom shard images are pinned to the commit you built from — they do **not** update
with `docker compose pull`. On every Fluxer upgrade, re-run
`./images/build-cassandra-images.sh <new-tag> ~/fluxer-src` so they track the same
version as the stock images and avoid protocol skew. (Ideal long-term fix: ask upstream
to publish Cassandra-enabled image variants.)
## Files
| Path | Purpose |
|---|---|
| `migrate.mjs` | The migration script (type-aware PG→Cassandra copy). |
| `.env.example` | Template for `migrate.env` (gitignored). |
| `images/Dockerfile.{messages,users}.cassandra` | Stock Dockerfile + `--features scylla`. |
| `images/build-cassandra-images.sh` | One-command pinned build of both shard images. |
+30
View File
@@ -0,0 +1,30 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Identical to Dockerfile but compiled with --features scylla for Apache Cassandra support.
# The feature is named "scylla" after the Rust crate — the runtime target is Cassandra.
FROM rust:1-bookworm AS builder
WORKDIR /usr/src/app
COPY . .
RUN cargo build --release -p fluxer-messages --features scylla
FROM debian:bookworm-slim
ARG BUILD_VERSION=""
WORKDIR /usr/local/bin
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && \
rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/src/app/target/release/fluxer-messages /usr/local/bin/fluxer-messages
ENV BUILD_VERSION="${BUILD_VERSION}"
USER 65532:65532
EXPOSE 8090
CMD ["/usr/local/bin/fluxer-messages"]
+30
View File
@@ -0,0 +1,30 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Identical to Dockerfile but compiled with --features scylla for Apache Cassandra support.
# The feature is named "scylla" after the Rust crate — the runtime target is Cassandra.
FROM rust:1-bookworm AS builder
WORKDIR /usr/src/app
COPY . .
RUN cargo build --release -p fluxer-users --features scylla
FROM debian:bookworm-slim
ARG BUILD_VERSION=""
WORKDIR /usr/local/bin
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && \
rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/src/app/target/release/fluxer-users /usr/local/bin/fluxer-users
ENV BUILD_VERSION="${BUILD_VERSION}"
USER 65532:65532
EXPOSE 8090
CMD ["/usr/local/bin/fluxer-users"]
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Build the two Fluxer Rust shard services WITH the `scylla` feature, which the
# public ghcr.io images are compiled WITHOUT. Tag them to match your stack's
# FLUXER_IMAGE_TAG so the custom images never drift out of sync on upgrades.
#
# Usage:
# ./build-cassandra-images.sh <git-tag> [path-to-fluxer-source]
# Example:
# ./build-cassandra-images.sh v1 ~/fluxer-src
set -euo pipefail
TAG="${1:?usage: build-cassandra-images.sh <git-tag> [source-dir]}"
SRC="${2:-$HOME/fluxer-src}"
HERE="$(cd "$(dirname "$0")" && pwd)"
if [[ ! -f "$SRC/Cargo.toml" ]]; then
echo "error: $SRC is not a Fluxer source checkout (no Cargo.toml)." >&2
echo "Clone it first: git clone https://github.com/fluxerapp/fluxer.git $SRC" >&2
exit 1
fi
echo ">> checking out $TAG in $SRC"
git -C "$SRC" fetch --tags --quiet
git -C "$SRC" checkout "$TAG"
# Drop the scylla-enabled Dockerfiles into the source tree (build context = repo root).
cp "$HERE/Dockerfile.messages.cassandra" "$SRC/fluxer_messages/Dockerfile.cassandra"
cp "$HERE/Dockerfile.users.cassandra" "$SRC/fluxer_users/Dockerfile.cassandra"
echo ">> building fluxer-messages-cassandra:$TAG"
docker build -f "$SRC/fluxer_messages/Dockerfile.cassandra" -t "fluxer-messages-cassandra:$TAG" "$SRC"
echo ">> building fluxer-users-cassandra:$TAG"
docker build -f "$SRC/fluxer_users/Dockerfile.cassandra" -t "fluxer-users-cassandra:$TAG" "$SRC"
echo ">> done. images:"
docker images | grep -E "fluxer-(messages|users)-cassandra" | grep "$TAG"
+308
View File
@@ -0,0 +1,308 @@
#!/usr/bin/env node
/**
* Fluxer: PostgreSQL KV table → ScyllaDB/Cassandra migration
*
* Prerequisites:
* 1. ScyllaDB is running and the 'fluxer' keyspace exists
* 2. fluxer_api has started once against ScyllaDB so all tables are created
* 3. PostgreSQL is still up and serving the original data
*
* Usage (prefer an --env-file with docker; values are taken literally):
* PG_HOST=10.1.10.36 PG_USER=fluxer PG_PASSWORD=... PG_DATABASE=fluxer \
* SCYLLA_HOSTS=10.1.10.38:9042 SCYLLA_DC=atl \
* CASSANDRA_USERNAME=fluxer CASSANDRA_PASSWORD=... \
* npm run dry-run # verify without writing
* npm run migrate # real migration
*/
import pg from 'pg';
import cassandra from 'cassandra-driver';
const { Client: PgClient } = pg;
const { types: cassTypes, auth: cassAuth } = cassandra;
// ── Configuration ──────────────────────────────────────────────────────────
// PostgreSQL connection comes from PG_HOST/PG_PORT/PG_USER/PG_PASSWORD/PG_DATABASE (see main()).
const SCYLLA_HOSTS = (process.env.SCYLLA_HOSTS ?? '10.1.10.X:9042').split(','); // replace X
const SCYLLA_DC = process.env.SCYLLA_DC ?? 'datacenter1';
const KEYSPACE = process.env.CASSANDRA_KEYSPACE ?? 'fluxer';
const KV_TABLE = process.env.KV_TABLE ?? 'fluxer_kv';
const BATCH_SIZE = parseInt(process.env.BATCH_SIZE ?? '200');
const CONCURRENCY = parseInt(process.env.CONCURRENCY ?? '16');
// Fluxer snowflake epoch and message bucket window (10 days in ms)
const BUCKET_DURATION_MS = 864_000_000n;
function messageBucket(messageIdStr) {
return (BigInt(messageIdStr) >> 22n) / BUCKET_DURATION_MS;
}
// ── Column-type introspection ───────────────────────────────────────────────
// The generic transform can't know a destination column's real CQL type (e.g.
// varint vs bigint, or map<> vs a plain array), so we read the live schema from
// system_schema.columns and coerce each value to match its target column.
let colTypes = {}; // { tableName: { columnName: cqlType } }
// Fixed node (6 bytes) + clock (2 bytes) so timeuuids reconstructed from a stored
// timestamp are DETERMINISTIC — re-running upserts the same row instead of creating
// a duplicate. Postgres' KV row_key already guarantees a unique (pk, event_id)
// timestamp per partition, so ticks=0 won't collide within a partition.
const TUUID_NODE = Buffer.from('fluxer'); // exactly 6 bytes
const TUUID_CLOCK = Buffer.from('kv'); // exactly 2 bytes
async function loadColumnTypes(cassClient, keyspace) {
const res = await cassClient.execute(
`SELECT table_name, column_name, type FROM system_schema.columns WHERE keyspace_name = ?`,
[keyspace], { prepare: true }
);
const map = {};
for (const r of res.rows) (map[r.table_name] ??= {})[r.column_name] = r.type;
return map;
}
// Coerce a transformed value to satisfy its destination CQL column type.
function coerceForColumn(value, cqlType) {
if (value == null || !cqlType) return value;
// varint: the driver rejects Long (that's bigint); it accepts a string.
if (cqlType === 'varint') return value?.toString?.() ?? String(value);
// map<...>: source may give an array of [k,v] pairs or a plain object → Map.
if (cqlType.startsWith('map<')) {
if (value instanceof Map) return value;
if (Array.isArray(value)) return new Map(value);
if (typeof value === 'object') return new Map(Object.entries(value));
return value;
}
// timeuuid: Postgres stored these as a plain timestamp (the KV model has no
// timeuuid). Reconstruct a time-ordered UUID from the date so clustering order
// is preserved; a string value is already a uuid and passes through.
if (cqlType === 'timeuuid') {
return value instanceof Date
? cassTypes.TimeUuid.fromDate(value, 0, TUUID_NODE, TUUID_CLOCK)
: value;
}
// uuid: expect a real uuid string already.
if (cqlType === 'uuid') return value;
return value;
}
// ── Type transformation ─────────────────────────────────────────────────────
// Fluxer's PostgreSQL KV layer encodes non-JSON-native types with a
// __fluxer_type tag. Map them to the types cassandra-driver expects.
function transformValue(v) {
if (v === null || v === undefined) return v;
if (Array.isArray(v)) return v.map(transformValue);
if (typeof v === 'object') {
if ('__fluxer_type' in v) {
switch (v.__fluxer_type) {
case 'bigint':
// Snowflake IDs are 64-bit — use Long to avoid precision loss
return cassTypes.Long.fromString(String(v.value));
case 'date':
return new Date(v.value);
case 'buffer':
return Buffer.from(v.value, 'base64');
case 'set':
return Array.isArray(v.value) ? v.value.map(transformValue) : [];
case 'map': {
const out = {};
for (const [k, val] of Object.entries(v.value ?? {})) {
out[String(transformValue(k))] = transformValue(val);
}
return out;
}
default:
return v.value ?? null;
}
}
// Plain nested object — recurse
const out = {};
for (const [k, val] of Object.entries(v)) {
out[k] = transformValue(val);
}
return out;
}
return v;
}
// ── Concurrency helper ──────────────────────────────────────────────────────
async function runConcurrently(items, limit, fn) {
for (let i = 0; i < items.length; i += limit) {
const results = await Promise.allSettled(items.slice(i, i + limit).map(fn));
const failed = results.filter(r => r.status === 'rejected');
if (failed.length > 0) throw failed[0].reason;
}
}
// ── Per-table migration ─────────────────────────────────────────────────────
async function migrateTable(pgClient, cassClient, tableName, dryRun) {
const { rows: [{ count }] } = await pgClient.query(
`SELECT COUNT(*) FROM ${KV_TABLE} WHERE table_name = $1`,
[tableName]
);
const total = parseInt(count);
if (total === 0) {
console.log(`[${tableName}] empty, skipping`);
return { migrated: 0, errors: 0 };
}
const seenBuckets = new Set();
let migrated = 0;
let errors = 0;
process.stdout.write(`[${tableName}] 0/${total}`);
for (let offset = 0; offset < total; offset += BATCH_SIZE) {
const { rows } = await pgClient.query(
`SELECT row_data, expires_at
FROM ${KV_TABLE}
WHERE table_name = $1
ORDER BY row_key
LIMIT $2 OFFSET $3`,
[tableName, BATCH_SIZE, offset]
);
await runConcurrently(rows, CONCURRENCY, async (row) => {
try {
const data = transformValue(row.row_data);
// Messages: derive the bucket from the snowflake message_id and also
// populate channel_message_buckets so the Rust shard can list buckets.
if (tableName === 'messages' && data.message_id != null) {
const bucket = messageBucket(String(data.message_id));
data.bucket = cassTypes.Long.fromString(String(bucket));
const key = `${data.channel_id}:${bucket}`;
if (!seenBuckets.has(key)) {
seenBuckets.add(key);
if (!dryRun) {
await cassClient.execute(
`INSERT INTO ${KEYSPACE}.channel_message_buckets (channel_id, bucket) VALUES (?, ?)`,
[data.channel_id, data.bucket],
{ prepare: true }
).catch(() => {}); // idempotent
}
}
}
const cols = Object.keys(data);
const types = colTypes[tableName] ?? {};
const vals = cols.map(c => coerceForColumn(data[c], types[c]));
const phs = cols.map(() => '?').join(', ');
const ttlSec = row.expires_at
? Math.max(1, Math.floor((new Date(row.expires_at).getTime() - Date.now()) / 1000))
: null;
let query = `INSERT INTO ${KEYSPACE}.${tableName} (${cols.join(', ')}) VALUES (${phs})`;
if (ttlSec) query += ` USING TTL ${ttlSec}`;
if (!dryRun) {
await cassClient.execute(query, vals, { prepare: true });
}
migrated++;
} catch (err) {
errors++;
// Log first few errors per table to surface schema mismatches
if (errors <= 3) console.error(`\n [${tableName}] row error: ${err.message}`);
}
});
process.stdout.write(`\r[${tableName}] ${migrated}/${total} (${Math.round(migrated / total * 100)}%)`);
}
const errStr = errors > 0 ? `${errors} errors` : '';
console.log(`${errStr}`);
return { migrated, errors };
}
// Migrate in dependency order; messages last (needs bucket table populated)
const PRIORITY_ORDER = [
'users', 'guilds', 'roles', 'channels', 'guild_members',
'auth_sessions', 'read_states', 'guild_bans', 'emojis', 'stickers',
'attachments', 'webhooks', 'invites',
];
// ── Main ────────────────────────────────────────────────────────────────────
async function main() {
const dryRun = process.argv.includes('--dry-run');
if (dryRun) console.log('=== DRY RUN — nothing will be written to ScyllaDB ===\n');
// PostgreSQL — component-based config so passwords with URL-special chars
// (@, #, %, etc.) don't need percent-encoding; node-postgres handles them raw.
const pgClient = new PgClient({
host: process.env.PG_HOST ?? '10.1.10.36',
port: parseInt(process.env.PG_PORT ?? '5432'),
user: process.env.PG_USER ?? 'fluxer',
password: process.env.PG_PASSWORD ?? '',
database: process.env.PG_DATABASE ?? 'fluxer',
});
await pgClient.connect();
console.log(`PostgreSQL ✓ ${process.env.PG_USER ?? 'fluxer'}@${process.env.PG_HOST ?? '10.1.10.36'}:${process.env.PG_PORT ?? '5432'}/${process.env.PG_DATABASE ?? 'fluxer'}`);
// ScyllaDB
const authProvider = process.env.CASSANDRA_USERNAME
? new cassAuth.PlainTextAuthProvider(
process.env.CASSANDRA_USERNAME,
process.env.CASSANDRA_PASSWORD ?? ''
)
: undefined;
const cassClient = new cassandra.Client({
contactPoints: SCYLLA_HOSTS,
localDataCenter: SCYLLA_DC,
keyspace: KEYSPACE,
authProvider,
socketOptions: { connectTimeout: 30_000 },
queryOptions: { consistency: cassTypes.consistencies.localQuorum },
});
await cassClient.connect();
colTypes = await loadColumnTypes(cassClient, KEYSPACE);
console.log(`ScyllaDB ✓ ${SCYLLA_HOSTS.join(', ')} / keyspace=${KEYSPACE} / ${Object.keys(colTypes).length} tables introspected\n`);
// Discover tables
const { rows: tableRows } = await pgClient.query(
`SELECT table_name, COUNT(*) AS cnt
FROM ${KV_TABLE}
GROUP BY table_name
ORDER BY table_name`
);
const allTables = tableRows.map(r => r.table_name);
const ordered = [
...PRIORITY_ORDER.filter(t => allTables.includes(t)),
...allTables.filter(t => !PRIORITY_ORDER.includes(t) && t !== 'messages'),
...(allTables.includes('messages') ? ['messages'] : []),
];
const width = Math.max(...tableRows.map(r => r.table_name.length));
console.log('Tables to migrate:');
for (const r of tableRows) {
console.log(` ${r.table_name.padEnd(width)} ${r.cnt} rows`);
}
console.log();
let totalMigrated = 0;
let totalErrors = 0;
for (const tableName of ordered) {
try {
const { migrated, errors } = await migrateTable(pgClient, cassClient, tableName, dryRun);
totalMigrated += migrated;
totalErrors += errors;
} catch (err) {
console.error(`\n[${tableName}] FAILED: ${err.message}`);
console.error(' Ensure fluxer_api has run once with FLUXER_DATABASE_BACKEND=cassandra');
console.error(' to auto-create the Cassandra schema before running this script.\n');
totalErrors++;
}
}
await pgClient.end();
await cassClient.shutdown();
console.log('\n' + '─'.repeat(55));
console.log(`Total migrated: ${totalMigrated} rows | Errors: ${totalErrors}`);
if (dryRun) console.log('(dry run — no data was written)');
if (totalErrors > 0) process.exit(1);
}
main().catch(err => { console.error(err); process.exit(1); });
+15
View File
@@ -0,0 +1,15 @@
{
"name": "fluxer-pg-to-cassandra",
"version": "1.0.0",
"description": "Migrate a self-hosted Fluxer instance from the PostgreSQL KV backend to the native Apache Cassandra schema.",
"type": "module",
"license": "AGPL-3.0-or-later",
"scripts": {
"dry-run": "node migrate.mjs --dry-run",
"migrate": "node migrate.mjs"
},
"dependencies": {
"cassandra-driver": "^4.7.2",
"pg": "^8.13.3"
}
}