Redis downgrade migration

[leonchen83]

Redis Downgrade Migration

1. Source of the Issue

Related issues and discussions in Redis:

• Downgrade support in Redis
• How to restore redis 3.0.6 with redis 4.0's dump.rdb
• Sync data between KeyDB v6 - Redis v7
• How to migrate RDB to new format supported by Redis 7?

2. Common Solutions for Downgrade Migration

Solution 1: Conversion to AOF Migration, Pseudocode as Follows

let kvs = parse(RDB)
for (key, val) in kvs
    if val is map 
        for (field, value) in val
            // execute migration
            redis.HSET(key, field, vlaue)
    if val is string
        // execute migration
        redis.SET(key, val)
    ...
    ...

Advantages of this solution:

1. Simple implementation
2. Can handle large key downgrade migration

Disadvantages of this solution:

1. Uses non-compact AOF protocol for migration, consuming bandwidth
2. Cannot guarantee atomicity of migration, needs to ensure either successful or failed migration

Solution 2: Conversion to Lower Version DUMP Format, Pseudocode as Follows

let kvs = parse(RDB)
for (key, val) in kvs
    let dump-val = convertToDump(val)
    let lower-dump-val = convertToLowerDump(dump-val)
    // execute migration
    redis.RESTORE(key, lower-dump-val)

Advantages of this solution:

1. Achieves atomic migration (critical in most migration tools)
2. Compact migration format, reducing bandwidth usage

Disadvantages of this solution:

1. Complex implementation
2. Requires ensuring sufficient memory in migration tool for large key migration and adjusting parameters of source and target Redis instances

Note: Both Redis-replicator and Redis-rdb-CLI adopt Solution 2 for migration.

3. How Redis-replicator Performs Downgrade Migration

Code Example:

// Downgrade migration from redis-7.0.0 to redis-6.0.0
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379");
replicator.setRdbVisitor(new DumpRdbVisitor(replicator, 9));
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof DumpKeyValuePair) {
            DumpKeyValuePair dkv = (DumpKeyValuePair) event;
            byte[] serialized = dkv.getValue();
            // Use redis RESTORE command to migrate serialized data to target redis.
        }
    }
});
replicator.open();

Explanation:

// Create a DumpRdbVisitor Rdb parser to parse rdbv10 format into rdbv9 format
replicator.setRdbVisitor(new DumpRdbVisitor(replicator, 9));

// Register event listener, where serialized is the byte array downgraded to rdbv9
// It can be directly migrated to target database using the RESTORE command
replicator.addEventListener(new EventListener() {
    @Override
    public void onEvent(Replicator replicator, Event event) {
        if (event instanceof DumpKeyValuePair) {
            DumpKeyValuePair dkv = (DumpKeyValuePair) event;
            byte[] serialized = dkv.getValue();
        }
    }
});

4. How Redis-rdb-CLI Performs Downgrade Migration

# Redis-rdb-CLI internally relies on Redis-replicator
# The first step of migration requires changing the configuration file /path/to/redis-rdb-cli/conf/redis-rdb-cli.conf
# Change dump_rdb_version from -1 to 9
$ sed -i 's/dump_rdb_version=-1/dump_rdb_version=9/g' /redis-rdb-cli/conf/redis-rdb-cli.conf

# Execute migration command, data will be automatically downgraded to lower version DUMP format
$ rmt -s redis://com.redis7:6379 -m redis://com.redis6:6379 -r

5. Known Issues and Limitations

Cannot downgrade migrate data types not supported by lower versions of Redis. For example, cannot migrate MODULE format to versions below Redis 4.0 or STREAM format to versions below Redis 5.0. No other limitations known.