Instant update mirroring

Instant Update Mirroring

The future tier'ed CPAN mirroring will use File::Rsync::Mirror::Recent (also known as rrr, version 0.1.1 or later required). It is also available from git.

This setup is just in testing for now; don't try it unless you want to help make sure it works!

You can use this for the full CPAN; no need to separately use rrr for the PAUSE directories. When we move this to "full production" the changes from PAUSE will come through with an extra delay of just 10-30 seconds.

How to install

curl --compressed -LO http://xrl.us/cpanm
chmod +x ./cpanm
./cpanm JSON File::Rsync::Mirror::Recent

Also check that you have rsync installed; version 3.x or newer is highly recommended.

Run the client

Use rrr-client:

   rrr-client --source cpan-rsync.perl.org::CPAN --target /home/mirrors/CPAN

It will take a while to get synced up (and use a good deal of CPU and momentarily up to 1GB memory), but after that it'll get the latest changes every 20 seconds with very little overhead. Basically just keep it running forever, and it'll get all updates.

What to test

The goal of this is to find and fix any bugs in the rrr tools and make sure that all updates to CPAN comes through to your mirror the same as the traditional "full" rsync.

If you want to test if all updates from the master have been fetched, you can run

   rsync -avn cpan-rsync.perl.org::CPAN /home/mirrors/CPAN

There should only be a few changes - files that have changed in the last minute. If there are more, run rsync again without the -n parameter to update your mirror. Please report any discrepancies!

In production it'll likely be recommended to run a full rsync every few days in case rrr missed something; but for the testing please don't do that. Maybe it won't be necessary at all.

rsync masters

• Public, no authentication required: cpan-rsync.perl.org
• For "Tier 1" mirrors, cpan-rsync-master.perl.org (the "master mirror")

Right now the two hostnames actually point to the same box, but when we're happy with the performance and reliability of the rrr tool, the plan is to make cpan-rsync.perl.org point to a group of instantly updated mirrors -- the "Tier 1" mirrors. With some clever DNS, we can even have clients of the tier 1 mirrors pull appropriately from a mirror in their country or on their continent.

If you wish to be a tier 1 mirror, please email ask@perl.org for a username/password for the master mirror for testing.