This help site has been deprecated. Please send your requests to

rsync mirrors?

smcclanahan's Avatar


05 Apr, 2013 07:30 PM

I've read several blogs & threads about running gem mirror following by generate_index but even after provisioning a 16GB VM with huge swap & waiting 2 days I still cannot get generate_index to complete successfully on the entire rubygems mirror. Is possible that I could become a mirror partner with rsync? Do you have the gems & indices built with some private rsync mechanism as I've seen hinted at that I could become a partner of? I need to provide a mirror to offline networks & this is really painful.

Things like CPAN, pypy, nodejs, cywin, etc have convenient functional mirror capabilities & the best one to date is definitely rsync. It would be great if we could use rsync or get guidance on how the hell to actually build the index for these gems. Thanks.

  1. 1 Posted by smcclanahan on 09 Apr, 2013 08:14 PM

    smcclanahan's Avatar


  2. Support Staff 2 Posted by Nick Quaranto on 08 May, 2013 12:15 AM

    Nick Quaranto's Avatar

    Agreed. Check out this thread:

    We really just need people who are excited and passionate about making sure mirroring works to help make it work and stick around to help.

  3. 3 Posted by Jeffrey Thalham... on 21 Nov, 2014 09:42 PM

    Jeffrey Thalhammer's Avatar

    It appears that there has been some recent work on rubygems-mirror to simply mirror the index files rather than rebuilding them locally. But an rsync still seems like the most sensible solution.

    I've done a lot of work with the CPAN ecosystem, and I'm starting to poke around in Ruby-land. For CPAN, we have the rrr client which uses rsync under the hood.

    rrr uses a set of RECENT files on the server to figure out exactly which things need to be mirrored since a certain point in time, rather than letting rsync scan the entire filesystem every time. This greatly reduces the load on the upstream server (and clients too) and allows CPAN to support hundreds of mirrors that are synchronized in near real-time.

    Would something like that make sense for I could certainly cook it up.

  4. 4 Posted by Steve Snodgrass on 06 Jan, 2015 10:22 PM

    Steve Snodgrass's Avatar

    I'm wishing for a working solution to this as well, as generate_index is not behaving well for me either. Trying to figure out the best way to handle this.

Discussions are closed to public comments.
If you need help with please start a new discussion.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac