rsync mirrors?

smcclanahan's Avatar

smcclanahan

05 Apr, 2013 07:30 PM

I've read several blogs & threads about running gem mirror following by generate_index but even after provisioning a 16GB VM with huge swap & waiting 2 days I still cannot get generate_index to complete successfully on the entire rubygems mirror. Is possible that I could become a mirror partner with rsync? Do you have the gems & indices built with some private rsync mechanism as I've seen hinted at that I could become a partner of? I need to provide a mirror to offline networks & this is really painful.

Things like CPAN, pypy, nodejs, cywin, etc have convenient functional mirror capabilities & the best one to date is definitely rsync. It would be great if we could use rsync or get guidance on how the hell to actually build the index for these gems. Thanks.

  1. 1 Posted by smcclanahan on 09 Apr, 2013 08:14 PM

    smcclanahan's Avatar

    bump

  2. Support Staff 2 Posted by Nick Quaranto on 08 May, 2013 12:15 AM

    Nick Quaranto's Avatar

    Agreed. Check out this thread:

    http://help.rubygems.org/discussions/problems/5225-any-successful-i...

    We really just need people who are excited and passionate about making sure mirroring works to help make it work and stick around to help.

  3. 3 Posted by Jeffrey Thalham... on 21 Nov, 2014 09:42 PM

    Jeffrey Thalhammer's Avatar

    It appears that there has been some recent work on rubygems-mirror to simply mirror the index files rather than rebuilding them locally. But an rsync still seems like the most sensible solution.

    I've done a lot of work with the CPAN ecosystem, and I'm starting to poke around in Ruby-land. For CPAN, we have the rrr client which uses rsync under the hood.

    rrr uses a set of RECENT files on the server to figure out exactly which things need to be mirrored since a certain point in time, rather than letting rsync scan the entire filesystem every time. This greatly reduces the load on the upstream server (and clients too) and allows CPAN to support hundreds of mirrors that are synchronized in near real-time.

    Would something like that make sense for rubygems.org? I could certainly cook it up.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac