Any successful internal mirroring solutions out there for

cwalquist's Avatar


25 Apr, 2013 09:27 PM


We have been maintaining an internal gem mirror for awhile for security reasons (using our own script, not rubygems-mirror, see for details), and recently became aware of the fact that it's been failing due to (a) bad gems that "gem generate_index" refuses to process, and (b) running out of memory, perhaps because the gem repo has doubled in size in the past year or so.

Unfortunately, apparently doesn't eat its own dog food (see end of, so has no incentive to maintain a rubygems-mirror or a generate_index that would maintain a viable internal mirror.

We are therefore looking to create and update our own mirror (on a larger-memory machine) through the script posted in this thread (, then create a "bad gems" blacklist by running "gem generate_index" on each new gem before adding it to our internal mirror, and finally run a "gem generate_index" on the cleaned gem list. The whole process takes around 110 minutes.

Does anyone have a better way for creating and keeping an internal mirror up-to-date? Does rubygems-mirror now have an active maintainer we could work with? Any proven approaches that improve on the above-described procedure?


  1. 1 Posted by kent on 06 May, 2013 04:36 PM

    kent's Avatar

    (N.B. I work with @cwalquist)

    We've brute-forced a solution to this. We no longer try to generate our own quick gemspecs or indices - we download them from the mirror as well.

    We start with mirroring the newest gems via the script Chris referenced here: (the $MIRRORSCRIPT in the following gist) and simply download the $GEMspec.rz and spec.4.8.Z files based on the new gems downloaded.

  2. Support Staff 2 Posted by Nick Quaranto on 07 May, 2013 11:51 PM

    Nick Quaranto's Avatar

    I heard good things about this at Railsconf...maybe it will work for you too?

  3. 3 Posted by cwalquist on 08 May, 2013 04:00 PM

    cwalquist's Avatar

    Interesting, thanks Nick. Given that the github project is 8 days old and we already have something that works, we'll go with what we have for now. As Kent mentioned, we abandoned generate_index because we found that even after filtering out bad gems as best we could so that generate_index would run to completion, the resulting index was still missing large numbers of gems.

    If our internal gem repo indexes (which *are* still OK with gem generate_index) begin suffering, we may be forced to revisit. I am hoping we'll be OK though, since we have more control over the quality of gems that go into our internal repo, and more traceability as well.

    Ultimately, we'd like to see generate_index do the same thing that does, whatever that is.

  4. 4 Posted by mkeisler on 09 Apr, 2015 04:02 PM

    mkeisler's Avatar

    How is your solution today, cwalquist? I'm looking at doing the same thing.

  5. 5 Posted by cwalquist on 09 Apr, 2015 07:38 PM

    cwalquist's Avatar

    We haven't touched anything in the past couple of years. The internal index occasionally corrupts (seems correlated to removing a gem, but not entirely sure), but so far a full re-index sets it right. Nick, do you have news of work on the indexing machinery? We're still running it with Ruby 1.8.7, kind of fearful of touching it :-)

  6. Support Staff 6 Posted by indirect on 09 Apr, 2015 08:35 PM

    indirect's Avatar

    The Bundler team has a complete mirroring (well, actually, pass-through caching) solution on our road map. Would either of you be willing to work with us on that project? If so, drop us an email at [email blocked].

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:


Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac