Any successful internal mirroring solutions out there for rubygems.org?

cwalquist's Avatar

cwalquist

25 Apr, 2013 09:27 PM

Hi,

We have been maintaining an internal gem mirror for awhile for security reasons (using our own script, not rubygems-mirror, see http://help.rubygems.org/discussions/problems/592-rubygems-mirror-p... for details), and recently became aware of the fact that it's been failing due to (a) bad gems that "gem generate_index" refuses to process, and (b) running out of memory, perhaps because the gem repo has doubled in size in the past year or so.

Unfortunately, rubygems.org apparently doesn't eat its own dog food (see end of
http://help.rubygems.org/discussions/problems/810-create-rubygems-m...), so has no incentive to maintain a rubygems-mirror or a generate_index that would maintain a viable internal mirror.

We are therefore looking to create and update our own mirror (on a larger-memory machine) through the script posted in this thread (http://help.rubygems.org/discussions/problems/592-rubygems-mirror-p...), then create a "bad gems" blacklist by running "gem generate_index" on each new gem before adding it to our internal mirror, and finally run a "gem generate_index" on the cleaned gem list. The whole process takes around 110 minutes.

Does anyone have a better way for creating and keeping an internal mirror up-to-date? Does rubygems-mirror now have an active maintainer we could work with? Any proven approaches that improve on the above-described procedure?

Thanks,
-chris

  1. 2 Posted by kent on 06 May, 2013 04:36 PM

    kent's Avatar

    (N.B. I work with @cwalquist)

    We've brute-forced a solution to this. We no longer try to generate our own quick gemspecs or indices - we download them from the mirror as well.

    We start with mirroring the newest gems via the script Chris referenced here: http://help.rubygems.org/discussions/problems/592-rubygems-mirror-p... (the $MIRRORSCRIPT in the following gist) and simply download the $GEMspec.rz and spec.4.8.Z files based on the new gems downloaded.

    https://gist.github.com/kcowgill/5526236

  2. Support Staff 3 Posted by Nick Quaranto on 07 May, 2013 11:51 PM

    Nick Quaranto's Avatar

    I heard good things about this at Railsconf...maybe it will work for you too?

    https://github.com/yoshiori/rubygems-mirror-command

  3. 4 Posted by cwalquist on 08 May, 2013 04:00 PM

    cwalquist's Avatar

    Interesting, thanks Nick. Given that the github project is 8 days old and we already have something that works, we'll go with what we have for now. As Kent mentioned, we abandoned generate_index because we found that even after filtering out bad gems as best we could so that generate_index would run to completion, the resulting index was still missing large numbers of gems.

    If our internal gem repo indexes (which *are* still OK with gem generate_index) begin suffering, we may be forced to revisit. I am hoping we'll be OK though, since we have more control over the quality of gems that go into our internal repo, and more traceability as well.

    Ultimately, we'd like to see generate_index do the same thing that rubygems.org does, whatever that is.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

29 Aug, 2014 06:23 PM
28 Aug, 2014 07:18 PM
27 Aug, 2014 12:03 PM
26 Aug, 2014 07:09 PM
26 Aug, 2014 07:07 PM