RubyGems mirror extremely slow?

Kent Cowgill's Avatar

Kent Cowgill

15 Oct, 2010 08:08 PM

Is it just me?

In the past, our mirroring of rubygems.org would take at most a couple of hours. Since the recent Marshal.4.8.Z corruption trouble, it's been exceedingly slow. For example, last night we kicked off a mirror at 2:00am, and 13 hours later it's still running. It's still on the "Fetching 81,590 gems" part - and only gotten through 55,096 of them.

Is the server just being hammered having been effectively out of commission for a while?

Are there performance issues on rubygems.org?

  1. Support Staff 1 Posted by James Tucker on 15 Oct, 2010 10:22 PM

    James Tucker's Avatar

    Hi,

    I've started replacing this code with an outboard gem plugin so that it can pull in dependencies (in particular net/http/persistent). I still have a little way to go until it's in a state that I want to release it (I have a bit of test coverage to catch up on, as I did some naughty things during the authoring process), and I want to abstract it out so that it can be a library suitable for use by gauntlet.

    All that being said, it works, today, and will keep a gem mirror up to date, and will do a full mirror in a couple of hours (tested on servers with 100mb pipes, but also took about the same amount of time on a home ADSL iirc).

    You just need to clone the repo, and run rake mirror:update:

    http://github.com/rubygems/rubygems-mirror/blob/master/Rakefile#L23-32

    HTH

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac