RubyGems mirror extremely slow?

Kent Cowgill's Avatar

Kent Cowgill

15 Oct, 2010 08:08 PM

Is it just me?

In the past, our mirroring of rubygems.org would take at most a couple of hours. Since the recent Marshal.4.8.Z corruption trouble, it's been exceedingly slow. For example, last night we kicked off a mirror at 2:00am, and 13 hours later it's still running. It's still on the "Fetching 81,590 gems" part - and only gotten through 55,096 of them.

Is the server just being hammered having been effectively out of commission for a while?

Are there performance issues on rubygems.org?

  1. 1 Posted by James Tucker on 15 Oct, 2010 10:22 PM

    James Tucker's Avatar

    Hi,

    I've started replacing this code with an outboard gem plugin so that it can pull in dependencies (in particular net/http/persistent). I still have a little way to go until it's in a state that I want to release it (I have a bit of test coverage to catch up on, as I did some naughty things during the authoring process), and I want to abstract it out so that it can be a library suitable for use by gauntlet.

    All that being said, it works, today, and will keep a gem mirror up to date, and will do a full mirror in a couple of hours (tested on servers with 100mb pipes, but also took about the same amount of time on a home ADSL iirc).

    You just need to clone the repo, and run rake mirror:update:

    http://github.com/rubygems/rubygems-mirror/blob/master/Rakefile#L23-32

    HTH

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac