RubyGems mirror extremely slow?
Kent Cowgill
Is it just me?
In the past, our mirroring of rubygems.org would take at most a couple of hours. Since the recent Marshal.4.8.Z corruption trouble, it's been exceedingly slow. For example, last night we kicked off a mirror at 2:00am, and 13 hours later it's still running. It's still on the "Fetching 81,590 gems" part - and only gotten through 55,096 of them.
Is the server just being hammered having been effectively out of commission for a while?
Are there performance issues on rubygems.org?
Support Staff 2 Posted by James Tucker on 15 Oct, 2010 10:22 PM
Hi,
I've started replacing this code with an outboard gem plugin so that it can pull in dependencies (in particular net/http/persistent). I still have a little way to go until it's in a state that I want to release it (I have a bit of test coverage to catch up on, as I did some naughty things during the authoring process), and I want to abstract it out so that it can be a library suitable for use by gauntlet.
All that being said, it works, today, and will keep a gem mirror up to date, and will do a full mirror in a couple of hours (tested on servers with 100mb pipes, but also took about the same amount of time on a home ADSL iirc).
You just need to clone the repo, and run
rake mirror:update:http://github.com/rubygems/rubygems-mirror/blob/master/Rakefile#L23-32
HTH