Rubygems mirror package is chronically missing gems, please fix

Chris Walquist's Avatar

Chris Walquist

12 May, 2011 03:19 PM

Hi,

The rubygems mirror package is often missing gems published on rubygems.org. Our organization mirrors rubygems internally for security reasons, and has had to spend significant time building "missing-gems" band-aid scripts to work around these gaps.

The latest issue is with rack-mount 0.7.1 and 0.6.14, which are missing from the mirror package.

Lately, the developers have begun asking for a self-service utility to poke gems in themselves as they discover holes.

Will this be fixed in the next few days, or should we go ahead and build a remedial utility internally?

Thanks,
-chris

  1. 2 Posted by cwalquist on 12 May, 2011 04:38 PM

    cwalquist's Avatar
  2. Support Staff 3 Posted by Nick Quaranto on 27 May, 2011 12:13 AM

    Nick Quaranto's Avatar

    Hi,

    Please use rubygems-mirror instead and not the legacy mirroring option...this repo supports the modern indexes and is much faster:

    https://github.com/rubygems/rubygems-mirror

  3. 4 Posted by cwalquist on 27 May, 2011 04:20 PM

    cwalquist's Avatar

    Thanks Nick, if this pans out it will transform our enterprise users' interaction with the ruby gems repo. They are getting quite frustrated waiting the 2 to 3 hours for a reindex to happen.

  4. 5 Posted by cwalquist on 27 May, 2011 04:21 PM

    cwalquist's Avatar

    Just to confirm...does this also fix the issue with gems randomly missing from the mirror?

  5. 6 Posted by cwalquist on 27 May, 2011 04:34 PM

    cwalquist's Avatar

    Hi, I'm reading the github README, and this comment makes me a little nervous:

    "Right now the API is not completely stable (it will change several times before release), however, I will maintain stability in master."

    What does this mean, exactly?

    We currently do this to update our internal mirror:

    $ gem --backtrace mirror --verbose $ gem --backtrace generate_index --no-legacy --modern --verbose -d /data/rubygems.mirror

    Will this interface remain the same?

    Thanks,
    -chris

  6. Support Staff 7 Posted by Nick Quaranto on 12 Jun, 2011 04:50 PM

    Nick Quaranto's Avatar

    I've added James (raggi) to this conversation, who wrote rubygems-mirror.

    James, maybe we should just ship a 1.0.0 of rubygems-mirror?

  7. 8 Posted by cwalquist on 14 Jun, 2011 04:19 PM

    cwalquist's Avatar

    Does "gem list --remote --all" produce a comprehensive list of gems from rubygems.org? If so, I'm thinking we can implement our own incremental mirroring, and bypass rubygems-mirror completely.

  8. 9 Posted by cwalquist on 14 Jun, 2011 09:52 PM

    cwalquist's Avatar

    FYI...Based on the output of a "gem list --all --remote" and breaking it out into its indicated gems, the mirror created by "gem mirror" is missing 19,301 gems, compared to the contents of http://rubygems.org/

    -chris

  9. Support Staff 10 Posted by Eric Hodel on 14 Jun, 2011 10:13 PM

    Eric Hodel's Avatar

    Instead of implementing your own internal mirroring can we give you access to rubygems-mirror to fix it? Or can you fix it and send a pull request?

  10. 11 Posted by Chris Walquist on 15 Jun, 2011 01:59 AM

    Chris Walquist's Avatar

    Hi Eric,

    Great idea. Sure, I'll have a try at figuring out what's going on, although I'm not an accomplished rubyist. Meantime, I'm still going to proceed with an internal mirror (I got a simple one pretty much working today), because we have to get something going that's reliable and efficient; it's becoming a critical piece of our enterprise development services as more of our developers do Ruby, and the last six months of wrestling with gem mirror have been painful for my support team. You may well ask, then why didn't you fix it six months ago, but we're just learning Ruby ourselves. I will try to feed back into rubygems-mirror if and where it makes sense.

    Thanks,
    -chris

  11. 12 Posted by cwalquist on 15 Jun, 2011 08:14 PM

    cwalquist's Avatar

    Interesting...I tried moving to Rubyg 1.9.2-p0 (from 1.8.6) and updating to Rubygems 1.8.5 (from 1.3.7), and now "gem list --remote --all" finds 307 gems that do not actually resolve to downloadable packages (that I can tell). Almost all of them end in '-universal-java-1.6.gem'. For instance:

    xml_to_hash-0.0.1-universal-java-1.6.gem

  12. 13 Posted by cwalquist on 17 Jun, 2011 09:21 PM

    cwalquist's Avatar

    For what it's worth, here is my mirror script. After it runs, I do a generate_index with --update, and all seems well.

    
    require 'fileutils'
    
    # Update internal gem mirror.
    $gem_repo_path = "/data/rubygems.mirror/gems"
    $gem_url = 'http://production.s3.rubygems.org'
    
    def fetch_gems_for gem_entry
      r = /(?\S+) \((?.*)\)/.match  gem_entry
      r[:versions_n_platforms].split(", ").each do |ver_n_platforms|
        version, *platforms = ver_n_platforms.split(" ")
        retrieve_gem r[:gem_name], version, platforms
      end
    end
    
    def retrieve_gem name, version, platforms=['ruby']
      platforms.each do |p|
        wget_unless_exists fq_gem_name(name, version, p)   # fq = "fully qualified"
      end
    end
    
    def wget_unless_exists fqname
      FileUtils.cd $gem_repo_path do
        return if File.exists?(fqname)
        puts `wget #{$gem_url}/gems/#{fqname}`
      end
    end
    
    def fq_gem_name name, version, platform=nil
      platform_string = (platform=='ruby' || platform.nil?) ? "" : "-#{platform}"
      "#{name}-#{version}#{platform_string}.gem"
    end
    
    ### Main Logic...
    
    `gem list ${ARGV[0]} --remote --all --source #{$gem_url}/`.lines.each do |gem_entry|
      fetch_gems_for  gem_entry
    end
    
  13. 14 Posted by cwalquist on 24 Jun, 2011 10:00 PM

    cwalquist's Avatar

    Bug fix: Replace this...

    
    def retrieve_gem name, version, platforms=['ruby']
    

    ...with this...

    
    def retrieve_gem name, version, platforms
       platforms = ['ruby'] if platforms.empty? || platforms.nil?
    
  14. Support Staff 15 Posted by Ryan Davis on 22 Jul, 2011 12:33 AM

    Ryan Davis's Avatar

    Please, if you can work this into the rubygems-mirror command directly that'd be great. Right now there is no active maintainer for it.

  15. 16 Posted by cwalquist on 01 Aug, 2011 09:07 PM

    cwalquist's Avatar

    Hi Ryan,

    To be honest, with the little Ruby I have, I'm loath to touch such a fundamental piece of the Ruby ecosystem without some deep collaboration with at least one pillar of the Ruby community. If I knew someone like that, and that someone was willing to step forward and mentor, I can approach my boss about donating some of my bandwidth. Any ideas for mentoring candidates?

  16. Support Staff 17 Posted by Eric Hodel on 02 Aug, 2011 12:58 AM

    Eric Hodel's Avatar

    There's usually someone around in irc://chat.freenode.net/#rubygems to answer questions about RubyGems so if you start down this path there will be at least two or three mentors to help you out.

  17. 18 Posted by cwalquist on 11 Aug, 2011 05:01 PM

    cwalquist's Avatar

    Fyi, another bug-fix...one of our dev guys pointed out that "--pre" pulls down release-candidate gems. This adds another 12,000 or so gems to the mirror.

    gem list ${ARGV[0]} --pre --remote etc...

    -chris

    P.S. Chatted on rubygems, but no response.

  18. 19 Posted by jsvd on 30 Aug, 2011 09:48 AM

    jsvd's Avatar

    Doing gem mirror now does not yield rails gems > 3.0.3 in our mirror, is this happening to anyone else?

  19. Support Staff 20 Posted by Nick Quaranto on 23 Sep, 2011 02:46 PM

    Nick Quaranto's Avatar

    jsvd: are you using rubygems-mirror?

    https://github.com/rubygems/rubygems-mirror

  20. Support Staff 21 Posted by James Tucker on 02 Oct, 2011 12:15 AM

    James Tucker's Avatar

    I have released rubygems-mirror as a gem now.

    I have tested it against the next release of rubygems.

    It may/may not work against current releases, depending on your previously installed versions of rubygems and some other details. It WILL work with the next rubygems release.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

29 Aug, 2014 06:23 PM
28 Aug, 2014 07:18 PM
27 Aug, 2014 12:03 PM
26 Aug, 2014 07:09 PM
26 Aug, 2014 07:07 PM