metadata for all gems in YAML available anymore?

Kyle's Avatar

Kyle

25 Apr, 2010 03:56 PM

Correct me if I'm wrong, but it seems like the newer rubygems doesn't download detailed remote gem metadata for all available gems in one go anymore. It just fetches a list of gem names and versions, and grabs the full gem specs one at a time as needed. This, of course, makes sense for rubygems itself as it was getting obnoxiously slow with constant updating. I wrote my own custom package searching tool that I've been using to quickly find projects for whatever task at hand as the need comes up, which of course requires the more detailed gem specs including descriptions. It's beyond impractical and bandwidth-intensive to download 5000 gemspecs one at a time, of course. So I just wanted to ask if you still have a full set of metadata available on the site somewhere still? Thanks (and thanks for the work y'all have been doing on this site)

  1. Support Staff 1 Posted by Nick Quaranto on 25 Apr, 2010 05:06 PM

    Nick Quaranto's Avatar

    Yeah, right now the only index supported is the old Marshal index, at

    http://rubygems.org/Marshal.4.8.Z

    which is updated nightly. A little code sample of how to use it:

    chef % irb -rubygems -ropen-uri
    irb(main):003:0> gems = Marshal.load(Gem.inflate(open("http://rubygems.org/Marshal.4.8.Z").read)); nil
    => nil
    irb(main):004:0> gems.size
    => 55891
    irb(main):005:0> gems.first
    => ["cafepress-search-1.0.0", #<Gem::Specification:0x11fd0dc @licenses=[], @summary="A client library for the Cafepress search API that allows you to search for designs and products on Cafepress.com", @post_install_message=nil, @description="A client library for the Cafepress search API that allows you to search for\ndesigns and products on Cafepress.com", @dependencies=[<Gem::Dependency type=:runtime name="0.3.0" requirements=">= 0">, <Gem::Dependency type=:runtime name="happymapper" requirements=">= 0">, <Gem::Dependency type=:runtime name="hoe" requirements=">= 2.3.1">], @requirements=[], @bindir="bin", @require_paths=["lib"], @date=Thu Dec 17 00:00:00 -0500 2009, @name="cafepress-search", @new_platform="ruby", @loaded=false, @extensions=[], @authors=["Britt Crawford", "Dimple Joseph"], @cert_chain=[], @required_rubygems_version=#<Gem::Requirement:0x1200534 @requirements=[[">=", #<Gem::Version "0">]]>, @loaded_from=nil, @has_rdoc=true, @specification_version=3, @original_platform=nil, @version=#<Gem::Version "1.0.0">, @license=[], @extra_rdoc_files=[], @files=[], @signing_key=nil, @default_executable=nil, @email=nil, @test_files=[], @required_ruby_version=#<Gem::Requirement:0x1200624 @requirements=[[">=", #<Gem::Version "0">]]>, @rdoc_options=[], @rubygems_version="1.3.5", @homepage=nil, @executables=[], @platform="ruby", @autorequire=nil, @rubyforge_project=nil>]
    

    If there's ways we can extend our API to help with this task you have, please let me know...definitely one of the goals of Gemcutter going forward is building out the API to better support developers and the RubyGems ecosystem as a whole, I'm sure you won't be the only one who finds it useful. If the source for this tool of yours is available I'd love to check it out.

  2. 2 Posted by Kyle on 25 Apr, 2010 05:49 PM

    Kyle's Avatar

    Oh, thanks -- that's exactly what I was looking for!

    My little search tool is pretty slap-dash and quickly assembled -- I don't have it up online anywhere right now, though I suppose if you'd really like to see it I can copy and paste. Basically, I have a very simple SQLite3 database set up to house metadata for various ruby-oriented command line tools I use often like rubygems and cheat. I have a cron job that fetches gem info from this site and the other main gem sources -- chiefly github -- and stuffs it into a flat table every few days or so. I use SQLite's built in full text search feature to index it and then I have a very minimal command line script to search the database against name, description, and summary fields. Then the script formats it cleanly for console usage and it works really quickly since it uses the database instead of combing through a list of objects.

    With regard to your API, I'm not sure that I'd suggest anything specifically, as you have basic search functionality built in already, I believe? The reason I wrote my own little tool is because sometimes I'm working and some random need will come up. Earlier today I thought "what was that encoding detection package I had a while ago?" because I had some screwy text encoding issues while trying to fix tags in some mp3s. It's just quicker to search all my sources for the word "detection" and see that it was "chardet" than to open up a web browser. Also, perhaps it's changed in recent versions, but I think the gems program only queries names and versions, no?

    Thanks for the link

  3. Nick Quaranto closed this discussion on 17 May, 2010 02:18 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac