[enhancement] gem serve app should implement rubygems API

Dave's Avatar

Dave

12 Oct, 2011 01:49 PM

Ideally gem serve should boot an app that implements the Rubygems API. For example, it should be possible to push and yank gems.

Motivations:

  • make it easier to run local gem mirrors
  • make it easier to run private gem servers
  • allow per-site config and skins for gem servers

I am aware of the geminabox project, which is great so far as it goes. However, there are a number of drawbacks. It implements an entirely new gem command whereas it should integrate with the gem push/gem yank CLI. There are no tests or specs of any kind.

My suggestion is that the server code is decoupled from the main Rubygems project and refactored, and to reuse the API cucumber features from the rubygems.org project.

Any thoughts?

  1. Support Staff 2 Posted by Nick Quaranto on 12 Oct, 2011 01:53 PM

    Nick Quaranto's Avatar

    I agree, but this should be a separate project that both Gemcutter itself could use and could be installed separately as a gem. Talk is cheap though, show me the code :) If you want to bounce ideas, I'm in #rubygems on freenode.

  2. 3 Posted by Dave on 12 Oct, 2011 02:04 PM

    Dave's Avatar

    I'm up for it - and I agree it should be a separate upstream prj. Definitely an itch I need scratching :)

  3. Support Staff 4 Posted by Nick Quaranto on 12 Oct, 2011 02:56 PM

    Nick Quaranto's Avatar

    There's really two problems at work here:

    1) The original RG api (how everyone installs/lists/fetches gems)

    2) The new API (at /api/v1/*) and as we continue to the grow the site, that will grow too.

    Basically, I think it's worth it to extract both separately, and allow an application to hook into each api endpoint.

  4. 5 Posted by Dave on 21 Oct, 2011 02:19 PM

    Dave's Avatar

    TL;DR I've extracted a "core" rubygems API and a server that implements them

    As you say, talk is cheap so here is some code:

    I consider push/yank and the original RG api (indices, gemspecs, installs) as part of a core "rubygems-source" API. I mean it makes no sense to me to have gem server that you cannot push to or yank from. So I wrote a suite of cucumber features that covers that API: https://github.com/kapoq/rubygems-source-features.

    Originally I tried extracting existing features from the Rubygems.org suite, but they are too tied tangled into the website itself. Many scenarios are defined in terms of what a website user sees, not what an API user would see. Some of it is surely implementation dependent too, like vanity stats and not being able to push a gem that's been yanked already.

    Then I created a reference implementation: https://github.com/kapoq/rubygems-source. It's a teeny Sinatra app, but because of the dependency on Sinatra, I reckon there's no chance of it replacing the existing rubygems WEBrick server (at least until the stdlib is sorted http://www.ruby-forum.com/topic/216542).

    Finally I wrote some patches to gem commands. There are some bugs in the existing commands that hinder their use with non-Rubygems.org hosts e.g.:

    • gem yank does not take a --host option
    • gem yank and gem push try and sign you into Rubygems.org when no key is available, even if the --host is something else.

    In fact there are a number of little issues with those commands.

    It'd be great to have some input from the core team. I would like to merge in my patches but should make pull request on the gemcutter repo or rubygems? Would you accept patch to rubygems.org that integrates the new cucumber API features?

  5. 6 Posted by Dave on 21 Oct, 2011 02:21 PM

    Dave's Avatar

    TL;DR I've extracted a "core" rubygems API and a server that implements them

    As you say, talk is cheap so here is some code:

  6. 7 Posted by Dave on 21 Oct, 2011 02:22 PM

    Dave's Avatar

    I consider push/yank and the original RG api (indices, gemspecs, installs) as part of a core "rubygems-source" API. I mean it makes no sense to me to have gem server that you cannot push to or yank from. So I wrote a suite of cucumber features that covers that API: https://github.com/kapoq/rubygems-source-features.

    Originally I tried extracting existing features from the Rubygems.org suite, but they are too tied tangled into the website itself. Many scenarios are defined in terms of what a website user sees, not what an API user would see. Some of it is surely implementation dependent too, like vanity stats and not being able to push a gem that's been yanked already.

    Then I created a reference implementation: https://github.com/kapoq/rubygems-source. It's a teeny Sinatra app, but because of the dependency on Sinatra, I reckon there's no chance of it replacing the existing rubygems WEBrick server (at least until the stdlib is sorted http://www.ruby-forum.com/topic/216542).

    Finally I wrote some patches to gem commands. There are some bugs in the existing commands that hinder their use with non-Rubygems.org hosts e.g.:

    • gem yank does not take a --host option
    • gem yank and gem push try and sign you into Rubygems.org when no key is available, even if the --host is something else.

    In fact there are a number of little issues with those commands.

    It'd be great to have some input from the core team. I would like to merge in my patches but should make pull request on the gemcutter repo or rubygems? Would you accept patch to rubygems.org that integrates the new cucumber API features?

  7. Support Staff 8 Posted by Nick Quaranto on 21 Oct, 2011 03:30 PM

    Nick Quaranto's Avatar

    Hey Dave,

    Just catching up here. These projects are cool, I think we should talk this out on IRC or something. I understand the need to have the server separated from RubyGems itself, and I'm leaning towards it should just be broken out to a separate project.

    Also, another heads up, stickler has a huge head start up on this initiative: https://github.com/copiousfreetime/stickler You should definitely read through that code base (I need to reread it myself!)

    Basically my points are:

    • Yes, we need more of this.
    • We need one clean API that can be used to do all 3 current use cases: Local gem server, RubyGems.org, and internal gem server
    • The code as is I've seen in this thread is a long way off from doing this.

    I'm moving cities in a week so my OSS time is in a crunch, but I would love to work on this with you regardless. I'm in #rubygems on freenode, we can talk about the plan going forward and what can be done.

  8. 9 Posted by Dave on 21 Oct, 2011 03:57 PM

    Dave's Avatar

    Hey Nick, thanks for looping back. See you in #rubygems and good luck w/ the move.
    D

  9. Support Staff 10 Posted by Nick Quaranto on 21 Oct, 2011 05:15 PM

    Nick Quaranto's Avatar

    In the meantime, a good offline discussion place to have this would be the Gemcutter mailing list. More developers follow that and it would be better for feedback than this.

    In any case, I'm going to add this as a "Help Wanted" on http://contribute.rubygems.org (once I get around to adding that section..today!)

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac