Host bb-m.rubygems.org refuse to serve some gems for aws sydney

stas.kraev's Avatar

stas.kraev

28 Apr, 2014 04:31 AM

Hello,

I'm trying to install ruby-ldap to the ec2 instance in sydney region. Gem returns me

ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    too many connection resets (http://bb-m.rubygems.org/gems/ruby-ldap-0.9.16.gem)
    /usr/lib/ruby/1.9.1/rubygems/remote_fetcher.rb:462:in `rescue in request'
...............

when i'm tring to fetch gem with wget it stalls

root@ip-10-240-53-122:/home/ubuntu# wget http://bb-m.rubygems.org/gems/ruby-ldap-0.9.16.gem
--2014-04-28 04:27:33--  http://bb-m.rubygems.org/gems/ruby-ldap-0.9.16.gem
Resolving bb-m.rubygems.org (bb-m.rubygems.org)... 199.91.171.93
Connecting to bb-m.rubygems.org (bb-m.rubygems.org)|199.91.171.93|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65024 (64K)
Saving to: ‘ruby-ldap-0.9.16.gem.1’

 0% [                                                                                                       ] 0           --.-K/s

But it works fine from my office in New Zealand.

root@b2b752305baa:/# wget http://bb-m.rubygems.org/gems/ruby-ldap-0.9.16.gem
--2014-04-28 04:27:46--  http://bb-m.rubygems.org/gems/ruby-ldap-0.9.16.gem
Resolving bb-m.rubygems.org (bb-m.rubygems.org)... 199.91.171.93
Connecting to bb-m.rubygems.org (bb-m.rubygems.org)|199.91.171.93|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65024 (64K)
Saving to: 'ruby-ldap-0.9.16.gem'

100%[=====================================================================================================>] 65,024       130KB/s   in 0.5s   

2014-04-28 04:27:48 (130 KB/s) - 'ruby-ldap-0.9.16.gem' saved [65024/65024]
  1. 2 Posted by Tim Moore on 28 Apr, 2014 05:43 AM

    Tim Moore's Avatar

    I'm having the same problem, also from the Sydney EC2 region.

    $ wget http://bb-m.rubygems.org/gems/newrelic_rpm-3.8.0.218.gem
    --2014-04-28 15:28:32--  http://bb-m.rubygems.org/gems/newrelic_rpm-3.8.0.218.gem
    Resolving bb-m.rubygems.org (bb-m.rubygems.org)... 199.91.171.93
    Connecting to bb-m.rubygems.org (bb-m.rubygems.org)|199.91.171.93|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 551424 (538K)
    Saving to: `newrelic_rpm-3.8.0.218.gem'
    
     0% [                                                                                                  ] 0           --.-K/s              ^
    

    No data ever comes back. Switching the URL to https also hangs.

    It works fine from my office workstation in Melbourne.

  2. 3 Posted by stas.kraev on 28 Apr, 2014 05:55 AM

    stas.kraev's Avatar

    Just go a success in Sydney.

    UPD. nope, still failing on other gems ...

  3. 4 Posted by Ben Bruscella on 28 Apr, 2014 11:46 PM

    Ben Bruscella's Avatar

    For the record, I'm seeing the same issue today

  4. 5 Posted by Kunal Parikh on 29 Apr, 2014 12:49 AM

    Kunal Parikh's Avatar

    +1 We are seeing this issue from some of our hosts in AWS VPC as well.

    Only for host: bb-m.rubygems.org.

    Here's the curl log: https://gist.github.com/techthumb/0521a03d3af808271da8

  5. 6 Posted by Trevor on 29 Apr, 2014 01:06 AM

    Trevor's Avatar

    +1

    Our deploys are failing in Sydney and I've traced it to bb-m.rubygems.org issues as well.
    Simplified, trying this command fails in Sydney, passes elsewhere (for example):

    wget http://bb-m.rubygems.org/gems/rake-10.1.1.gem

    It does an initial http connect, and gets a 200 OK, but then no data is received. The exact same wget works in a Singapore instance. Packet traces on both regions shows it connecting to the same server (mirror01.c45627.blueboxgrid.com)

  6. 7 Posted by Josh Goodall on 29 Apr, 2014 01:28 AM

    Josh Goodall's Avatar

    Confirming same symptom from all our AWS Sydney hosts, looks like a hang in the SSL handshake to bb-m.rubygems.org.

    If I managed that host, first thing I'd check is that it isn't doing reverse DNS lookup on clients.

  7. 8 Posted by James Holmes on 29 Apr, 2014 04:14 AM

    James Holmes's Avatar

    I can confirm the same observation. All our builds fail on bundle when deploying to AWS.

  8. 9 Posted by Joshua Mesilane on 29 Apr, 2014 04:23 AM

    Joshua Mesilane's Avatar

    Same problem here. This is breaking our CI systems. Please resolve ASAP

  9. 10 Posted by Joshua Mesilane on 29 Apr, 2014 04:31 AM

    Joshua Mesilane's Avatar

    FYI - For those experiencing this problem. Add the following to your hosts file:

    103.7.164.82 bb-m.rubygems.org

    It's not a fantastic solution, but it'll get you up and running for now.

  10. 11 Posted by stas.kraev on 29 Apr, 2014 04:39 AM

    stas.kraev's Avatar

    That's not very fair to demand anything ASAP from this guys - they are working for free.

    Anyone looking for immediate fix can try my solution - start instance in singapoure with geminabox running as a proxy. Simplest way is

    gem install geminabox
    sudo mkdir /mnt/gems
    sudo chown <username> /mnt/gems
    cat >config.ru<<EOF
    require "rubygems"
    require "geminabox"
    
    Geminabox.rubygems_proxy = true
    Geminabox.allow_remote_failure = true
    Geminabox.data = "/mnt/gems" # ... or wherever
    run Geminabox::Server
    EOF
    rackup
    

    and then redirect your gem to this server.

    /opt/chef/embedded/bin/gem sources -a http://publichostname:9292/
    
  11. 12 Posted by Joshua Mesilane on 29 Apr, 2014 04:52 AM

    Joshua Mesilane's Avatar

    Yep, that works, if you have easy control over your server at that stage.

    What about all of the automatically provisioned servers out there that are using puppet/chef/

    At the end of the day if you're providing a well-known, reputable, public service then it's not all that unreasonable to expect it to work, especially when it's known to be relied upon for multiple production systems all over the world.

    I know if it was my service I'd be pretty embarrassed - and to point the DNS at another server isn't exactly difficult and would mitigate the problem until the broader problem was fixed.

  12. 13 Posted by Luke Chadwick on 29 Apr, 2014 05:36 AM

    Luke Chadwick's Avatar

    After doing some troubleshooting with Evan it looks like the problem is somewhere on this side of the pacific. https://bb-m.rubygems.org works fine from my linodes.

    Evan is removing https://bb-m.rubygems.org from the list of mirrors temporarily.

  13. 14 Posted by Joshua Mesilane on 29 Apr, 2014 05:39 AM

    Joshua Mesilane's Avatar

    Hey Luke,

    Thanks for the prompt response.

    Cheers
    Josh

  14. 15 Posted by Ryan on 29 Apr, 2014 05:42 AM

    Ryan's Avatar

    +1

    Also on ec2 sydney. 2 hosts effected.

    Joshua, thanks for the host file tip, that appears to be working.

  15. 16 Posted by Luke Robins on 29 Apr, 2014 06:26 AM

    Luke Robins's Avatar

    Thanks Luke and Evan.
    Am getting gems on servers where had been getting failures as above before.

  16. Nick Quaranto closed this discussion on 16 Jun, 2014 01:49 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac