Sporadic SSL errors

Jon Leighton's Avatar

Jon Leighton

18 Oct, 2013 10:46 AM

Hi there,

I often get sporadic SSL errors when bundling. I can *not* reliably reproduce the error, so I don't think it simply a configuration issue on my end. Sometimes bundling works. Often it doesn't. When it doesn't, I typically get halfway through bundling when it exits with:

Could not verify the SSL certificate for https://rubygems.org/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without
using SSL, edit your Gemfile sources and change 'https' to 'http'.

I tried running "openssl s_client -connect rubygems.org:443" a few times with the following result:

https://gist.github.com/jonleighton/7039769

Notice that it works the first two times, and then fails on the third.

Is it possible that this could be some sort of load balancing issue on rubygems.org?

Thanks for your time.

Jon

PS. I also asked on Twitter and it seems that others may have similar issues: https://twitter.com/jonleighton/status/390903921528410112 (but note that I'm using MRI, not JRuby)

  1. 1 Posted by Jon Leighton on 18 Oct, 2013 10:51 AM

    Jon Leighton's Avatar

    Sometimes I also get:

    OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server hello A: wrong version numbe

  2. 2 Posted by Endoze on 21 Oct, 2013 01:32 AM

    Endoze's Avatar

    I seem to be experiencing both issues almost reliably. System is Ubuntu 12.04, ruby version is 2.0.0p247, and RVM version is 1.23.10.

  3. 3 Posted by Matt on 21 Oct, 2013 12:49 PM

    Matt's Avatar

    I'm also running 1.23.10 and I'm having the exact same issue pulling gems for rubies 1.9.3 and 2.0.0. Sometimes it works, sometimes it gets halfway through the Gemfile and throws that SSL error and sometimes it throws it right after "Fetching source index from https://rubygems.org/".

  4. 4 Posted by Jon Leighton on 21 Oct, 2013 12:55 PM

    Jon Leighton's Avatar

    Matt are you on Linux too? I run Fedora on my dev machine. So I'm wondering if that's a common element.

  5. 5 Posted by Matt on 21 Oct, 2013 02:09 PM

    Matt's Avatar

    Ah yes, I forgot to add that I'm on Xubuntu 12.10 or 13.04, after a good nice update && upgrade. I gotta check my update history in apt, maybe something there is causing this inconvenience.

  6. 6 Posted by Fabian on 21 Oct, 2013 04:43 PM

    Fabian's Avatar

    I had the same problem, but updating `rubygems` solved the problem for me. It may be connected to the following: https://github.com/rubygems/rubygems/blob/e0f36770d0df08ad02beb0474a341aa3a4378f1f/History.txt#L94

  7. 7 Posted by Jon Leighton on 21 Oct, 2013 05:20 PM

    Jon Leighton's Avatar

    I'm using Rubygems 2.1.9 and still seeing the problem.

  8. 8 Posted by marko on 22 Oct, 2013 08:27 AM

    marko's Avatar

    We've been dealing with this on Semaphore too, with the following setup:

    • Ubuntu 12.04 with latest base packages
    • RubyGems 2.1.9
    • any Ruby it seems

    Upgrading RubyGems is one step but if you read what people who've really solved it on Mac OS with RVM did, the second step is to somehow update the root certificates on the system.

    The problem on Linux (somehow only now exposed) seems to be that Ruby's OpenSSL wrapper is looking at the wrong place for root certificates. On Ubuntu these are provided by the ca-certificates package. See eg this discussion on SO.

    So our current solution is latest RubyGems + setting the SSL_CERT_FILE environment variable. Still need to wait a bit for lack of error reports from users though, because it is not possible to reproduce the error reliably.

  9. 9 Posted by Jon Leighton on 22 Oct, 2013 11:45 AM

    Jon Leighton's Avatar

    I don't think it's a certificates issue on my system. Everything looks up to date and ok on that front as far as I can tell.

    I edited by lib/bundler/fetcher.rb to turn on debugging on the net-http-persistent connection. This is what a failure looks like: https://gist.github.com/jonleighton/7099162

    I'm not sure what to make of that, but it seems possible it might be a bug in net-http-persistent to do with the retrying functionality.

  10. 10 Posted by hakan.ensari on 22 Oct, 2013 04:41 PM

    hakan.ensari's Avatar

    Same issue on a Mac. Sporadic, seemingly random SSL errors.

  11. 11 Posted by marko on 23 Oct, 2013 10:35 AM

    marko's Avatar

    @Jon, how is your Ruby installed (rvm, rbenv, package, manual compilation)?

  12. 12 Posted by Jon Leighton on 23 Oct, 2013 11:14 AM

    Jon Leighton's Avatar

    rbenv

  13. 13 Posted by dev on 23 Oct, 2013 08:15 PM

    dev's Avatar

    OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server hello A: wrong version number

    I'm experiencing this on my Bamboo build server. It's irritating to be getting constant build failures when running tests or deploying to heroku, where the problem also seems to manifest, but much rarer.

    Bamboo takes 3 or 4 tries to successfully bundle, heroku fails maybe 1 out of 6 deploys.

    I don't know off the top of my head what custom version of linux Bamboo runs on, but this occurs with whatever my previous version was, and my newly updated rubygems, no rbenv or rvm.

    Heroku has their own stack, too.

    Interestingly this hasn't happened once on my Mac with rbenv, locally, old version of rubygems, on Lion nor Mavericks, rubies MRI 1.9, 2.0, 2.1, or rbx 2.0.

  14. 14 Posted by Nathan Youngman on 01 Nov, 2013 08:29 PM

    Nathan Youngman's Avatar

    Also seeing the SSL certificate error a lot here. Mac OS X 10.9 Mavericks with Ruby 2.0 installed with chruby.

  15. 15 Posted by Tonci Damjanic on 03 Nov, 2013 01:56 PM

    Tonci Damjanic's Avatar

    I'm also receiving the same error after running bundle update on Windows (RubyGems v2.1.10).

    Error trace:

    Gem::RemoteFetcher::FetchError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.global.ssl.fastly.net/gems/aws-sdk-1.24.0.gem)
    An error occurred while installing aws-sdk (1.24.0), and Bundler cannot continue.
    Make sure that `gem install aws-sdk -v '1.24.0'` succeeds before bundling.
    

    Environment dump:

    RubyGems Environment:
      - RUBYGEMS VERSION: 2.1.10
      - RUBY VERSION: 2.0.0 (2013-06-27 patchlevel 247) [i386-mingw32]
      - INSTALLATION DIRECTORY: C:/Ruby/Ruby200/lib/ruby/gems/2.0.0
      - RUBY EXECUTABLE: C:/Ruby/Ruby200/bin/ruby.exe
      - EXECUTABLE DIRECTORY: C:/Ruby/Ruby200/bin
      - SPEC CACHE DIRECTORY: C:/Users/Tonci/.gem/specs
      - RUBYGEMS PLATFORMS:
        - ruby
        - x86-mingw32
      - GEM PATHS:
         - C:/Ruby/Ruby200/lib/ruby/gems/2.0.0
         - C:/Users/Tonci/.gem/ruby/2.0.0
      - GEM CONFIGURATION:
         - :update_sources => true
         - :verbose => true
         - :backtrace => false
         - :bulk_threshold => 1000
      - REMOTE SOURCES:
         - https://rubygems.org/
    
  16. 16 Posted by neo on 03 Nov, 2013 02:44 PM

    neo's Avatar

    Occasionally I also receive the same error on mac, with ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-darwin12.4.0], and rvm

  17. 17 Posted by Nathan Youngman on 08 Nov, 2013 12:46 AM

    Nathan Youngman's Avatar

    Correction: I'm experiencing widespread SSL issues on OS X Mavericks (with fog/excon/etc.) so I suspect it's a problem with my Ruby installation and not specific to RubyGems.

  18. 18 Posted by dev on 08 Nov, 2013 12:53 AM

    dev's Avatar

    This occurred to me before the Mavericks update.

  19. 19 Posted by @glebm on 14 Nov, 2013 04:07 PM

    @glebm's Avatar

    Happened just now:

    Bundler::Fetcher::CertificateFailureError: Could not verify the SSL certificate for https://rubygems.org/.
    There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.
    An error occurred while installing activemerchant (1.37.0), and Bundler cannot continue.
    Make sure that `gem install activemerchant -v '1.37.0'` succeeds before bundling.
    

    Environment: ruby-p247, openssl 1.0.1e, bundler 1.3.5, OS X 10.8.5 (12F45).

  20. 20 Posted by Tass on 19 Nov, 2013 03:45 PM

    Tass's Avatar

    Experienced this yesterday and today, on different computers on different networks. Both Ubuntu 12.04 using RVM.

    This is crippling.

  21. 21 Posted by mkent on 19 Nov, 2013 10:54 PM

    mkent's Avatar

    Seeing this issue as well on Ubuntu precise in production. rubygems 2.1.11 across the board. We've got these failures coming out of bundler a few times of day:

    OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server hello A: wrong version number
    

    definitely sporadic.

  22. 22 Posted by Amed Rodriguez on 20 Nov, 2013 06:14 PM

    Amed Rodriguez's Avatar

    Just happened to me as well.

    I'm on Mavericks, ruby 2.0, rails 4.0, rbenv 0.4.0, rubygems 2.1.11

  23. 23 Posted by Adria Walker on 23 Nov, 2013 02:57 PM

    Adria Walker's Avatar
  24. 24 Posted by ShefIrrence on 25 Nov, 2013 01:42 AM

    ShefIrrence's Avatar

    true

  25. 25 Posted by Dan Kubb on 27 Nov, 2013 09:41 PM

    Dan Kubb's Avatar

    It might also be a good idea to make sure rubygems.org passes with the SSL Labs system: https://www.ssllabs.com/ssltest/analyze.html?d=rubygems.org .. currently it scores a "C".

    I can't say for sure it's the source of the issues on the server side, but I'd probably fix the obvious issues this reports first before digging in and debugging further.

    Feel free to email me if you'd like a hand with this.

  26. 26 Posted by Jared Beck on 23 Dec, 2013 09:50 PM

    Jared Beck's Avatar

    Our CircleCI builds are still failing sporadically.

    Bundler::Fetcher::CertificateFailureError: Could not verify the SSL certificate for https://rubygems.org/.
    There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification.
    

    Should CircleCI be taking some action?

  27. 27 Posted by Paul on 02 Jan, 2014 03:27 AM

    Paul's Avatar

    I'm experiencing the same issue on Windows.

    Updating https://github.com/codahale/bcrypt-ruby.git
    Fetching source index from http://gemcutter.org/
    Fetching source index from https://rubygems.org/
    Resolving dependencies...
    Could not verify the SSL certificate for
    https://rubygems.org/quick/Marshal.4.8/haml-rails-0.5.1.gemspec.rz.
    There is a chance you are experiencing a man-in-the-middle attack, but most
    likely your system doesn't have the CA certificates needed for verification. For
    
    information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without
    using SSL, edit your Gemfile sources and change 'https' to 'http'.
    

    These are the sources from my Gemfile

    source 'http://rubygems.org'
    source 'http://gemcutter.org'
    

    I can't work on the project currently since my version was out of date on my main development machine, and now I cannot update. I'm prevented from getting anything done as a result! Any ideas for help anyone?

  28. Support Staff 28 Posted by Nick Quaranto on 16 Apr, 2014 06:10 PM

    Nick Quaranto's Avatar

    Is this still happening? Also, I would remove gemcutter.org as a source. It's now rubygems.org! :)

  29. Nick Quaranto closed this discussion on 16 Apr, 2014 06:10 PM.

  30. Nick Quaranto re-opened this discussion on 16 Apr, 2014 06:10 PM

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

30 Oct, 2014 04:34 PM
30 Oct, 2014 02:46 PM
28 Oct, 2014 09:51 AM
23 Oct, 2014 05:37 PM
21 Oct, 2014 10:47 AM