Timeout error

guard.16's Avatar

guard.16

10 Apr, 2018 02:55 PM

Hey - I am getting a timeout error on every gem update I am trying to do. after doing some reading - some peopel had this issue but it was specific to their projects. I removed the project from my computer, uninstalled rails - and then attempted to update the gems. I can download and install each gem individually if I go to a specific gem URL after the timeout happens. if I run gem update I get the following:

ERROR: While executing gem ... (Gem::RemoteFetcher::UnknownHostError)
    timed out (https://api.rubygems.org/latest_specs.4.8.gz)

gem install rails gives me:
ERROR: Could not find a valid gem 'rails' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - timed out (https://api.rubygems.org/specs.4.8.gz)

All help related to this issue will be appreciated -it's been going on for about 2 weeks but is only now a critical issue.

  1. Support Staff 1 Posted by sonalkr132 on 22 Apr, 2018 12:53 PM

    sonalkr132's Avatar

    hi guard.16,

    Looks like api.rubygems.org is not resolvable/reachable from your host. Can you please try to download spec file using wget or curl and see what error comes up?

  2. 2 Posted by William Makley on 23 Apr, 2018 07:28 PM

    William Makley's Avatar

    Hi, I'm having the exact same issue as guard.16. It is on one specific server, and I can't find any indication anywhere of what may be happening.

    /opt/rubies$ ./ruby-2.3.7/bin/gem install bundler
    ERROR: Could not find a valid gem 'bundler' (>= 0), here is why:
              Unable to download data from https://rubygems.org/ - timed out (https://api.rubygems.org/specs.4.8.gz)

    However, curl works fine:

    curl https://api.rubygems.org/specs.4.8.gz > specs.4.8.gz

    No problems with that!

  3. 3 Posted by William Makley on 24 Apr, 2018 02:40 PM

    William Makley's Avatar

    *This is on a fully updated Debian 8 server. I have other servers with the same OS at the same host without this issue. I used ruby-build to install ruby 2.3.7 and ruby 2.4.4, and I can't find a single error in any system log (including the firewall). I'd be ecstatic if anyone has any idea of where to look or what to check. All rubies on this server suddenly started having the same issue as OP a couple weeks ago.

  4. 4 Posted by Ivan on 25 Apr, 2018 02:47 AM

    Ivan's Avatar

    I'm experiencing the same problem, running NixOS and I've tried ruby 2.4.1 and 2.5.0. It seems to affect every gem I've tried so far (ffi, coderay, rouge, spoon). I've tried https and http. I'm able to curl a specific gem (e.g. `curl http://rubygems.org/gems/ffi-1.9.23.gem -o outfile`) successfully.

  5. Support Staff 5 Posted by kerrizor on 04 May, 2018 05:35 PM

    kerrizor's Avatar

    Checking in: is this still an issue for you?

  6. 6 Posted by cabasp on 08 May, 2018 12:44 AM

    cabasp's Avatar

    I'm having the same issue, i'm able to curl https://api.rubygems.org/specs.4.8.gz > specs.4.8.gz but running wget https://api.rubygems.org/specs.4.8.gz times out.

    Running debian 9.4 on this machine, but on a macbook pro (running High Sierra) it works perfectly. The problem started around mid april for me

  7. 7 Posted by Alexei Khlebnik... on 09 May, 2018 08:59 AM

    Alexei Khlebnikov's Avatar

    The issue seems to be that https://api.rubygems.org/ is unreachable by IPv6.

  8. 8 Posted by Alexei Khlebnik... on 09 May, 2018 09:05 AM

    Alexei Khlebnikov's Avatar

    Here is (part of) Nmap report for IPv6 addresses of api.rubygems.com:

    # Nmap 7.70 scan initiated Wed May  9 10:50:14 2018 as: nmap -p 443 -6 -Pn -v -oG - 2a04:4e42::514 2a04:4e42:600::514 2a04:4e42:400::514 2a04:4e42:200::514
    
    Host: 2a04:4e42::514 () Ports: 443/filtered/tcp//https///
    Host: 2a04:4e42:600::514 ()     Ports: 443/filtered/tcp//https///
    Host: 2a04:4e42:400::514 ()     Ports: 443/filtered/tcp//https///
    Host: 2a04:4e42:200::514 ()     Ports: 443/filtered/tcp//https///
    

    Report for IPv4 looks fine though:

    # Nmap 7.70 scan initiated Wed May  9 10:56:22 2018 as: nmap -p 443 -4 -Pn -v -oG - 151.101.130.2 151.101.194.2 151.101.2.2
    
    Host: 151.101.130.2 ()  Ports: 443/open/tcp//https///
    Host: 151.101.194.2 ()  Ports: 443/open/tcp//https///
    Host: 151.101.2.2 ()    Ports: 443/open/tcp//https///
    
  9. 9 Posted by Ivan on 10 May, 2018 02:55 AM

    Ivan's Avatar

    I just tried the same nmap commands and IPv6 looks fine now as well. I'm also now able to fetch dependencies successfully. It's not quite definitive evidence, but does lend support to the theory that IPv6 was the issue. Thanks for pointing that out, Alexei!

  10. 10 Posted by William Makley on 10 May, 2018 04:01 PM

    William Makley's Avatar

    I'm still having the same problem with my Rackspace server, but I will look into IPv6 when I get a chance.

  11. 11 Posted by David Bailey on 10 May, 2018 08:16 PM

    David Bailey's Avatar

    The same problem is currently being experienced by me.

    Rubygems is unable to download from https://api.rubygems.org/specs.4.8.gz, yet that address is reachable by me (via ping) and I can download the file using my browser without problem.
    I only just now updated to the latest rubygems (2.7.6, using Ruby 2.4.4p296) (had issues before and after update).

  12. 12 Posted by Alexei Khlebnik... on 11 May, 2018 10:37 AM

    Alexei Khlebnikov's Avatar

    I am still experiencing the issue. And nmap still shows "filtered" for IPv6 addresses of api.rubygems.org. If I disable IPv6 - everything works fine, gem's remote commands work as expected.

  13. 13 Posted by Alexei Khlebnik... on 11 May, 2018 10:44 AM

    Alexei Khlebnikov's Avatar

    yet that address is reachable by me (via ping)

    Your ping may have chosen the IPv4 address maybe? My ping fails to reach api.rubygems.org via IPv6. Succeeds via IPv4 though.

  14. 14 Posted by Alexei Khlebnik... on 11 May, 2018 11:18 AM

    Alexei Khlebnikov's Avatar

    I can download the file using my browser

    Many browsers implement Happy Eyeballs algorithm (fast fallback to IPv4, https://en.wikipedia.org/wiki/Happy_Eyeballs ). Thus a browser may work, but gem may not.

  15. 15 Posted by Alexei Khlebnik... on 11 May, 2018 11:25 AM

    Alexei Khlebnikov's Avatar

    I'm having the same issue ... Running debian 9.4 on this machine, but on a macbook pro (running High Sierra) it works perfectly

    OS X implements Happy Eyeballs on the system level. Linux does not.

    Ruby could implement Happy Eyeballs in its standard library (in TCPClient?), but so far it didn't happen.

  16. 16 Posted by Alexei Khlebnik... on 11 May, 2018 11:27 AM

    Alexei Khlebnikov's Avatar

    Dear Support Staff members. It would be nice if you fixed this issue soon. Now, when Ubuntu 18.04 has been released, many people are upgrading their gems and having this issue.

  17. Support Staff 17 Posted by sonalkr132 on 19 May, 2018 09:28 PM

    sonalkr132's Avatar

    Hi Alexei,

    Thanks for pointing out that our ipv6 isn't work. We will try to fix it as soon as possible. It shouldn't cause your gem install to fail or timeout tho. rubygems client uses ipv4 addresses. Can you please verify you see ipv6 requests are being made when you run gem install ?

  18. 18 Posted by Tien Do on 20 May, 2018 09:57 AM

    Tien Do's Avatar

    Got the same issue, disable ipv6 for now as a temporary solution:
    https://wiki.archlinux.org/index.php/IPv6#Disable_IPv6

  19. 19 Posted by Alexei Khlebnik... on 22 May, 2018 12:08 PM

    Alexei Khlebnikov's Avatar

    I also published another temporary solution on StackOverflow, that selectively deprioritizes IPv6 addresses of api.rubygems.org only:

    https://stackoverflow.com/a/50349235/1016580

  20. 20 Posted by Alexei Khlebnik... on 22 May, 2018 12:27 PM

    Alexei Khlebnikov's Avatar

    Hi Aditya,

    I have made a small test that you asked me to do. I used gem search instead of gem install though.

    In one terminal I am running gem:

    $ gem search --remote rails
    
    *** REMOTE GEMS ***
    

    gem is stalling for ~1 minute in this terminal.

    While gem is stalling, I am running netstat in another terminal:

    # netstat -tnp | grep ruby
    tcp6       0      1 (my ipv6 address here):39080 2a04:4e42:400::514:443  SYN_SENT    16339/ruby2.5
    

    And 2a04:4e42:400::514 is one of the addresses of api.rubygems.org.

    My conclusion is that gem uses IPv6 when it can.

  21. 21 Posted by Alexei Khlebnik... on 22 May, 2018 12:44 PM

    Alexei Khlebnikov's Avatar

    Here is gem's debug output, if it will be helpful:

    $ time gem search --remote --verbose --debug rails
    NOTE:  Debugging mode prints all exceptions even when rescued
    Exception `LoadError' at /usr/lib/ruby/2.5.0/resolv.rb:170 - LoadError
    
    *** REMOTE GEMS ***
    
    Exception `Errno::EEXIST' at /usr/lib/ruby/2.5.0/fileutils.rb:232 - File exists @ dir_s_mkdir - /home/alexei/.gem/specs/api.rubygems.org%443
    Exception `Errno::ENOENT' at /usr/lib/ruby/2.5.0/rubygems/remote_fetcher.rb:325 - No such file or directory @ rb_file_s_stat - /home/alexei/.gem/specs/api.rubygems.org%443/latest_specs.4.8
    Exception `Net::OpenTimeout' at /usr/lib/ruby/2.5.0/net/http.rb:937 - execution expired
    Exception `Net::OpenTimeout' at /usr/lib/ruby/2.5.0/net/http.rb:939 - execution expired
    Exception `Net::OpenTimeout' at /usr/lib/ruby/2.5.0/timeout.rb:114 - execution expired
    Exception `Net::OpenTimeout' at /usr/lib/ruby/2.5.0/net/http.rb:997 - execution expired
    Exception `Gem::RemoteFetcher::UnknownHostError' at /usr/lib/ruby/2.5.0/rubygems/remote_fetcher.rb:306 - timed out (https://api.rubygems.org/latest_specs.4.8.gz)
    Exception `Gem::RemoteFetcher::UnknownHostError' at /usr/lib/ruby/2.5.0/rubygems/spec_fetcher.rb:269 - timed out (https://api.rubygems.org/latest_specs.4.8.gz)
    
    
    real    1m0,176s
    user    0m0,139s
    sys     0m0,011s
    
  22. 22 Posted by Zach Huntington... on 17 Jul, 2018 02:33 AM

    Zach Huntington-Meath's Avatar

    I also just hit this issue, disabling IPV6 on my workstation fixed it.

  23. Support Staff 23 Posted by kerrizor on 22 Jul, 2018 05:11 PM

    kerrizor's Avatar

    Closing for now. We can follow up on IPV6 work on the project, I believe, as at this point we're not really providing support in this thread.

  24. kerrizor closed this discussion on 22 Jul, 2018 05:11 PM.

  25. sonalkr132 re-opened this discussion on 25 Aug, 2020 06:13 AM

  26. Support Staff 24 Posted by sonalkr132 on 25 Aug, 2020 06:15 AM

    sonalkr132's Avatar

    Hi everyone,

    Sorry about the delay in getting this resolved. I have filed a support ticket on Fastly and hopefully, we will get this resolved soon.

    It would be very helpful if any of you can take some to run some commands for debugging info. Specifically, what is the output of the following commands when running from the host which has IPv6 connection issue to rubygems.org:

    $ wget -6 rubygems.org
    $ wget -6 www.ttora.com
    $ traceroute 2a04:4e42::70
    
    If you are feeling particularly generous, please add send pcap file for requests to rubygems.org over IPv6 as well. You can use tshark -i any -f "host rubygems.org" -w rubygems.pcap or sudo tcpdump -i any host rubygems.org -w rubygems.pcap
  27. 25 Posted by Curtis Rueden on 27 Aug, 2020 03:38 AM

    Curtis Rueden's Avatar

    Thanks sonalkr132 for following up!

    Using latest Ruby from an up-to-date Homebrew installation:

    $ sw_vers
    ProductName: Mac OS X
    ProductVersion: 10.14.6
    BuildVersion: 18G6020

    $ ruby --version
    ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin18]

    $ wget -6 rubygems.org
    --2020-08-26 22:24:39-- http://rubygems.org/
    Resolving rubygems.org (rubygems.org)... 2a04:4e42::70
    Connecting to rubygems.org (rubygems.org)|2a04:4e42::70|:80... failed: Operation timed out.
    Retrying.

    --2020-08-26 22:25:55-- (try: 2) http://rubygems.org/
    Connecting to rubygems.org (rubygems.org)|2a04:4e42::70|:80... ^C

    $ wget -6 www.ttora.com
    --2020-08-26 22:26:08-- http://www.ttora.com/
    Resolving www.ttora.com (www.ttora.com)... 2606:4700::6810:ad66, 2606:4700::6810:4552
    Connecting to www.ttora.com (www.ttora.com)|2606:4700::6810:ad66|:80... failed: Operation timed out.
    Connecting to www.ttora.com (www.ttora.com)|2606:4700::6810:4552|:80... ^C

    $ traceroute 2a04:4e42::70
    traceroute: unknown host 2a04:4e42::70

    $ traceroute6 2a04:4e42::70
    traceroute6 to 2a04:4e42::70 (2a04:4e42::70) from 2600:6c44:97f:fa4a:6010:8efa:2829:ed0f, 64 hops max, 12 byte packets
     1 2600:6c44:97f:fa4a:6238:e0ff:fe73:2187 2.125 ms 2.838 ms 2.362 ms
     2 * * *
     3 2001-0506-0100-2216-0000-0000-0000-0005.inf6.spectrum.com 18.902 ms 13.942 ms 12.446 ms
     4 2001-0506-0100-80ba-0000-0000-0000-0001.inf6.spectrum.com 14.032 ms 15.166 ms *
     5 2001-0506-0100-206c-0000-0000-0000-0001.inf6.spectrum.com 20.474 ms 17.439 ms *
     6 2001-0506-0100-2076-0000-0000-0000-0005.inf6.spectrum.com 22.806 ms 19.869 ms *
     7 2001-0506-0100-0202-0000-0000-0000-0009.inf6.spectrum.com 33.397 ms * 26.311 ms
     8 2001-0506-0100-0013-0000-0000-0000-0005.inf6.spectrum.com 24.628 ms * 30.159 ms
     9 * * *
    10 * * *
    11 * * *
    12 * * *
    13 * * *
    14 * * *
    15 * * *
    16 * * *
    17 * * *
    18 * * *
    19 * * *
    ^C

    ... meanwhile ...

    $ sudo tcpdump -i any host rubygems.org -w rubygems.pcap
    tcpdump: data link type PKTAP
    tcpdump: listening on any, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
    ^C72 packets captured
    16797 packets received by filter
    0 packets dropped by kernel

    $ ls -l rubygems.pcap
    -rw-r--r-- 1 root admin 10988 Aug 26 22:33 rubygems.pcap

  28. Support Staff 26 Posted by sonalkr132 on 27 Aug, 2020 11:38 AM

    sonalkr132's Avatar

    Hi Curtis,

    Thank you for taking the time to send us this.
    Unfortunately, your issue seems to be specific to your host/network.

    $ wget -6 www.ttora.com --2020-08-26 22:26:08-- http://www.ttora.com/ Resolving www.ttora.com (www.ttora.com)... 2606:4700::6810:ad66, 2606:4700::6810:4552
    Connecting to www.ttora.com (www.ttora.com)|2606:4700::6810:ad66|:80... failed: Operation timed out.
    Connecting to www.ttora.com (www.ttora.com)|2606:4700::6810:4552|:80... ^C

    users who have the issue with accessing rubygems.org specifically are able to access other sites on IPv6. Even wget -6 google.com will timeout for you. Perhaps you can run some tests on your network about why you are not able to connect to IPv6 sites when you have an IPv6 address on your host https://test-ipv6.com/.

    Please feel free to let us know if you are abl to access sites beside rubygems.org on IPv6 but not rubygems.org.

    At this time I am intrested debug output from user who can't access rubygems.org specifically.

    If you are interested in knowing why gem install is timing out even tho you have IPv4 connectivity, please check this.

  29. 27 Posted by Curtis Rueden on 27 Aug, 2020 02:35 PM

    Curtis Rueden's Avatar

    Thanks sonalkr132 for the quick reply, and sorry that I can't provide the data you need.

    I look forward to your PR being finalized and merged, so that people with broken IPv6 support (I'm guessing there are many besides only me) can still have working gem commands out of the box. I greatly appreciate your efforts in driving this forward!

  30. Support Staff 28 Posted by sonalkr132 on 29 Sep, 2020 12:45 PM

    sonalkr132's Avatar

    No problem. I am hopeful that the fix will be included in the next release.
    You can help us test it and confirm that it fixed your issue, either right from my branch or you can wait until it has a rc.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

29 Sep, 2020 01:29 PM
29 Sep, 2020 12:45 PM
29 Sep, 2020 12:37 PM
27 Sep, 2020 05:01 AM
06 Sep, 2020 12:36 PM