tag:help.rubygems.org,2010-01-19:/discussions/problems/22609-intermittent-problems-when-pulling-gemsRubyGems.org: Discussion 2016-10-15T16:29:32Ztag:help.rubygems.org,2010-01-19:Comment/406004332016-08-28T18:28:39Z2016-08-30T13:55:28ZIntermittent problems when pulling gems <div><p>I believe I'm experiencing a similar issue. My automated
container builds are intermittently failing with errors similar to
the following:</p>
<p>The containers build with <code>ruby 2.3.1</code> on alpine
linux using <code>bundler 1.12.5</code></p>
<pre>
<code>Step 8 : RUN bundle install
---> Running in 7e37cf700710
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Fetching dependency metadata from https://rubygems.org/
Fetching git://github.com/coderjoe/sc2ranks.git
Installing rake 11.2.2
... excess output snipped...
Installing pry 0.10.4
Installing guard-minitest 2.4.6
Installing rspec-core 3.5.2
Gem::RemoteFetcher::UnknownHostError: timed out (https://rubygems.org/gems/rspec-expectations-3.5.0.gem)
Installing rspec-mocks 3.5.0
...excess output snipped...
An error occurred while installing rspec-expectations (3.5.0), and Bundler
cannot continue.
Make sure that `gem install rspec-expectations -v '3.5.0'` succeeds before
bundling.
ERROR: Service 'web' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 5</code>
</pre></div>coderjoetag:help.rubygems.org,2010-01-19:Comment/406004332016-08-29T20:20:49Z2016-08-29T20:20:52ZIntermittent problems when pulling gems <div><p>Same issue here</p>
<p>Gem::RemoteFetcher::UnknownHostError: timed out (<a href="https://rubygems.org/gems/wait-0.5.2.gem">https://rubygems.org/gems/wait-0.5.2.gem</a>)</p>
<p>Except that it is a random gem (sometimes 2) that times out.</p>
<p>Majority of builds failing here</p></div>trevortag:help.rubygems.org,2010-01-19:Comment/406004332016-08-29T20:45:47Z2016-08-29T20:45:47ZIntermittent problems when pulling gems <div><p>We are having the same issue since this morning (~10 hours
ago)</p>
<p>Gem::RemoteFetcher::UnknownHostError: timed out (<a href="https://rubygems.org/gems/rails-api-0.4.0.gem">https://rubygems.org/gems/rails-api-0.4.0.gem</a>)</p>
<p>It fails with a different gem every time, sometimes it install
everything without issues.</p></div>Lucastag:help.rubygems.org,2010-01-19:Comment/406004332016-08-29T22:31:32Z2016-08-29T22:31:33ZIntermittent problems when pulling gems <div><p>We're having the same issue running builds in containers on
digital ocean. 9 out of 10 builds fail with timeouts when pulling
gems from rubygems or github.</p>
<pre>
</pre></div>Bentag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T12:11:05Z2016-08-30T12:11:06ZIntermittent problems when pulling gems <div><p>We are having the same issue as well. We are seeing the failures
both on our production infrastructure in a private hosting
environment, and on our development infrastructure in digital
ocean, so it's not tied to a specific network or machine. The only
common piece is that it's only failing within docker container
builds.</p></div>Andrew Swerlicktag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T13:51:09Z2016-08-30T13:55:28ZIntermittent problems when pulling gems <div><p>I vaguely remember it happening both for local <code>bundle
install</code> as well as on my containers when debugging, but it
has been so long I wonder if it's a false memory.</p>
<p>Is everybody in the thread experiencing the problem in Docker
containers?</p></div>coderjoetag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T18:00:38Z2016-08-30T18:00:42ZIntermittent problems when pulling gems <div><p>I'm running into the same issue but not with Docker. Connection
timeouts are causing my gitlab installation to fail. It's always
with a different gem and it's always with connections to
<strong>fastly.net</strong>.</p></div>emrystag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T22:55:58Z2016-08-30T22:55:58ZIntermittent problems when pulling gems <div><p>@coderjoe this issue in only occurring in Docker containers for
us. Bundling elsewhere is fine.</p></div>Bentag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T23:24:42Z2016-08-30T23:24:45ZIntermittent problems when pulling gems <div><p>Related: <a href="https://github.com/rubygems/rubygems.org/issues/1404">https://github.com/rubygems/rubygems.org/issues/1404</a></p>
<hr>
<p>I used Gemstash and created my private mirror to work-around
these problems that literally prevent any build from finishing.
<a href="https://github.com/bundler/gemstash">https://github.com/bundler/gemstash</a></p></div>nowakertag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T23:24:56Z2016-08-30T23:24:58ZIntermittent problems when pulling gems <div><p>Hi,</p>
<p>We've been seeing this over the past few days at Wikimedia. We
run our own Jenkins CI (integration.wikimedia.org), and run tests
in OpenStack VMs. We don't use docker.</p>
<p>Not all of our builds are failing, just some. We haven't been
able to identify any pattern. I can provide logs if that would be
helpful, but I think the error messages are the same as what other
people have said.</p></div>Kunal Mehtatag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T23:34:45Z2016-08-30T23:34:48ZIntermittent problems when pulling gems <div><p>@Kunal, what cloud provider / data center are you running it in?
Wondering if it's U.S. or global issue.</p></div>nowakertag:help.rubygems.org,2010-01-19:Comment/406004332016-08-30T23:57:30Z2016-08-30T23:57:33ZIntermittent problems when pulling gems <div><p>@nowaker: Wikimedia's Jenkins node pool currently runs in
Virginia, US. (<a href="https://wikitech.wikimedia.org/wiki/Clusters">Clusters</a>,
<a href="https://wikitech.wikimedia.org/wiki/Eqiad_cluster">Eqiad</a>)</p></div>Timo Tijhoftag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T00:00:38Z2016-08-31T00:00:38ZIntermittent problems when pulling gems <div><p>There seem to be a few different types of issue being reported
here in multiple environments.</p>
<p>Here is a summary of what I see above:</p>
<p><strong>UnknownHostError rubygems.org</strong></p>
<p>rubygems.org uses dnsimple for name resolution, and they have
<a href="http://dnsimplestatus.com">no recent indecent reports</a>
(but the problem could still be there).</p>
<p>Some of you have reported that you only have an issue inside
containers, so can you check if something can be adjusted with the
container host's DNS resolver?</p>
<p>If you don't have an issue outside containers can you reply with
more information about your environment?</p>
<p><strong>Timeout with rubygems.global.ssl.fastly.net</strong></p>
<p>If you're having this problem inside a CI environment can you
capture a traceroute to rubygems.global.ssl.fastly.net and reply
here with it?</p>
<p>If you're having this problem in an environment with a browser
can you reply with the base64 block from <a href="http://www.fastly-debug.com">http://www.fastly-debug.com</a>?</p></div>Eric Hodeltag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T04:10:06Z2016-08-31T04:10:06ZIntermittent problems when pulling gems <div><p>On the machine on which I'm trying to install gitlab ce,
fastly-debug.com is stuck at "Collecting data please wait."</p>
<p>traceroute to rubygems.global.ssl.fastly.net (151.101.56.249),
30 hops max, 60 byte packets<br>
1 192.168.1.43 (192.168.1.43) 10.635 ms 10.894 ms 11.201 ms 2
96.120.62.169 (96.120.62.169) 13.979 ms 15.224 ms 16.015 ms 3
te-0-4-0-11-sur01.pittsburgh.pa.pitt.comcast.net (68.86.102.57)
16.452 ms 16.688 ms 16.920 ms 4
te-0-1-0-11-sur01.pittsburgh.pa.pitt.comcast.net (68.86.100.26)
19.476 ms 20.498 ms 20.499 ms 5
te-0-0-0-0-ar01.mckeesport.pa.pitt.comcast.net (68.86.147.109)
21.120 ms 23.903 ms 24.870 ms 6
be-7016-cr02.ashburn.va.ibone.comcast.net (68.86.91.25) 30.875 ms
17.725 ms 19.005 ms 7 be-10114-cr02.56marietta.ga.ibone.comcast.net
(68.86.85.10) 30.918 ms 36.277 ms 36.539 ms 8
hu-0-11-0-0-pe04.56marietta.ga.ibone.comcast.net (68.86.89.90)
35.853 ms 34.571 ms 35.575 ms 9 * * * 10 * * *<br>
11 * * *<br>
12 * * *<br>
13 * * *<br>
14 * * *<br>
15 * * *<br>
16 * * *<br>
17 * * *<br>
18 * * *<br>
19 * * *<br>
20 * * *<br>
21 * * *<br>
22 * * *<br>
23 * * *<br>
24 * * *<br>
25 * * *<br>
26 * * *<br>
27 * * *<br>
28 * * *<br>
29 * * *<br>
30 * * *</p></div>emrystag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T04:34:48Z2016-08-31T04:34:50ZIntermittent problems when pulling gems <div><p>I wasn't able to do much investigation tonight, but I did throw
together this gist.</p>
<p><a href="https://gist.github.com/coderjoe/b31527758998331ed6ae9f2359350723">
https://gist.github.com/coderjoe/b31527758998331ed6ae9f2359350723</a></p>
<p>It contains a compose script and docker container I used to
reproduce the issue, and logs from both a container build (which
outputs resolv.conf, bundle env, and bundle install --verbose), and
from traceroutes from within the containers themselves to the
various failing locations.</p>
<p>I'm experiencing the following two errors relatively often (and
they are both present in the docker-bundle-verbose log in the
gist:</p>
<pre>
<code>Net::OpenTimeout: execution expired</code>
</pre>
<p>and</p>
<pre>
<code>Gem::RemoteFetcher::UnknownHostError: timed out (https://rubygems.org/gems/pkg-config-1.1.7.gem)</code>
</pre>
<p>Hopefully this information is helpful.</p></div>coderjoetag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T05:12:49Z2016-08-31T05:12:49ZIntermittent problems when pulling gems <div><p>Can you retry now?</p>
<p>Fastly made some configuration changes at one of our POPs that
may have fixed this issue.</p>
<p>If this didn't fix your issue please include a traceroute to
help narrow it down.</p></div>Eric Hodeltag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T05:22:48Z2016-08-31T05:22:49ZIntermittent problems when pulling gems <div><p>Both traceroute and fastly-debug.com give the same results for
me.</p>
<p>If it does help, I do see the following fields in
fastly-debug.com:<br>
Server Connection Info<br>
IP 151.101.56.64<br>
Datacenter ATL</p>
<p>Request Info<br>
Time Wed Aug 31 2016 01:19:50 GMT-0400 (EDT)<br>
Host <a href="http://www.fastly-debug.com">www.fastly-debug.com</a><br>
Accept
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<br>
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101
Firefox/48.0<br>
Accept-Language en-US,en;q=0.5<br>
Accept-Encoding gzip<br>
X-Forwarded-For</p></div>emrystag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T05:41:49Z2016-08-31T05:41:50ZIntermittent problems when pulling gems <div><p>Actually, I just tried the gitlab install again and it worked.
Not sure why fastly-debug.com isn't working for me though.</p></div>emrystag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T06:10:11Z2016-08-31T06:10:12ZIntermittent problems when pulling gems <div><p>I'm away from my machine for the night. I'll give it another go
tomorrow as<br>
soon as I can.</p></div>coderjoetag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T23:12:21Z2016-08-31T23:12:23ZIntermittent problems when pulling gems <div><p>For what it's worth I've been completely unable to reproduce the
problem as of today. I've spent the last hour trying and haven't
once received intermittent errors.</p></div>coderjoetag:help.rubygems.org,2010-01-19:Comment/406004332016-08-31T23:26:30Z2016-08-31T23:26:31ZIntermittent problems when pulling gems <div><p>All looking good here too. Thanks for following up Eric.</p></div>Bentag:help.rubygems.org,2010-01-19:Comment/406004332016-09-01T03:12:03Z2016-09-01T03:12:04ZIntermittent problems when pulling gems <div><p>Agreed, our builds seem to be working again</p>
<p>~Andrew S.</p></div>Andrew Swerlicktag:help.rubygems.org,2010-01-19:Comment/406004332016-10-15T16:27:20Z2016-10-15T16:27:20ZIntermittent problems when pulling gems <div><p>Hi everyone,</p>
<p>Looks like this was resolved. I am closing it.</p></div>sonalkr132