Gateway error on https://api.rubygems.org/api/v1/dependencies with more than 127 dependencies

jon's Avatar

jon

19 Sep, 2018 03:34 PM

Hi,

I'm trying to install the aws-sdk gem and it's super, super slow (see http://help.rubygems.org/discussions/problems/31028-gem-install-loo... for related discussion).

As posted in the bad.txt output there, it seems to first fetch the dependencies of the aws-sdk gem, the recursively of its dependencies, until some point where it tries to fetch lot of aws-sdk-* dependencies information using the https://api.rubygems.org/api/v1/dependencies endpoint, in which case the API returns a 502 error.

I managed to reproduce this locally with curl and it seems past 127 dependencies, the API starts to return this error:

$ echo "$deps"
aws-sdk-acm
aws-sdk-acmpca
aws-sdk-alexaforbusiness
aws-sdk-apigateway
aws-sdk-applicationautoscaling
aws-sdk-applicationdiscoveryservice
aws-sdk-appstream
aws-sdk-appsync
aws-sdk-athena
aws-sdk-autoscaling
aws-sdk-autoscalingplans
aws-sdk-batch
aws-sdk-budgets
aws-sdk-cloud9
aws-sdk-clouddirectory
aws-sdk-cloudformation
aws-sdk-cloudfront
aws-sdk-cloudhsm
aws-sdk-cloudhsmv2
aws-sdk-cloudsearch
aws-sdk-cloudsearchdomain
aws-sdk-cloudtrail
aws-sdk-cloudwatch
aws-sdk-cloudwatchevents
aws-sdk-cloudwatchlogs
aws-sdk-codebuild
aws-sdk-codecommit
aws-sdk-codedeploy
aws-sdk-codepipeline
aws-sdk-codestar
aws-sdk-cognitoidentity
aws-sdk-cognitoidentityprovider
aws-sdk-cognitosync
aws-sdk-comprehend
aws-sdk-configservice
aws-sdk-connect
aws-sdk-costandusagereportservice
aws-sdk-costexplorer
aws-sdk-databasemigrationservice
aws-sdk-datapipeline
aws-sdk-dax
aws-sdk-devicefarm
aws-sdk-directconnect
aws-sdk-directoryservice
aws-sdk-dlm
aws-sdk-dynamodb
aws-sdk-dynamodbstreams
aws-sdk-ec2
aws-sdk-ecr
aws-sdk-ecs
aws-sdk-efs
aws-sdk-eks
aws-sdk-elasticache
aws-sdk-elasticbeanstalk
aws-sdk-elasticloadbalancing
aws-sdk-elasticloadbalancingv2
aws-sdk-elasticsearchservice
aws-sdk-elastictranscoder
aws-sdk-emr
aws-sdk-firehose
aws-sdk-fms
aws-sdk-gamelift
aws-sdk-glacier
aws-sdk-glue
aws-sdk-greengrass
aws-sdk-guardduty
aws-sdk-health
aws-sdk-iam
aws-sdk-importexport
aws-sdk-inspector
aws-sdk-iot
aws-sdk-iot1clickdevicesservice
aws-sdk-iot1clickprojects
aws-sdk-iotanalytics
aws-sdk-iotdataplane
aws-sdk-iotjobsdataplane
aws-sdk-kinesis
aws-sdk-kinesisanalytics
aws-sdk-kinesisvideo
aws-sdk-kinesisvideoarchivedmedia
aws-sdk-kinesisvideomedia
aws-sdk-kms
aws-sdk-lambda
aws-sdk-lambdapreview
aws-sdk-lex
aws-sdk-lexmodelbuildingservice
aws-sdk-lightsail
aws-sdk-machinelearning
aws-sdk-macie
aws-sdk-marketplacecommerceanalytics
aws-sdk-marketplaceentitlementservice
aws-sdk-marketplacemetering
aws-sdk-mediaconvert
aws-sdk-medialive
aws-sdk-mediapackage
aws-sdk-mediastore
aws-sdk-mediastoredata
aws-sdk-mediatailor
aws-sdk-migrationhub
aws-sdk-mobile
aws-sdk-mq
aws-sdk-mturk
aws-sdk-neptune
aws-sdk-opsworks
aws-sdk-opsworkscm
aws-sdk-organizations
aws-sdk-pi
aws-sdk-pinpoint
aws-sdk-polly
aws-sdk-pricing
aws-sdk-rds
aws-sdk-redshift
aws-sdk-rekognition
aws-sdk-resourcegroups
aws-sdk-resourcegroupstaggingapi
aws-sdk-route53
aws-sdk-route53domains
aws-sdk-s3
aws-sdk-sagemaker
aws-sdk-sagemakerruntime
aws-sdk-secretsmanager
aws-sdk-serverlessapplicationrepository
aws-sdk-servicecatalog
aws-sdk-servicediscovery
aws-sdk-ses
aws-sdk-shield
aws-sdk-signer
aws-sdk-simpledb
aws-sdk-sms
aws-sdk-snowball
aws-sdk-sns
aws-sdk-sqs
aws-sdk-ssm
aws-sdk-states
aws-sdk-storagegateway
aws-sdk-support
aws-sdk-swf
aws-sdk-transcribeservice
$ echo "$deps" | wc -l
138
$ curl https://api.rubygems.org/api/v1/dependencies?gems=$(echo $(echo "$deps" | head -n 127) | tr " " ",") --dump-header -
HTTP/1.1 200 OK
Content-Type: application/octet-stream; charset=utf-8
Date: Wed, 19 Sep 2018 15:31:02 GMT
Via: 1.1 varnish
[...]
$ curl https://api.rubygems.org/api/v1/dependencies?gems=$(echo $(echo "$deps" | head -n 128) | tr " " ",") --dump-header -
HTTP/1.1 502 Bad Gateway
Retry-After: 0
Content-Type: text/html; charset=utf-8
Content-Length: 428
Accept-Ranges: bytes
Date: Wed, 19 Sep 2018 15:31:15 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-fra19142-FRA
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1537371075.869384,VS0,VE465
Server: RubyGems.org


<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>502 Bad Gateway</title>
  </head>
  <body>
    <h1>Error 502 Bad Gateway</h1>
    <p>Bad Gateway</p>
    <h3>Guru Mediation:</h3>
    <p>Details: cache-fra19142-FRA 1537371075 2098365379</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

This makes installing this aws-sdk gems very very slow (it looks like there's a fallback mechanism to discover the dependencies once that request failed.)

The initial command to execute was the following:

$ gem install -V aws-sdk
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=aws-sdk
200 OK
Getting SRV record failed: DNS result has no information for _rubygems._tcp.api.rubygems.org
GET https://api.rubygems.org/api/v1/dependencies?gems=aws-sdk-resources
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=aws-sdk-acm,aws-sdk-acmpca,aws-sdk-alexaforbusiness,aws-sdk-apigateway,aws-sdk-applicationautoscaling,aws-sdk-applicationdiscoveryservice,aws-sdk-appstream,aws-sdk-appsync,aws-sdk-athena,aws-sdk-autoscaling,aws-sdk-autoscalingplans,aws-sdk-batch,aws-sdk-budgets,aws-sdk-cloud9,aws-sdk-clouddirectory,aws-sdk-cloudformation,aws-sdk-cloudfront,aws-sdk-cloudhsm,aws-sdk-cloudhsmv2,aws-sdk-cloudsearch,aws-sdk-cloudsearchdomain,aws-sdk-cloudtrail,aws-sdk-cloudwatch,aws-sdk-cloudwatchevents,aws-sdk-cloudwatchlogs,aws-sdk-codebuild,aws-sdk-codecommit,aws-sdk-codedeploy,aws-sdk-codepipeline,aws-sdk-codestar,aws-sdk-cognitoidentity,aws-sdk-cognitoidentityprovider,aws-sdk-cognitosync,aws-sdk-comprehend,aws-sdk-configservice,aws-sdk-connect,aws-sdk-costandusagereportservice,aws-sdk-costexplorer,aws-sdk-databasemigrationservice,aws-sdk-datapipeline,aws-sdk-dax,aws-sdk-devicefarm,aws-sdk-directconnect,aws-sdk-directoryservice,aws-sdk-dlm,aws-sdk-dynamodb,aws-sdk-dynamodbstreams,aws-sdk-ec2,aws-sdk-ecr,aws-sdk-ecs,aws-sdk-efs,aws-sdk-eks,aws-sdk-elasticache,aws-sdk-elasticbeanstalk,aws-sdk-elasticloadbalancing,aws-sdk-elasticloadbalancingv2,aws-sdk-elasticsearchservice,aws-sdk-elastictranscoder,aws-sdk-emr,aws-sdk-firehose,aws-sdk-fms,aws-sdk-gamelift,aws-sdk-glacier,aws-sdk-glue,aws-sdk-greengrass,aws-sdk-guardduty,aws-sdk-health,aws-sdk-iam,aws-sdk-importexport,aws-sdk-inspector,aws-sdk-iot,aws-sdk-iot1clickdevicesservice,aws-sdk-iot1clickprojects,aws-sdk-iotanalytics,aws-sdk-iotdataplane,aws-sdk-iotjobsdataplane,aws-sdk-kinesis,aws-sdk-kinesisanalytics,aws-sdk-kinesisvideo,aws-sdk-kinesisvideoarchivedmedia,aws-sdk-kinesisvideomedia,aws-sdk-kms,aws-sdk-lambda,aws-sdk-lambdapreview,aws-sdk-lex,aws-sdk-lexmodelbuildingservice,aws-sdk-lightsail,aws-sdk-machinelearning,aws-sdk-macie,aws-sdk-marketplacecommerceanalytics,aws-sdk-marketplaceentitlementservice,aws-sdk-marketplacemetering,aws-sdk-mediaconvert,aws-sdk-medialive,aws-sdk-mediapackage,aws-sdk-mediastore,aws-sdk-mediastoredata,aws-sdk-mediatailor,aws-sdk-migrationhub,aws-sdk-mobile,aws-sdk-mq,aws-sdk-mturk,aws-sdk-neptune,aws-sdk-opsworks,aws-sdk-opsworkscm,aws-sdk-organizations,aws-sdk-pi,aws-sdk-pinpoint,aws-sdk-polly,aws-sdk-pricing,aws-sdk-rds,aws-sdk-redshift,aws-sdk-rekognition,aws-sdk-resourcegroups,aws-sdk-resourcegroupstaggingapi,aws-sdk-route53,aws-sdk-route53domains,aws-sdk-s3,aws-sdk-sagemaker,aws-sdk-sagemakerruntime,aws-sdk-secretsmanager,aws-sdk-serverlessapplicationrepository,aws-sdk-servicecatalog,aws-sdk-servicediscovery,aws-sdk-ses,aws-sdk-shield,aws-sdk-signer,aws-sdk-simpledb,aws-sdk-sms,aws-sdk-snowball,aws-sdk-sns,aws-sdk-sqs,aws-sdk-ssm,aws-sdk-states,aws-sdk-storagegateway,aws-sdk-support,aws-sdk-swf,aws-sdk-transcribeservice,aws-sdk-translate,aws-sdk-waf,aws-sdk-wafregional,aws-sdk-workdocs,aws-sdk-workmail,aws-sdk-workspaces,aws-sdk-xray
502 Bad Gateway
[...]

Also, see related issue on Github: https://github.com/aws/aws-sdk-ruby/issues/1750

  1. Support Staff 1 Posted by kerrizor on 29 Sep, 2018 08:23 PM

    kerrizor's Avatar

    This is unfortunately a limitation of the old API, which currently isn't on our schedule for addressing. The issue lies, I believe, in part with the aws-sdk gem itself, and is documented here: https://github.com/aws/aws-sdk-ruby/issues/1811

    Unfortunately, at this time, this is a #wontfix on our side of things, but thank you for bringing it to our attention!

  2. kerrizor closed this discussion on 29 Sep, 2018 08:23 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac