Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers

[snemetz]

Terraform Version

Terraform v0.13.4
Also tried 0.13.3 got same issue

Terraform Configuration Files

terraform {
  required_version = ">= 0.13"
  required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}

Expected Behavior

Expected init to work

Actual Behavior

Initializing provider plugins...

• Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

Steps to Reproduce

terraform init

Additional Context

I can use curl and access the url that terraform says it can't access with no issues
I've tried multiple times over 2 days and issue doesn't change. Terraform always fails, curl always succeeds
Had no issues with 0.12, but 0.13 has not worked yet
Terraform section was the only change made going to 0.13
This is on Mac
I also tried with only the posted terraform config and still get same issue

[crosbymichael1]

Im having the same issue trying to use the helm provider:

terraform {
  required_providers {
    helm = {
      source = "hashicorp/helm"
      version = "1.3.2"
    }
  }
}

provider "helm" {

  kubernetes {
    host                   = local.kube_host
    cluster_ca_certificate = local.kube_cluster_ca_certificate_decoded
    token                  = local.kube_token
    load_config_file       = false
  }
}

---

Error Message:

`Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

• Finding latest version of hashicorp/aws...
• Finding hashicorp/helm versions matching "1.3.2"...
• Using hashicorp/aws v3.9.0 from the shared cache directory

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/helm:
no available releases match the given constraints 1.3.2
`

[pselle]

Thank you for the report! Short of more information to help replicate this issue, I've tagged this as a registry issue as perhaps the concern may lie there.

@crosbymichael1 -- it looks like your message cut of at the error message, so I'm not sure you have the same error here (the timeout).

[crosbymichael1]

@pselle I figured out what was going on

so I did a tg apply in one of the helm folders and got this error, which I thought was pretty weird because there is 100% a

---

helm provider version > 1.3
Initializing provider plugins...

• Finding hashicorp/helm versions matching "~> 1.3"...
• Finding latest version of hashicorp/aws...
• Using hashicorp/aws v3.9.0 from the shared cache directory
  Error: Failed to query available provider packages
  Could not retrieve the list of available versions for provider hashicorp/helm:
  no available releases match the given constraints ~> 1.3

---

After changing the version a bunch of times I found out that since I already had a different version of the binary in my tf plugins folder, that instead of downloading the new version, terraform just checked locally to see what I had installed, and since only 10.5 is there, well that is the only version I'd be allowed to use
/Users/michael.crosby/.terraform.d/plugins/registry.terraform.io/hashicorp/helm/10.5
So I just deleted my local helm provider folder and then did a tg apply and it downloaded the version I specified and it worked fine then

[pselle]

Thanks for the additional information @crosbymichael1! I've re-added the new label here because that gives more information that might help someone create a reproduction case for this particular angle of the issue (which could be separate from the original post, unsure, but what you've described does seem problematic, and tricky to detangle!)

[crosbymichael1]

@snemetz Try to delete your local aws file in the terraform plugins folder and then rerun terraform init and see if it resolves your issue:

rm -rf ~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws

If your changing aws provider version I have a feeling if terraform see you already have a provider installed locally then its only checking for versions within your local folder and not out in the terraform registry.

[snemetz]

That dir was empty. plugin-cache had a lot of versions. Remove all
` ls -lR
total 8
-rw-r--r-- 1 stevennemetz staff 394 Nov 16 2018 checkpoint_signature
drwxr-xr-x 3 stevennemetz staff 96 Jul 1 2019 plugin-cache
drwxr-xr-x 3 stevennemetz staff 96 Jun 21 2019 plugins

./plugin-cache:
total 0
drwxr-xr-x 2 stevennemetz staff 64 Oct 14 19:37 darwin_amd64

./plugin-cache/darwin_amd64:

./plugins:
total 0
drwxr-xr-x 2 stevennemetz staff 64 Nov 5 2019 darwin_amd64

./plugins/darwin_amd64:`

Made no change. Same error.
The error claims it is about downloading the versions file. Which should be before comparing with local files.

[snemetz]

Debug log
`
2020/10/14 19:49:59 [INFO] Terraform version: 0.13.4
2020/10/14 19:49:59 [INFO] Go runtime version: go1.14.7
2020/10/14 19:49:59 [INFO] CLI args: []string{"/usr/local/bin/terraform13.4", "init"}
2020/10/14 19:49:59 [DEBUG] Attempting to open CLI config file: /Users/stevennemetz/.terraformrc
2020/10/14 19:49:59 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins"
2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64"
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/14 19:49:59 [DEBUG] will search for provider plugins in /Users/stevennemetz/.terraform.d/plugins
2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: /Users/stevennemetz/.terraform.d/plugins is a symlink to /Users/stevennemetz/.terraform.d/plugins
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Users/stevennemetz/Library/Application Support/io.terraform/plugins
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/10/14 19:49:59 [INFO] CLI command args: []string{"init"}
2020/10/14 19:49:59 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2020/10/14 19:49:59 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2020/10/14 19:49:59 [DEBUG] New state was assigned lineage "d727669d-a920-07ec-4a64-bc51e80e8402"
2020/10/14 19:49:59 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2020/10/14 19:49:59 [TRACE] Meta.Backend: instantiated backend of type
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "."
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins"
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64"
2020/10/14 19:49:59 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/10/14 19:49:59 [TRACE] Meta.Backend: backend does not support operations, so wrapping it in a local backend
2020/10/14 19:49:59 [TRACE] backend/local: state manager for workspace "default" will:

• read initial snapshot from terraform.tfstate
• write new snapshots to terraform.tfstate
• create any backup at terraform.tfstate.backup
  2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
  2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
  2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: read nil snapshot
  2020/10/14 19:49:59 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
  2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
  2020/10/14 19:49:59 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
  2020/10/14 19:49:59 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
  2020/10/14 19:50:02 [ERR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=darwin&signature=069ea326-f2fa-49a8-980f-5a8c5830d0cb&version=0.13.4": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  2020/10/14 19:50:10 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
  2020/10/14 19:50:10 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
  2020/10/14 19:50:20 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  2020/10/14 19:50:20 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
  2020/10/14 19:50:21 [INFO] Previous request to the remote registry failed, attempting retry.
  2020/10/14 19:50:21 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
  2020/10/14 19:50:31 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  `

[techdragon]

I have this issue as well, its been happening since I switched to using any 0.13.x version. To me it looks like the new provider resolution functionality doesn't function correctly with having set a plugin_cache_dir setting.