Resource link aws_glue_catalog_table produces the error Error: error setting partition_keys: Invalid address to set: []string{"partition_keys", "0", "parameters"}

[CarlinCyril]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.1
on linux_amd64

• provider registry.terraform.io/hashicorp/aws v4.29.0

Affected Resource(s)

*aws_glue_catalog_table

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_glue_catalog_table" "aws_glue_catalog_table" {
  database_name = "database_name"
  name          = "table_name"

  target_table {
    catalog_id    = "target_catalog_id"
    database_name = "target_database_name"
    name          = "target_table_name"
  }
}

Expected Behavior

The command terraform apply should go through without any error and the aws_glue_catalog_table resources should be created

Actual Behavior

When we try to execute the command terraform apply the following error appears:

Error: error setting partition_keys: Invalid address to set: []string{"partition_keys", "0", "parameters"}

However the resource is correctly created and the TF state is updated. But after that, any of terraform plan and terraform apply fails completely because of this error. Making it impossible to either delete the aws_glue_catalog_table resource or do anything else.

Steps to Reproduce

1. terraform apply

Potential Cause

The structure of the block partition_keys in catalog_table.go is defined as follow:

"partition_keys": {
Type:     schema.TypeList,
Optional: true,
Elem: &schema.Resource{
	Schema: map[string]*schema.Schema{
		"comment": {
			Type:         schema.TypeString,
			Optional:     true,
			ValidateFunc: validation.StringLenBetween(0, 255),
		},
		"name": {
			Type:         schema.TypeString,
			Required:     true,
			ValidateFunc: validation.StringLenBetween(1, 255),
		},
		"type": {
			Type:         schema.TypeString,
			Optional:     true,
			ValidateFunc: validation.StringLenBetween(0, 131072),
		},
	},
},
}

While the incoming struct of PartitionKeys is a glue.Column type defined as follow (from "github.com/aws/aws-sdk-go/service/glue" package):

// A column in a Table.
type Column struct {
	_ struct{} `type:"structure"`

	// A free-form text comment.
	Comment *string `type:"string"`

	// The name of the Column.
	//
	// Name is a required field
	Name *string `min:"1" type:"string" required:"true"`

	// These key-value pairs define properties associated with the column.
	Parameters map[string]*string `type:"map"`

	// The data type of the Column.
	Type *string `type:"string"`
}

As you can see, there is the "parameters" attribute missing from the structure in catalog_table.go. That's why the error occurs when we try to execute this block of code (catalog_table.go @ line 424 - 426):

if err := d.Set("partition_keys", flattenColumns(table.PartitionKeys)); err != nil {
    return fmt.Errorf("error setting partition_keys: %w", err)
}

Indeed after adding the attribute parameters to the struct as follow

"parameters": {
	Type:     schema.TypeMap,
	Optional: true,
	Elem:     &schema.Schema{Type: schema.TypeString},
},

...Everything works as expected.

[gelphilhowe]

Looks like the PR above was never merged - I'm still getting this error in 5.54.1. Worse, it seems to affect some operations but not others, so I managed to successfully deploy tables in a terraform workspace on a feature branch but once merged, it failed to deploy (failing at the plan stage) to our canonical test environment so we'll have to go back and re-think how to create our Glue tables.

I'm not a golang dev so I'm not the right person to advance this - wondering if one of the maintainers/regular contributors could take this on as it's still causing trouble and seems to be a somewhat straightforward fix? Would appreciate any prioritisation that could be given here as it'd make our lives a lot easier!

[anuradhawick]

I have the same problem. Especially on subsequent deployments of terraform

[gelphilhowe]

Incidentally, for others having this issue, we ended up using CloudFormation as a workaround. Obviously not ideal as you lose nice terraform lifecycle things, and you can't import Glue tables into CloudFormation for some reason, but at least it brings the tables into IaC.

Example CloudFormation terraform here:

resource "aws_cloudformation_stack" "table" {
  name = "GlueTableStack"

  parameters = {
    DatabaseName = var.db_name
    TableName    = var.table_name
  }

  template_body = jsonencode({
    Parameters = {
      DatabaseName = {
        Type = "String"
      }
      TableName = {
        Type = "String"
      }
    }
    Resources = {
      ThisGlueTable = {
        Type = "AWS::Glue::Table"
        Properties = {
          CatalogId    = data.aws_caller_identity.current.account_id
          DatabaseName = { Ref = "DatabaseName" }
          TableInput = {
            Name = { Ref = "TableName" }
            Parameters = {
              EXTERNAL              = "TRUE"
              numFiles              = "-1"
              ...

            }
            TableType = "EXTERNAL_TABLE"
            PartitionKeys = [
              {
                Name = "partition_key_name"
                Type = "string"
              },
              ...
            ]
            StorageDescriptor = {
              Compressed             = false
              InputFormat            = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"
              Location               = "s3://${var.bucket_name}/${var.db_name}/${var.table_name}/"
              OutputFormat           = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"
              StoredAsSubDirectories = false
              Columns = [
                {
                  Name = "some_column"
                  Type = "string"
                },
                {
                  Name = "another_column"
                  Type = "string"
                },
                ...
              ]
              SerdeInfo = {
                SerializationLibrary = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
              }
            }
          }
        }
        Tags = [
          for key, value in var.tags_dictionary :
          {
            Key   = key
            Value = value
          }
        ]
      }
    }
  })
}

[github-actions]

Warning

This Issue has been closed, meaning that any additional comments are much easier for the maintainers to miss. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.