insecure_kubelet_readonly_port_enabled does not work on the cluster level via google_container_cluster

[twingate-blee]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.2
on darwin_arm64

• provider registry.terraform.io/carlpett/sops v1.1.1
• provider registry.terraform.io/hashicorp/google v6.3.0
• provider registry.terraform.io/hashicorp/google-beta v6.4.0
• provider registry.terraform.io/hashicorp/null v3.2.3
• provider registry.terraform.io/hashicorp/random v3.6.3
• provider registry.terraform.io/hashicorp/tls v4.0.6
• provider registry.terraform.io/mrolla/circleci v0.6.1
• provider registry.terraform.io/opsgenie/opsgenie v0.6.37
• provider registry.terraform.io/twingate/twingate v3.0.11
• provider registry.terraform.io/vancluever/acme v2.26.0

Affected Resource(s)

google_container_cluster

Terraform Configuration

resource "google_container_cluster" "app_cluster" {
  ...
  remove_default_node_pool    = true
  ...
  node_config {
    kubelet_config {
      cpu_manager_policy                     = "none"
      insecure_kubelet_readonly_port_enabled = "FALSE"
    }
  }
  ...

Debug Output

No response

Expected Behavior

Terraform should not error and insecureKubeletReadonlyPortEnabled should be set to False

gcloud container clusters describe <name> \
    --location=<location> \
    --flatten=nodePoolDefaults.nodeConfigDefaults \
    --format="value(nodeKubeletConfig)"
insecureKubeletReadonlyPortEnabled=False

Actual Behavior

│ Error: googleapi: Error 400: Node pool "default-pool" not found on update.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.RequestInfo",
│     "requestId": "0x46b14d914594c365"
│   }
│ ]
│ , badRequest

We do not use the "default-pool". Node pools created using google_container_node_pool

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

[wyardley]

Not a Google employee, but I did add this feature. The setting can be set in a bunch of places. While I agree this behavior is confusing and not ideal, see the notes in the provider docs about node_config.

Generally, this field should not be used at the same time as a google_container_node_pool or a node_pool block; this configuration manages the default node pool, which isn't recommended to be used with Terraform

Also, looking at your example, you're looking for nodeConfigDefaults, which is in the node_pool_defaults block, not the node_config.kubelet_config block.

So, if you're using remove_default_node_pool = true, you will need to set it for each pool separately. This may not line up with your exact use case, but basically:

resource "google_container_cluster" "app_cluster" {
  ...
  remove_default_node_pool = true

  node_pool_defaults {
    node_config_defaults {
      insecure_kubelet_readonly_port_enabled = "FALSE"
    }
  }
}

resource "google_container_node_pool" "your_node_pool" {
  name = "your-node-pool"
  cluster    = google_container_cluster.app_cluster.id
  node_count = 1

  node_config {
    kubelet_config {
      insecure_kubelet_readonly_port_enabled = "FALSE"
    }
  }
}

HTH

[wyardley]

All that said, it might be a good idea (if it's possible) for the provider team to add some kind of provider level validation / erroring if someone tries to set node_config when remove_default_node_pool is set, but I suspect that removing it should resolve the issue.