Hey everyone,

I'm just starting using Flux + TF controller, so please let me know if I'm missing some basic step I have it running apparently properly for Kustomization, however, the tf-controller keeps hanging in Reconciliation in progress

I have created the GitRepo resource, pointing to my repository and correct path The resource was created successfully and it is in Ready state

but the Terraform resource is not, and stays forever in Reconciliation in progress

Here is my manifests: (the Terraform state is an existing one)

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: myrepo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/MyOrg/MyRepo.git
  ref:
    branch: master
  secretRef:
    name: flux-system
---
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: saas-github
  namespace: flux-system
spec:
  interval: 1m
  approvePlan: "disable"
  backendConfig:
    customConfiguration: |
      backend "s3" {
        bucket                      = "my-state-bucket"
        key                         = "my-bucket-key"
        region                      = "eu-west-1"
        dynamodb_table              = "lock-table"
        role_arn                    = "arn:aws:iam::XXXXXXX:role/role"
        encrypt                     = true
      }
  path: ./terraform/path
  sourceRef:
    kind: GitRepository
    name: myrepo
    namespace: flux-system

On the flux-system namespace I see those pods in running state

• helm-controller-xxxx
• kustomize-controller-xxxx
• notification-controller-xxxx
• saas-github-tf-runner
• source-controller-xxxxx
• tf-controller-xxxxx

And from the logs of tf-controller:

{"level":"info","ts":"2022-09-27T11:46:43.650Z","logger":"controller.terraform","msg":"getting source","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system"}
{"level":"info","ts":"2022-09-27T11:46:43.656Z","logger":"controller.terraform","msg":"trigger namespace tls secret generation","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system"}
{"level":"info","ts":"2022-09-27T11:46:43.656Z","logger":"cert-rotation","msg":"TLS already generated for ","namespace":"flux-system"}
{"level":"info","ts":"2022-09-27T11:46:43.657Z","logger":"controller.terraform","msg":"show runner pod state: ","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system","name":"saas-github","state":"running"}
{"level":"error","ts":"2022-09-27T11:47:13.657Z","logger":"controller.terraform","msg":"unable to lookup or create runner","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system","error":"context deadline exceeded"}
{"level":"error","ts":"2022-09-27T11:47:13.658Z","logger":"controller.terraform","msg":"Reconciler error","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system","error":"context deadline exceeded"}
{"level":"info","ts":"2022-09-27T11:47:13.658Z","logger":"controller.terraform","msg":"getting source","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system"}
{"level":"info","ts":"2022-09-27T11:47:13.658Z","logger":"controller.terraform","msg":"trigger namespace tls secret generation","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system"}
{"level":"info","ts":"2022-09-27T11:47:13.658Z","logger":"cert-rotation","msg":"TLS already generated for ","namespace":"flux-system"}
{"level":"info","ts":"2022-09-27T11:47:13.658Z","logger":"controller.terraform","msg":"show runner pod state: ","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"saas-github","namespace":"flux-system","name":"saas-github","state":"running"}

I see an error looking up for the runner, but I don't see any more information on why it did happen and what is missing I also looked up the documentation and I can't find more details of things that I need to setup

If I see the logs of the saas-github-tf-runner the last activity was hours ago

I0927 09:15:38.384584       7 request.go:601] Waited for 1.043722403s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s

I also saw that at the moment there's no way to set the terraform version, I wonder if due to my state using 0.13.6 it might be an issue

Any idea what I'm missing?

I have a terraform controller for Flux running with a Github provider, however, it seems to be picking up the wrong Terraform state, so it keeps trying to recreate the resources again and again (and fails because they already exist)

This is how it is configured

apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: saas-github
  namespace: flux-system
spec:
  interval: 2h
  approvePlan: "auto"
  workspace: "prod"
  backendConfig:
    customConfiguration: |
      backend "s3" {
        bucket                      = "my-bucket"
        key                         = "my-key"
        region                      = "eu-west-1"
        dynamodb_table              = "state-lock"
        role_arn                    = "arn:aws:iam::11111:role/my-role"
        encrypt                     = true
      }
  path: ./terraform/saas/github
  runnerPodTemplate:
    metadata:
      annotations:
        iam.amazonaws.com/role: pod-role
  sourceRef:
    kind: GitRepository
    name: infrastructure
    namespace: flux-system

locally running terraform init with a state.config file that has a similar/same configuration it works fine and it detect the current state properly:

bucket = "my-bucket" key = "infrastructure-github" region = "eu-west-1" dynamodb_table = "state-lock" role_arn = "arn:aws:iam::111111:role/my-role" encrypt = true

Reading the documentation I saw also a configPath that could be used, so I tried to point it to the state file, but then I got the error: Failed to initialize kubernetes configuration: error loading config file couldn't get version/kind; json parse error

Which is weird, like it tries to load Kuberntes configuration, not Terraform, or at least it expects a json file, which is not the case of my state configuration

I'm running Terraform 1.3.1 on both locally and on the tf runner pod

On the runner pod I can see the generated_backend_config.tf and it is the same configuration and .terraform/terraform.tfstate also points to the bucket

The only suspicious thing on the logs that I could find is this:

- Finding latest version of hashicorp/github...
- Finding integrations/github versions matching "~> 4.0"...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/github v5.9.1...
- Installed hashicorp/github v5.9.1 (signed by HashiCorp)
- Installing integrations/github v4.31.0...
- Installed integrations/github v4.31.0 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)
- Installing hashicorp/aws v4.41.0...
- Installed hashicorp/aws v4.41.0 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.


Warning: Additional provider information from registry

The remote registry returned warnings for
registry.terraform.io/hashicorp/github:
- For users on Terraform 0.13 or greater, this provider has moved to
integrations/github. Please update your source in required_providers.

It seems that it installs 2 github providers, one from hashicorp and one from integrations... I have changed versions of Terraform/provider during the development, and I have removed any reference to the hashicorp one, but this warning still happens

However, it also happens locally, where it reads the correct state, so I don't think it is related.

Currently we support only Kubernetes backend. But some users still require to use their existing backends like S3 or Dynamo tables outside Kubernetes.

It would be great to make the backend configurable to support those existing systems.

Here's a possible API design for custom backend.

---
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: tf-controller-ng-cc5a1ac4
  namespace: flux-system
spec:
  interval: 1h
  path: ./_artifacts/20-zz-terraform
  approvePlan: "auto"
  backend:
    customConfiguration: |
      backend "s3" {
        bucket = "mybucket"
        key    = "path/to/my/key"
        region = "us-east-1"
      }
  varsFrom:
  - kind: Secret
    name: aws-vars
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system

Health check for external resources like Redis, RDS or other managed services like Mongo Atlas requires an extra step to perform a TCP connection to ensure that the target service is already up and running.

The Flagger project has some generic way to declare this test step, from which we could learn.

Our resources maybe something like TCP services, HTTP services. Here's a health check spec proposal.

apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
spec:
  path: ./terraform
  healthChecks:
  - name: rds
    kind: TCP
    address: ${output.rds_ip_address}:3306
    timeout: 15m
  - name: nginx
    kind: HTTP
    address: https://${output.nginx_ip_address}/ping
    timeout: 15m

It would be very useful if it was possible to tell tf-controller how to perform authentication and which endpoint to use explicitly (e.g. separate CRD?), and have multiple strategies active at the same time, similar to how Crossplane does it with ProviderConfig resource.

Example use case (this is possible to do with Crossplane): I want to use the controller to create resources in both AWS and AWS China (which is a separate partition) from the same cluster. For AWS, I want to use IRSA, while for AWS China I want to use static user credentials stored in a Secret. Furthermore, in both cases I want controller to assume another role, and use that role to actually work with resources (because we have roles in different accounts, and original role/user from cluster owner's account is supposed to assume them).

In the case of using a module that is stored in a git repository (self-hosted GitLab for example), the runner requires a username and password for cloning this module. Currently, it looks like there is no such way.

Log:

{"level":"error","ts":"2022-08-29T09:44:26.811Z","logger":"controller.terraform","msg":"error in terraform setup","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"NAME","namespace":"NS","error":"error running Init: rpc error: code = Unknown desc = exit status 1
Error: Failed to download module
Could not download module \"MODULE\" (main.tf:1) source code from
"git::https://HOSTED_GIT_ADDR/MODULE_PATH.git\":
error downloading
'https://HOSTED_GIT_ADDR/MODULEPATH.git':
/usr/bin/git exited with 128: Cloning into '.terraform/modules/MODULE'...
fatal: could not read Username for 'https://HOSTED_GIT_ADDR': No such
device or address
Error: Failed to download module
Could not download module \"MODULE\" (main.tf:1) source code from
\"git::https://HOSTED_GIT_ADDR/MODULE_PATH.git\":
error downloading
'https://HOSTED_GIT_ADDR/MODULE_PATH.git':
/usr/bin/git exited with 128: Cloning into '.terraform/modules/MODULE'...
fatal: could not read Username for 'https://HOSTED_GIT_ADDR': No such
device or address"}

I want to be able to pass it an access token, username, password

Hello, I have installed the terraform controller on our kubernetes cluster using the Helm Chart, and successfully configured a gitRepository pointing to a terraform workspace. Howerver, this workspace uses the azure_rm provider to build Azure resources, which requires Azure CLI to be installed in the runner :

{"level":"error","ts":"2022-05-03T09:49:31.375Z","logger":"controller.terraform","msg":"Reconciliation failed after 16.905672785s, next try in 1m0s","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"terraform-playground-postgresql","namespace":"flux","revision":"master/8b307366408515292837f38cd8a0d97498c8f89f","error":"error running Plan: rpc error: code = Unknown desc = exit status 1\n\nError: building AzureRM Client: Please ensure you have installed Azure CLI version 2.0.79 or newer. Error parsing json result from the Azure CLI: Error launching Azure CLI: exec: \"az\": executable file not found in $PATH.\n\n  with provider[\"registry.terraform.io/hashicorp/azurerm\"],\n  on common.tf line 3, in provider \"azurerm\":\n   3: provider \"azurerm\" {\n\n"}

Is there a way to customize the runner to add the missing tools ?

Hello,

Excited with latest development of custom backends, I was trying to get azurerm backend working without a success. I have latest flux and tf-controller installed on the cluster. Here is my Terraform:

apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: tf-app
  namespace: flux-system
spec:
  approvePlan: "auto"
  backendConfig:
    customConfiguration: |
      backend "azurerm" {
        resource_group_name  = "{rg}"
        storage_account_name = "{sa_name}"
        container_name       = "tfstate"
        key                  = "{key}"
        use_msi              = true
        subscription_id      = "{sub_id}"
        tenant_id            = "{tenant_id}"
      }
  path: {path_to_tf-app}
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  disableDriftDetection: true
  alwaysCleanupRunnerPod: true
  runnerTerminationGracePeriodSeconds: 30

I am using AKS pod identity, hence the backend has use_msi = true as it would normally have. Once this is deployed, in logs tf-controller complains that:

Error loading state: the secret name tfstate-default- is invalid, a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
This is a requirement for Kubernetes secret names. 
The workspace name and key must adhere to Kubernetes naming conventions.

When setting .spec.backendConfig.secretSuffix it happily creates kubernetes backend tfstate secret, and of course no state file is stored in Storage Account.

What did I miss?

Hi Guys, I've hit an issue I can't resolve for few hours. According to #232 and #223 env vars are injected into runner pod. So I added following line to Terraform custom resource:

  runnerPodTemplate:
    spec:
      env:
        - name: VAULT_TOKEN
          valueFrom:
            secretKeyRef:
              name: vault-token
              key: token

Whenever runner start it's duty and call terraform plan I see in logs such lines and plan fails:

0804 14:25:28.653503       8 request.go:601] Waited for 1.046720264s due to client-side throttling, not priority and fairness, request: GET:https://172.16.64.1:443/apis/storage.k8s.io/v1beta1?timeout=32s
{"level":"info","ts":"2022-08-04T14:25:29.382Z","logger":"runner.terraform","msg":"preparing for Upload and Extraction"}
{"level":"info","ts":"2022-08-04T14:25:29.383Z","logger":"runner.terraform","msg":"write backend config","path":"/tmp/terraform-operator-terraform-kv-engine-weaver/live/devopsprod000/engines/key-value","config":"generated_backend_config.tf"}
{"level":"info","ts":"2022-08-04T14:25:29.383Z","logger":"runner.terraform","msg":"write config to file","filePath":"/tmp/terraform-operator-terraform-kv-engine-weaver/live/devopsprod000/engines/key-value/generated_backend_config.tf"}
{"level":"info","ts":"2022-08-04T14:25:29.384Z","logger":"runner.terraform.runner.terraform","msg":"looking for path","file":"terraform"}
{"level":"info","ts":"2022-08-04T14:25:29.384Z","logger":"runner.terraform","msg":"creating new terraform","workingDir":"/tmp/terraform-operator-terraform-kv-engine-weaver/live/devopsprod000/engines/key-value","execPath":"/usr/local/bin/terraform"}
{"level":"info","ts":"2022-08-04T14:25:29.385Z","logger":"runner.terraform","msg":"setting envvars"}
{"level":"info","ts":"2022-08-04T14:25:29.385Z","logger":"runner.terraform","msg":"getting envvars from os environments"}
{"level":"info","ts":"2022-08-04T14:25:29.386Z","logger":"runner.terraform","msg":"initializing"}
{
  "terraform_version": "1.1.9",
  "platform": "linux_amd64",
  "provider_selections": {},
  "terraform_outdated": false
}

Initializing the backend...

Successfully configured the backend "kubernetes"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding latest version of hashicorp/vault...
- Installing hashicorp/vault v3.8.0...
- Installed hashicorp/vault v3.8.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!
{"level":"info","ts":"2022-08-04T14:25:30.242Z","logger":"runner.terraform","msg":"setting up the input variables"}
{"level":"info","ts":"2022-08-04T14:25:30.243Z","logger":"runner.terraform","msg":"mapping the Spec.Vars"}
{"level":"info","ts":"2022-08-04T14:25:30.243Z","logger":"runner.terraform","msg":"mapping the Spec.VarsFrom"}
{"level":"info","ts":"2022-08-04T14:25:30.257Z","logger":"runner.terraform","msg":"creating a plan"}

Error: no vault token found

  with provider["registry.terraform.io/hashicorp/vault"],
  on main.tf line 1, in provider "vault":
   1: provider "vault" {

{"level":"error","ts":"2022-08-04T14:25:30.799Z","logger":"runner.terraform","msg":"error creating the plan","error":"exit status 1\n\nError: no vault token found\n\n  with provider[\"registry.terraform.io/hashicorp/vault\"],\n  on main.tf line 1, in provider \"vault\":\n   1: provider \"vault\" {\n\n"}
{"level":"info","ts":"2022-08-04T14:25:30.800Z","logger":"runner.terraform","msg":"cleanup TmpDir","tmpDir":"/tmp/terraform-operator-terraform-kv-engine-weaver"}

If I exec into the runner pod and call:

echo $VAULT_TOKEN

I can see correct value. Also calling manually terraform init, plan and apply from the pod context works good!

Therefore, I'm in the opinion that env var is indeed injected into the pod, but NOT into the terraform process/binary that is triggered as no 1 process (tini).

I'm using:

• helm chart tf-controller version 0.3.8
• terraform 1.2.2
• flux2 version 0.31.5
• kubernetes AKS 1.22.6

Happy to hear your thoughts and tips Guys!

thanks Tomek

I am not really sure if i am just to stupid, but i keep getting the following message:

{"level":"error","ts":"2022-03-15T21:26:01.284Z","logger":"controller.terraform","msg":"Reconciliation failed after 3.886180395s, next try in 1m0s","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"helloworld-3","namespace":"flux-system","revision":"main/ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4","error":"failed to verify artifact: computed checksum 'a178d6af210cb22825e40a879dc892bf7099091e2bd2ee5ad9697a7b12409aa2' doesn't match advertised 'd2c7fe7ea7e54c2f695c191ff4c3131567389f3c7a6e18a9d422ccc80ba6ed18'"}

The Terraform looks like this:

apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: helloworld-3
  namespace: flux-system
spec:
  approvePlan: "auto"
  path: ./services/harbor/tenant/
  interval: "1m"
  sourceRef:
    kind: GitRepository
    name: tf-catalog
    namespace: flux-system
  vars:
  - name: password
    value: "none"
  - name: project
    value: "example"

The gitRepository status seems to be fine as well:

status:
  artifact:
    checksum: d2c7fe7ea7e54c2f695c191ff4c3131567389f3c7a6e18a9d422ccc80ba6ed18
    lastUpdateTime: "2022-03-15T21:09:38Z"
    path: gitrepository/flux-system/tf-catalog/ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4.tar.gz
    revision: main/ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/tf-catalog/ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4.tar.gz
  conditions:
  - lastTransitionTime: "2022-03-15T21:09:38Z"
    message: 'Fetched revision: main/ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4'
    reason: GitOperationSucceed
    status: "True"
    type: Ready
  observedGeneration: 1
  url: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/tf-catalog/latest.tar.gz

I downloaded the ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4.tar.gz and got the same checksum trough sha256sum:

$ sha256sum ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4.tar.gz

d2c7fe7ea7e54c2f695c191ff4c3131567389f3c7a6e18a9d422ccc80ba6ed18  ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4.tar.gz

I believe this happens because I have to set a proxy (bcs of company stuff). So then I add the following environment variables:

 env:
        - name: HTTP_PROXY
          value: http://company.proxy
        - name: HTTPS_PROXY
          value: http://company.proxy
        - name: NO_PROXY
          value: localhost,127.0.0.1,.cluster.local,10.0.0.0/8

That's the only thing that differs from the default installation. But if i remove those, I get the following:

{"level":"info","ts":"2022-03-15T21:35:09.000Z","logger":"controller.terraform","msg":"new terraform","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"helloworld-3","namespace":"flux-system","workingDir":"/tmp/flux-system-helloworld-3-227125539/services/harbor/tenant"}
{"level":"info","ts":"2022-03-15T21:35:09.216Z","logger":"controller.terraform","msg":"clean up dir: ok","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"helloworld-3","namespace":"flux-system"}
{"level":"error","ts":"2022-03-15T21:35:09.228Z","logger":"controller.terraform","msg":"Reconciliation failed after 14.522177049s, next try in 1m0s","reconciler group":"infra.contrib.fluxcd.io","reconciler kind":"Terraform","name":"helloworld-3","namespace":"flux-system","revision":"main/ece7cf37ed2bde070e2e3117bb25dcf04ae82fa4","error":"error running Init: rpc error: code = Unknown desc = exit status 1\n\nError: Failed to query available provider packages\n\nCould not retrieve the list of available versions for provider\nbestseller/harbor: could not connect to registry.terraform.io: Failed to\nrequest discovery document: Get\n\"https://registry.terraform.io/.well-known/terraform.json\": dial tcp\n-:443: connect: connection refused\n\n"}

That looks already better to me, meaning it doesn't fail at validating the artefact. Obviously I can't connect to the outside world. Has anyone an idea why the set proxy would change the checksum?

Using TF-controller's getting started command for installing TF-controller is not working.

To reproduce:

kind create cluster && flux install && kubectl apply -f https://raw.githubusercontent.com/weaveworks/tf-controller/main/docs/release.yaml 

The TF-controller will fail with this error in it log:

unknown flag: --runner-grpc-max-message-size
unknown flag: --runner-grpc-max-message-size
Usage of tf-controller:
      --ca-cert-validity-duration duration        The duration that the ca certificate certificates should be valid for. Default is 1 week. (default 168h0m0s)
      --cert-rotation-check-frequency duration    The interval that the mTLS certificate rotator should check the certificate validity. (default 30m0s)
      --cert-validity-duration duration           (Deprecated) The duration that the mTLS certificate that the runner pod should be valid for. (default 6h0m0s)
      --concurrent int                            The number of concurrent terraform reconciles. (default 4)
      --enable-leader-election                    Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
      --events-addr string                        The address of the events receiver.
      --health-addr string                        The address the health endpoint binds to. (default ":9440")
      --http-retry int                            The maximum number of retries when failing to fetch artifacts over HTTP. (default 9)
      --kube-api-burst int                        The maximum burst queries-per-second of requests sent to the Kubernetes API. (default 100)
      --kube-api-qps float32                      The maximum queries-per-second of requests sent to the Kubernetes API. (default 50)
      --leader-election-lease-duration duration   Interval at which non-leader candidates will wait to force acquire leadership (duration string). (default 35s)
      --leader-election-release-on-cancel         Defines if the leader should step down voluntarily on controller manager shutdown. (default true)
      --leader-election-renew-deadline duration   Duration that the leading controller manager will retry refreshing leadership before giving up (duration string). (default 30s)
      --leader-election-retry-period duration     Duration the LeaderElector clients should wait between tries of actions (duration string). (default 5s)
      --log-encoding string                       Log encoding format. Can be 'json' or 'console'. (default "json")
      --log-level string                          Log verbosity level. Can be one of 'trace', 'debug', 'info', 'error'. (default "info")
      --metrics-addr string                       The address the metric endpoint binds to. (default ":8080")
      --requeue-dependency duration               The interval at which failing dependencies are reevaluated. (default 30s)
      --runner-creation-timeout duration          Timeout for creating a runner pod. (default 2m0s)
      --runner-grpc-port int                      The port which will be exposed on the runner pod for gRPC connections. (default 30000)
      --watch-all-namespaces                      Watch for custom resources in all namespaces, if set to false it will only watch the runtime namespace. (default true)

This lead to invalid YAML because when set the resulting labels would format to something like this:

labels:
  customlabel: test
    selectorlabel: test

Hi everyone,

I'm currently testing this controller which is awesome because I'm a big fan of Flux generally speaking. I'm able to create simple resources and it's pretty straightforward. However, for my demo I'd like to create an EKS cluster and then install Flux and configure a few things. For that I need to configure the kubernetes provider. Without tf-controller I would have done that as follows

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args = [
      "eks",
      "get-token",
      "--cluster-name",
      var.cluster_name
    ]
  }
}

However I'm not sure that's even possible. The error I get is pretty clear but I want to know what is your recommendation please.

Error: Post "https://1B7085B8CBDF07A3521F20EE16512FCC.gr7.eu-west-3.eks.amazonaws.com/api/v1/namespaces": getting credentials: exec: executable aws not found

Thank you,

Hello,

There are scenarios where we want to run the same state plan from multiple k8s clusters, for example:

• All dev->prod clusters create/manage the same Sentry project with the same config
• Multi-cluster scenario where there are two or more prod clusters manage the same infra with the same config

However we sometimes run into state-lock errors, where timings of plans clash, and plan that started second cannot run due to the first plan locking the state.

What do you think about adding an option to set -lock=false on the plan stage?

Hi Team, We started to notice weird behaviour where many locks were not released when the tf-runners were killed, even when the graceful shutdown was set to 1 hour. After some investigation, we noticed that when the pod was restarted, the SIGKILL forcefully killed the Terraform process; therefore, it doesn’t start a graceful shutdown, which sometimes ends with the state lock not being released.

We found these related issues: https://github.com/hashicorp/terraform-exec/pull/332 https://github.com/hashicorp/terraform-exec/pull/334

logs:

2022-11-29T21:42:10.266800268Z		Error message: workspace already locked (lock ID:
2022-11-29T21:42:10.266795895Z		
2022-11-29T21:42:10.266789968Z		Error: Error acquiring the state lock
2022-11-29T21:42:10.266760286Z		
2022-11-29T21:42:07.606685256Z	info	creating a plan
2022-11-29T21:42:07.587542369Z	info	mapping the Spec.VarsFrom
2022-11-29T21:42:07.587537646Z	info	mapping the Spec.Vars
2022-11-29T21:42:07.587512719Z	info	setting up the input variables
2022-11-29T21:42:07.58182462Z		Terraform has been successfully initialized!
2022-11-29T21:42:07.581792524Z		
2022-11-29T21:42:07.581420214Z		you run "terraform init" in the future.
2022-11-29T21:42:07.581416395Z		so that Terraform can guarantee to make the same selections by default when
2022-11-29T21:42:07.58141258Z		selections it made above. Include this file in your version control repository
2022-11-29T21:42:07.581408747Z		Terraform has created a lock file .terraform.lock.hcl to record the provider
2022-11-29T21:42:07.581401961Z		
2022-11-29T21:42:00.820815414Z		Initializing provider plugins...
2022-11-29T21:42:00.820797552Z		
2022-11-29T21:42:00.481037098Z		use this backend unless the backend configuration changes.
2022-11-29T21:42:00.481032663Z		Successfully configured the backend "remote"! Terraform will automatically
2022-11-29T21:42:00.48101017Z		
2022-11-29T21:42:00.198431966Z		Initializing the backend...
2022-11-29T21:41:58.413545513Z		Upgrading modules...
2022-11-29T21:41:58.386167971Z		}
2022-11-29T21:41:58.38616475Z		  "terraform_outdated": true
2022-11-29T21:41:58.386161487Z		  "provider_selections": {},
2022-11-29T21:41:58.386157733Z		  "platform": "linux_amd64",
2022-11-29T21:41:58.386153382Z		  "terraform_version": "1.1.9",
2022-11-29T21:41:58.386133029Z		{
2022-11-29T21:41:58.251028908Z	info	mapping the Spec.BackendConfigsFrom
2022-11-29T21:41:58.249567118Z	info	initializing
2022-11-29T21:41:58.245264737Z	info	getting envvars from os environments
2022-11-29T21:41:58.24525401Z	info	setting envvars
2022-11-29T21:41:58.241270762Z	info	creating new terraform
2022-11-29T21:41:58.237230706Z	info	looking for path
2022-11-29T21:41:58.229338345Z	info	processing configuration
2022-11-29T21:41:58.225878148Z	info	write config to file
2022-11-29T21:41:58.225855343Z	info	write backend config
2022-11-29T21:41:58.223764885Z	info	preparing for Upload and Extraction
2022-11-29T21:36:25.886572001Z	info	cleanup TmpDir

2022-11-29T21:36:25.88396556Z	error	error creating the plan
2022-11-29T21:36:16.127120371Z	info	creating a plan

events:

2022-11-29 21:36:26	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Stopping container tf-runner
2022-11-29 21:36:26	Warning	Terraform	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d	error running Plan: rpc error: code = Internal desc = signal: killed
2022-11-29 21:36:25	Warning	Terraform	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d	Plan error: rpc error: code = Internal desc = signal: killed
2022-11-29 21:35:54	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Started container tf-runner
2022-11-29 21:35:53	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Created container tf-runner
2022-11-29 21:35:53	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Container image "ghcr.io/weaveworks/tf-runner:v0.13.0-rc.1" already present on machine
2022-11-29 21:35:51	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Successfully assigned terraform-runners/env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner
2022-11-29 21:35:46	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Successfully assigned terraform-runners/env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner
2022-11-29 21:35:46	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Stopping container tf-runner
2022-11-29 21:35:46	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Started container tf-runner
2022-11-29 21:35:46	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Created container tf-runner
2022-11-29 21:35:46	Normal	Pod	env-76461b07-9216-4d60-8a43-c5a30ddfdd8d-tf-runner	Container image "ghcr.io/weaveworks/tf-runner:v0.13.0-rc.1" already present on machine

Is there something we can do to avoid this behaviour without setting force unlock to auto?

Thanks

#336

With this PR, we can store terraform plans in Kubernetes PersistentVolumeClaim.
The given PVC should exist already and is not part of the controller's responsibility.

• Also supports all current options of storeReadablePlan

The runner will mount into a subPath that points to its own directory. The path looks like this:
/mnt/plan/terraform_controller/TERRAFORM_OBJECT_NAMESPACE/TERRAFORM_OBJECT_NAME/
This gives the security of a runner to mount only into its own object and also takes care the backend storage is organized and can serve other stuff then the terraform controller.

Example:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: plan-config-claim-pv
  labels:
    type: local
spec:
  storageClassName: standard
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/plan-test"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: plan-config-claim-pvc
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: helloworld-plan-config-claim
spec:
  interval: 10s
  approvePlan: "auto"
  storeReadablePlan: "human"
  path: ./
  planConfig:
    storage:
      claimName: "plan-config-claim-pvc"
  sourceRef:
    kind: GitRepository
    name: helloworld

Currently, Terraform objects contain a reference to a serviceAccount, an entity which must be present in order to proceed with the object.

This reference is currently just a string reference to a serviceAccount object in the Terraform objects namespace: https://github.com/weaveworks/tf-controller/blob/a4d31bb311e7d405bb9e0d3b357b4bb7766927b5/api/v1alpha1/terraform_types.go#L179-L183

Can this be switched to a CrossNamespaceReference, similar to SourceRef? https://github.com/weaveworks/tf-controller/blob/a4d31bb311e7d405bb9e0d3b357b4bb7766927b5/api/v1alpha1/terraform_types.go#L135-L137

In our current use-case, we are creating Terraform objects in a number of different namespaces, based on permissions, and to escape this limitation we are creating serviceAccounts with the correct ClusterRoleBinding in every namespace we create, but ideally I'd like to just point it to the service account that the TFController creates, so that we don't have any long-term compatibility issues (if permissions in the serviceAccount change).

What do you think?