What happened?

I tried to generate a Terraform provider in the repo https://github.com/zzxwill/provider-jet-alibaba and it failed.

➜  /Users/zhouzhengi/Programming/golang/src/github.com/zzxwill/provider-jet-alibaba git:(main) ✗ make generate
16:42:06 [ .. ] verify dependencies have expected content
all modules verified
16:42:21 [ OK ] go modules dependencies verified
16:42:22 [ .. ] go generate darwin_amd64
go: downloading github.com/turkenh/terraform-plugin-sdk v1.17.2-patch1
# github.com/hashicorp/terraform-plugin-sdk/internal/configs/configload
../.work/pkg/pkg/mod/github.com/turkenh/[email protected]/internal/configs/configload/loader_snapshot.go:52:29: cannot use fs (type snapshotFS) as type afero.Fs in argument to configs.NewParser:
	snapshotFS does not implement afero.Fs (missing Chown method)
../.work/pkg/pkg/mod/github.com/turkenh/[email protected]/internal/configs/configload/loader_snapshot.go:57:28: cannot use fs (type snapshotFS) as type afero.Fs in field value:
	snapshotFS does not implement afero.Fs (missing Chown method)
../.work/pkg/pkg/mod/github.com/turkenh/[email protected]/internal/configs/configload/loader_snapshot.go:200:5: cannot use snapshotFS{} (type snapshotFS) as type afero.Fs in assignment:
	snapshotFS does not implement afero.Fs (missing Chown method)
apis/generate.go:32: running "go": exit status 2
16:42:39 [FAIL]
make[1]: *** [go.generate] Error 1
make: *** [generate] Error 2

How can we reproduce it?

1. Clone https://github.com/zzxwill/provider-jet-alibaba.git
2. run make generate

What happened?

Trying to install provider after building and pushing to dockerhub

kubectl crossplane install provider ajwportalteam/provider-jet-azuread-amd64:v0.1.0

But providerrevision object reports

Did I miss a step in the guide? make run is fine

How can we reproduce it?

Use same image / source at https://github.com/ajwgroup/provider-jet-azuread

Im trying to convert ldap provider(https://registry.terraform.io/providers/Ouest-France/ldap/latest/docs) there: https://github.com/wNvSp1r1T/terraform-jet-ldap/commit/8dfc0a9c1c41559ce91e0bacf25d4340c9b7b85c

And i'm always get error: "summary":"LDAP Result Code 34 \"Invalid DN Syntax\" 1.664437610235876e+09 DEBUG provider-jet-ldap Cannot observe external resource {"controller": "managed/ldapgroup.ldap.jet.crossplane.io/v1alpha1, kind=group", "request": "/hello-crossplane", "uid": "574384b6-1ec8-4bd9-9f1c-4b08821b7adb", "version": "208755", "external-name": "hello-crossplane", "error": "cannot run refresh: readObjectStart: expect { or n, but found 2, error found in #1 byte of ...

my examples/providerconfig/providerconfig.yaml is default one my examples/providerconfig/secret.yaml is

    {
      "host": "ldap.ru",
      "port": "389",
      "bind_user": "bind_user",
      "bind_password": "test-ldap"
    }

my examples/ldapgroup/ldapgroup.yaml is:

kind: Group
metadata:
  name: hello-crossplane
spec:
  forProvider:
    description: "Managed with Crossplane ldap Provider (generated with Terrajet)"
    ou: "OU=test,OU=sre,OU=Resources,OU=Main,DC=X,DC=eu
  providerConfigRef:
    name: default

I will be grateful for your help

What problem are you facing?

Common controller works for our prototypes but we need to do more experiments to see its scalability. There are certain issues that need some data before taking action, like https://github.com/crossplane-contrib/terrajet/issues/38

How could Terrajet help solve your problem?

We can have a complex composition with 10+ resources and create 10 XRs using that. Then check the resource usage and errors across the board to see if we hit any limit. We can define some limits in the deployment of the controller and see at what point we start to see context deadline exceeded errors, meaning we can't complete a single reconciliation pass with those limits.

What problem are you facing?

Right now we have a single provider binary and all terraform invocations use that one but each of them spins up a new provider server to talk to. We haven't observed noticeable issues but it's likely that this will cause performance issues.

How could Terrajet help solve your problem?

@ulucinar discovered a way to make Terraform CLI talk to the existing provider server. He didn't feel great about it for the initial pass but we might need to reconsider it after testing the providers with 100+ custom resources using composition.

Description of your changes

Fixes #232

With this PR, configuring blocks with optional parameters will be possible. In terraform schemas, this logic works with a field: ConfigMode. If the value of the ConfigMode field is SchemaConfigModeAttr, we have to provide values for all options. Please see the details: https://stackoverflow.com/questions/69079945/terraform-inappropriate-value-for-attribute-ingress-while-creating-sg/69080432#69080432

Reference document: https://www.terraform.io/language/attr-as-blocks

In this PR, a new check was added for tfTag. If the parent field is in the SchemaConfigModeAttr, the omitempty tag is deleted from tftags of subfields.

I have:

• [x] Read and followed Crossplane's contribution process.
• [x] Run make reviewable to ensure this PR is ready for review.
• [x] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

This PR was tested in provider-jet-aws side by provisioning, updating and deleting SecurityGroup resource. In tests the manifest was used:

apiVersion: ec2.aws.jet.crossplane.io/v1alpha2
kind: SecurityGroup
metadata:
    name: example-securitygroup
spec:
  forProvider:
    name: example-securitygroup-sg
    description: "i am a description"
    region: us-west-2
    vpcId: vpc-somevpc
    ingress:
    - fromPort: 6379
      toPort: 6379
      protocol: tcp
      description: "i am also a description"
      cidrBlocks:
      - 10.0.0.0/16
      ipv6CidrBlocks: []
      self: false
      prefixListIds: []
      securityGroups: []
    tags:
      managed-by: crossplane
  providerConfigRef:
    name: jet-aws-provider

Please see the original issue in jet-aws side: https://github.com/crossplane-contrib/provider-jet-aws/issues/157

What problem are you facing?

We have produced a shared gRPC server-based implementation for provider-jet-azure in the context of https://github.com/crossplane/terrajet/issues/38. The provider-jet-azure packages ulucinar/provider-jet-azure-arm64:shared-grpc and ulucinar/provider-jet-azure-amd64:shared-grpc are modified to run the terraform-provider-azurerm binary plugin in the background as a shared gRPC server and the Terraform CLI does not have to fork the binary plugin for each of its requests.

How could Terrajet help solve your problem?

Similar to what we have done previously in https://github.com/crossplane/terrajet/issues/55, we need to reevaluate the performance of [email protected] and also the shared gRPC implementation using the above provider packages. This will allow us to assess and quantify any performance improvements with the shared gRPC implementation. Some of the previously used scripts for #55 are available in https://github.com/ulucinar/terrajet-scale.

Description of your changes

Fixes #175 in the scope of #131

I have:

• [x] Read and followed Crossplane's contribution process.
• [x] Run make reviewable to ensure this PR is ready for review.
• [ ] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

See the provider-gcp schema generated using the conversion here: https://github.com/crossplane-contrib/provider-jet-gcp/pull/38/

Tested with provider-github with the updated jet template in this PR: https://github.com/crossplane-contrib/provider-jet-template/pull/13

Description of your changes

This PR abstracts the terrajet code running generation pipelines from tf providers. With the changes here, terrajet based providers would only need to provide a Provider configuration (with resource-specific configurations) and Terrajet will take care of the rest.

This PR also makes some (breaking) changes to interact with Terrajet aiming for a more flexible (and hopefully more stable) interface.

Finally adds support for generating Terraform Providers depending on an old version (<v2) of Terraform plugin SDK.

Fixes #61 Fixes #129

I have:

• [x] Read and followed Crossplane's contribution process.
• [x] Run make reviewable to ensure this PR is ready for review.

How has this code been tested

Tested with autoscaling and ec2 groups in provider-tf-aws using this PR:

1. Run make generate
2. Verify that there is no unexpected changes in the generated code.

Description of your changes

Changes the bookkeeping of the operations from xp.lock and .store files to a global map of operations WorkspaceStore. In addition, there are a couple of things in this PR that's not directly related to bookeeping but help the flow. Main changes are as following:

• New WorkspaceStore to hold all Workspace objects that is created per CR instance, identified by their UUIDs.
• Handles Terraform crashes by removing Terraform state lock file if the last operation finished (crash, error or success).
• It spawns a new go routine for each async execution and runs the given callback once finished.
• The callback of Apply operation is to update the annotations to make sure we capture the server-side computed IDs as soon as possible and get enqueued.
• Introduces new optional synchronous versions of Apply and Destroy operations for resources whose creation take short time, letting us save the state as fast as it can be.
• Assumes that if terraform destroy completes successfully, the resource is gone so that we don't have to implement custom logic to handle resources whose read calls find the resource even after it's actually deleted (KMS keys, some configuration CRs)
• Makes the TF client such that you don't have to call ApplyAsync again to poll its results, allowing us to use the latest crossplane-runtime.
• [TBD] Moves the adapter into the ExternalClient implementation directly. There is only a single interface now, similar to native providers.
• Reorganizes the folder structure to be similar to native providers, i.e. terraform is client package and controller is the managed reconciler implementation package.
• Since the tracking is in-memory start/stop of the controller does not end up in a locked state.

This PR is mostly accumulation of our collective experience testing the controllers during implementation of features during the initial push that we released as tech preview.

Special shoutout to @ulucinar who had to deal with this complex beast in the initial rush. The state handling logic he implemented is still the core logic with only some refactoring that incorporated our understanding of the whole thing.

It's ready for review but there are TODOs before merging:

• Replace osfile calls with afero for easier testing.
• Unit tests for controller.
• Unit tests for terraform package.

Fixes https://github.com/crossplane-contrib/terrajet/issues/74 Fixes https://github.com/crossplane-contrib/terrajet/issues/39 Fixes https://github.com/crossplane-contrib/terrajet/issues/22 Fixes https://github.com/crossplane-contrib/terrajet/issues/21

I have:

• [x] Read and followed Crossplane's contribution process.
• [x] Run make reviewable to ensure this PR is ready for review.
• [x] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

I have tested it in both async and sync mode using provider-tf-aws. The example YAMLs will be there. Additionally, I tested with stopping/starting the controller and it was able to continue without any errors.

Sync:

apiVersion: vpc.aws.tf.crossplane.io/v1alpha1
kind: Vpc
metadata:
  name: sample-vpc
spec:
  forProvider:
    region: us-west-1
    cidrBlock: 10.0.0.0/16
    tagsAll:
      Name: muvaftest
    tags:
      Name: muvaftest
  providerConfigRef:
    name: example

It works just like other providers. There are a few gotchas. For example, after the deletion is completed we requeue immediately but for some reason that version of custom resource is old, hence, the finalizer removal Update fails but at that point we removed the directory. So, in the next reconcile we init again, check existence and then finalizer is removed successfully. Though this looks like a general problem that should be addressed in controller-runtime or crossplane-runtime. Other than that, I haven't caught an obvious problem.

Async:

apiVersion: rds.aws.tf.crossplane.io/v1alpha1
kind: RdsCluster
metadata:
  name: sample-cluster
spec:
  forProvider:
    region: us-west-1
    engine: aurora-postgresql
    masterUsername: cpadmin
    masterPassword: testPass23!
    skipFinalSnapshot: true
  providerConfigRef:
    name: example

This one also works as expected. One thing different regarding the async mode is that we introdue a new condition to report the last operation status. This would come in the schema of the resource for other providers but since we control the client, we return it as part of RefreshResult and put it under a new condition separately.

One thing I don't love about async mode is that even if the operation takes 5 seconds due to an error, we have to wait for a full poll interval in order to be aware of the error and I couldn't find a way to manually enqueue the resource. Same thing happens with native provider async resources as well but they usually do a client side validation with their async Create call.

I have a hunch that we might be able to get some simplification if we separate sync/async into two ExternalClient implementations but I held that off for now.

Description of your changes

Introduces late-initialization for Terraformed resources. Copies late-initialization library from provider-aws pkg/controller/cloudfront/ package.

Fixes #10

I have:

• [X] Read and followed Crossplane's contribution process.
• [X] Run make reviewable to ensure this PR is ready for review.
• [ ] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

I have manually tested this PR with provider-tf-azure. When a VirtualNetwork MR with the following manifest:

apiVersion: virtual.azure.tf.crossplane.io/v1alpha1
kind: VirtualNetwork
metadata:
  name: example
spec:
  deletionPolicy: Orphan
  forProvider:
    addressSpace:
    - 10.0.0.0/16
    dnsServers:
    - 10.0.0.1
    - 10.0.0.2
    - 10.0.0.3
    location: East US
    name: importDemo
    resourceGroupName: alper
    tags:
      purpose: demo
      key1: value1
  providerConfigRef:
    name: example

is provisioned and then deleted, and after a new VirtualNetwork MR is imported using the following manifest:

apiVersion: virtual.azure.tf.crossplane.io/v1alpha1
kind: VirtualNetwork
metadata:
  name: imported
  annotations:
    crossplane.io/external-name: /subscriptions/.../resourceGroups/alper/providers/Microsoft.Network/virtualNetworks/importDemo
spec:
  forProvider:
    name: "importDemo"
    location: "East US"
    resourceGroupName: "alper"
    addressSpace:
      - "10.0.0.0/16"
  providerConfigRef:
    name: example

, the imported MR is late-initialized as follows:

apiVersion: virtual.azure.tf.crossplane.io/v1alpha1
kind: VirtualNetwork
metadata:
  annotations:
    crossplane.io/external-name: /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/alper/providers/Microsoft.Network/virtualNetworks/importDemo
    creationTimestamp: "2021-09-10T03:18:29Z"
...
  finalizers:
  - finalizer.managedresource.crossplane.io
  name: imported
...
spec:
  deletionPolicy: Delete
  forProvider:
    addressSpace:
    - 10.0.0.0/16
    dnsServers:
    - 10.0.0.1
    - 10.0.0.2
    - 10.0.0.3
    location: East US
    name: importDemo
    resourceGroupName: alper
    tags:
      key1: value1
      purpose: demo
  providerConfigRef:
    name: example
status:
  atProvider:
    guid: 1222d5b8-41b7-4d84-bca0-a8282390422e
  conditions:
  - lastTransitionTime: "2021-09-10T03:18:39Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-09-10T03:18:39Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced

What problem are you facing?

Currently, when an MR is generated from Terrajet, the JSON tags are a consequence of the field type. If it is required, it doesn't have omitempty. If it's optional, it has the omitempty tag. Here is the code that decides it:

https://github.com/crossplane/terrajet/blob/8d0ed485f9511b65a8f3a83801092bcae60678dd/pkg/types/builder.go#L273-L279

This is fine for most use cases for managed resources. But there are some APIs and fields that would be benefited if this behavior can be configured.

The practical use case that I have for this is with the BranchProtection managed resource from GitHub terraform provider.

apiVersion: branchprotection.github.jet.crossplane.io/v1alpha1
kind: BranchProtection
metadata:
  name: main-branch
spec:
  deletionPolicy: Delete
  forProvider:
    repositoryId: my-repository
    pattern: main
    requiredStatusChecks:
      - strict: false
        contexts:
          - my-pipeline-check
  providerConfigRef:
    name: github-jet-provider-config

In the YAML above, the spec.forProvider.requiredStatusChecks[0].contexts field is optional (so the JSON tag has the omitempty value). It configures which status checks are mandatory to pass before a PR merge can be done.

By applying the YAML above, we expect that the status check my-pipeline-check MUST pass before any PR could be merged. This works as expected.

The problem is that, if I change the contexts field to contexts: [] -- for example:

apiVersion: branchprotection.github.jet.crossplane.io/v1alpha1
kind: BranchProtection
metadata:
  name: main-branch
spec:
  deletionPolicy: Delete
  forProvider:
    repositoryId: my-repository
    pattern: main
    requiredStatusChecks:
      - strict: false
        contexts: []
  providerConfigRef:
    name: github-jet-provider-config

I would expect that the external API would also have an empty list of status checks. But the Managed Resource code doesn't remove the my-pipeline-check status check from the API.

How could Terrajet help solve your problem?

My hypothesis is that the code is considering the contexts field as not_specified, so it isn't actively forcing the BranchProtection (in the external API) to not have the contexts field filled.

If the hypothesis is correct, we could configure this tags and the code could differ from not_specified, specified and empty_value and configures the external API accordingly.

What happened?

I'm trying to generate CRDS with make generate with custom provider

cannot generate crd for resource lb_instance: cannot build types for Instance: cannot build the Types: cannot infer type from schema of field listeners: invalid schema type TypeInvalid

there is my json schema:

              "listeners": {
                "nested_type": {
                  "attributes": {
                    "connection_limit": {
                      "type": "number",
                      "description_kind": "plain",
                      "required": true
                    },
                    "description": {
                      "type": "string",
                      "description_kind": "plain",
                      "computed": true
                    },
                    "headers": {
                      "nested_type": {
                        "attributes": {
                          "x_forwarded_for": {
                            "type": "bool",
                            "description_kind": "plain",
                            "optional": true
                          },
                          "x_forwarded_port": {
                            "type": "bool",
                            "description_kind": "plain",
                            "optional": true
                          },
                          "x_forwarded_proto": {
                            "type": "bool",
                            "description_kind": "plain",
                            "optional": true
                          }
                        },
                        "nesting_mode": "single"
                      },
                      "description_kind": "plain",
                      "optional": true
                    },
                    "id": {
                      "type": "string",
                      "description_kind": "plain",
                      "computed": true
                    },
                    "name": {
                      "type": "string",
                      "description": "displayed name",
                      "description_kind": "plain",
                      "required": true
                    },
                    "pool": {
                      "nested_type": {
                        "attributes": {
                          "algorithm": {
                            "type": "string",
                            "description": "\"LEAST_CONNECTIONS\", \"ROUND_ROBIN\", \"SOURCE_IP\"",
                            "description_kind": "plain",
                            "required": true
                          },
                          "description": {
                            "type": "string",
                            "description_kind": "plain",
                            "computed": true
                          },
                          "id": {
                            "type": "string",
                            "description_kind": "plain",
                            "computed": true
                          },
                          "members": {
                            "nested_type": {
                              "attributes": {
                                "address": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "backup": {
                                  "type": "bool",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "monitor_address": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "optional": true
                                },
                                "monitor_port": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "optional": true
                                },
                                "name": {
                                  "type": "string",
                                  "description": "name of the existed instance",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "network_name": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "port": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "weight": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "required": true
                                }
                              },
                              "nesting_mode": "list"
                            },
                            "description_kind": "plain",
                            "required": true
                          },
                          "monitor": {
                            "nested_type": {
                              "attributes": {
                                "delay": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "expected_codes": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "optional": true
                                },
                                "id": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "computed": true
                                },
                                "max_retries": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "max_retries_down": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "name": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "optional": true
                                },
                                "timeout": {
                                  "type": "number",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "type": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "required": true
                                },
                                "url_path": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "optional": true
                                }
                              },
                              "nesting_mode": "single"
                            },
                            "description_kind": "plain",
                            "optional": true
                          },
                          "name": {
                            "type": "string",
                            "description_kind": "plain",
                            "computed": true
                          },
                          "session_persistence": {
                            "nested_type": {
                              "attributes": {
                                "cookie_name": {
                                  "type": "string",
                                  "description_kind": "plain",
                                  "optional": true
                                },
                                "type": {
                                  "type": "string",
                                  "description": "\"NONE\", \"HTTP_COOKIE\", \"APP_COOKIE\", \"SOURCE_IP\"",
                                  "description_kind": "plain",
                                  "required": true
                                }
                              },
                              "nesting_mode": "single"
                            },
                            "description_kind": "plain",
                            "required": true
                          }
                        },
                        "nesting_mode": "single"
                      },
                      "description_kind": "plain",
                      "required": true
                    },
                    "port": {
                      "type": "number",
                      "description_kind": "plain",
                      "required": true
                    },
                    "protocol": {
                      "type": "string",
                      "description": "\"HTTP\", \"HTTPS\", \"TCP\", \"UDP\", \"TERMINATED_HTTPS\"",
                      "description_kind": "plain",
                      "required": true
                    },
                    "timeout_client_data": {
                      "type": "number",
                      "description_kind": "plain",
                      "optional": true
                    },
                    "timeout_member_connect": {
                      "type": "number",
                      "description_kind": "plain",
                      "optional": true
                    },
                    "timeout_member_data": {
                      "type": "number",
                      "description_kind": "plain",
                      "optional": true
                    },
                    "timeout_tcp_inspect": {
                      "type": "number",
                      "description_kind": "plain",
                      "optional": true
                    },
                    "tls_container_id": {
                      "type": "string",
                      "description_kind": "plain",
                      "optional": true
                    }
                  },
                  "nesting_mode": "list"
                },
                "description_kind": "plain",
                "required": true
              }

Can you please give me a tip what wrong with this schema and how i can fix this issue?

What happened?

Hi, This is more like JFYI. When client connects to provider with "Test":true in TF_REATTACH_PROVIDERS, it will cause a memory leak. More info here: https://github.com/hashicorp/terraform-provider-aws/issues/26130 https://github.com/hashicorp/go-plugin/issues/215

How can we reproduce it?

The easy way to reproduce is to run terraform-provider-aws --debug, and constantly run terraform plan against this provider in debug mode.

I'm trying to use https://registry.terraform.io/providers/pan-net/powerdns/latest as crossplane terrajet provider and when i'm trying to change A records by adding multiple values for A records: from

apiVersion: record.powerdns.jet.crossplane.io/v1alpha1
kind: Record
metadata:
  name: example
spec:
  forProvider:
   zone: "example.com."
   name: "www.example.com."
   type: "A"
   ttl: 300
   records: ["192.168.0.11"]
  providerConfigRef:
    name: powerdns

to

apiVersion: record.powerdns.jet.crossplane.io/v1alpha1
kind: Record
metadata:
  name: example
spec:
  forProvider:
   zone: "example.com."
   name: "www.example.com."
   type: "A"
   ttl: 300
   records: ["192.168.0.11", "192.168.0.12"]
  providerConfigRef:
    name: powerdns

error happens: "Resource powerdns_record.example has lifecycle.prevent_destroy set, but the plan calls for this resource to be destroyed."

How can i resolve this problem? is it possibly to set prevent_destroy=fasle somehow?

What happened?

I am having some "import cycle" issues when running make reviewable on my Terrajet project. Below is three resources that generates the import cycle not allowed error.

• Team Routing Rule resource references.

  • team_id => type Team
  • notify.id => type Escalation

• Escalation resource references,

  • owner_team_id => type Team
  • rules.recipient.id => type Team

This will generate the following error on running make reviewable.

WARN [runner] Can't run linter goanalysis_metalinter: bodyclose: failed prerequisites: [[email protected]/ok-amba/provider-jet-opsgenie/apis/team/v1alpha1: analysis skipped: errors in package: [/home/kerwood/Git/provider-jet-opsgenie/apis/team/v1alpha1/zz_generated.resolvers.go:23:11: could not import github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1 (-: import cycle not allowed: import stack: [github.com/ok-amba/provider-jet-opsgenie/apis github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1 github.com/ok-amba/provider-jet-opsgenie/apis/team/v1alpha1 github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1])]] 
WARN [runner] Can't run linter unused: buildir: analysis skipped: errors in package: [-: import cycle not allowed: import stack: [github.com/ok-amba/provider-jet-opsgenie/apis github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1 github.com/ok-amba/provider-jet-opsgenie/apis/team/v1alpha1 github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1]] 
ERRO Running error: buildir: analysis skipped: errors in package: [-: import cycle not allowed: import stack: [github.com/ok-amba/provider-jet-opsgenie/apis github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1 github.com/ok-amba/provider-jet-opsgenie/apis/team/v1alpha1 github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1]] 
11:35:52 [FAIL]
make[2]: *** [build/makelib/golang.mk:194: go.lint] Error 1
make[1]: *** [build/makelib/common.mk:357: lint] Error 2
make: *** [build/makelib/common.mk:425: reviewable] Error 2

My guess is that its importing github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1 multiple times for some reason.

[
 github.com/ok-amba/provider-jet-opsgenie/apis 
 github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1 
 github.com/ok-amba/provider-jet-opsgenie/apis/team/v1alpha1 
 github.com/ok-amba/provider-jet-opsgenie/apis/escalation/v1alpha1
]

Links to files

Team Routing Rule config.go file: https://github.com/ok-amba/provider-jet-opsgenie/blob/1d82bf7f2d5477d61db916c4868333187a0b10b1/config/teamroutingrule/config.go Escalation config.go file: https://github.com/ok-amba/provider-jet-opsgenie/blob/1d82bf7f2d5477d61db916c4868333187a0b10b1/config/escalation/config.go

How can we reproduce it?

git clone [email protected]:ok-amba/provider-jet-opsgenie.git
cd provider-jet-opsgenie
git checkout 1d82bf7f2d5477d61db916c4868333187a0b10b1
make submodules
make reviewable

Hello team!

Me and my team are developing a solution for developers to be able to manage infrastructure resources through an API. This solution relies on crossplane to manage cloud resources as kubernetes CRs.

We've started using terrajet for AWS, but changed to the native provider because of this issue. New we also need to manage other resources for which there are no native providers (namely vault, boundary and datadog) and it would be great to use terrajet.

Existing Context

initial slack thread with problem description

What happened?

The expected behaviour is that terrajet resources time-to-readiness wouldn't depend on the amount of resources that already exist in the cluster.

In reality, the terrajet controller takes roughly 3 * (number of existing resources) * (time it takes for one resource to be reconciled) seconds to consider a new resource ready.

The examples in this issue are using provider-jet-aws, but all terrajet providers have the same behaviour, from what I can see.

Below are the debug logs related to one recently created policy (terrajetloadtestslow). It was created on a provider that is already managing 200 other aws policies.

crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630252994777613e+09       DEBUG   provider-jet-aws        Reconciling     {"controller": "managed/iam.aws.jet.crossplane.io/v1alpha2, kind=policy", "request": "/terrajetloadtestslow"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630253028353422e+09       DEBUG   provider-jet-aws        refresh ended   {"workspace": "/tmp/636bbd3e-94bb-40ae-acde-f866e095b0e6", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.0.5\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:20.558331Z\",\"terraform\":\"1.0.5\",\"type\":\"version\",\"ui\":\"0.1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Refreshing state...\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.822179Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"id_key\":\"id\"},\"type\":\"refresh_start\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Refresh complete\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.822992Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null}},\"type\":\"refresh_complete\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Drift detected (delete)\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.823374Z\",\"change\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"action\":\"delete\"},\"type\":\"resource_drift\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.823487Z\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 0 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.829034Z\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.829081Z\",\"outputs\":{},\"type\":\"outputs\"}\n"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630253059385006e+09       DEBUG   provider-jet-aws        apply ended     {"workspace": "/tmp/636bbd3e-94bb-40ae-acde-f866e095b0e6", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.0.5\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:22.897931Z\",\"terraform\":\"1.0.5\",\"type\":\"version\",\"ui\":\"0.1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Plan to create\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:25.127798Z\",\"change\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"action\":\"create\"},\"type\":\"planned_change\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 1 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:25.127975Z\",\"changes\":{\"add\":1,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Creating...\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:25.630013Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"action\":\"create\"},\"type\":\"apply_start\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Creation complete after 0s [id=arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:25.924980Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"action\":\"create\",\"id_key\":\"id\",\"id_value\":\"arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow\",\"elapsed_seconds\":0},\"type\":\"apply_complete\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 1 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:25.932091Z\",\"changes\":{\"add\":1,\"change\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:28:25.932195Z\",\"outputs\":{},\"type\":\"outputs\"}\n"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.663025305947192e+09        DEBUG   provider-jet-aws        Successfully requested creation of external resource {"controller": "managed/iam.aws.jet.crossplane.io/v1alpha2, kind=policy", "request": "/terrajetloadtestslow", "uid": "636bbd3e-94bb-40ae-acde-f866e095b0e6", "version": "573953565", "external-name": "", "external-name": "arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630253059475644e+09       DEBUG   events  Normal  {"object": {"kind":"Policy","name":"terrajetloadtestslow","uid":"636bbd3e-94bb-40ae-acde-f866e095b0e6","apiVersion":"iam.aws.jet.crossplane.io/v1alpha2","resourceVersion":"573965831"}, "reason": "CreatedExternalResource", "message": "Successfully requested creation of external resource"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.663025826864625e+09        DEBUG   provider-jet-aws        Reconciling     {"controller": "managed/iam.aws.jet.crossplane.io/v1alpha2, kind=policy", "request": "/terrajetloadtestslow"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630258282078466e+09       DEBUG   provider-jet-aws        refresh ended   {"workspace": "/tmp/636bbd3e-94bb-40ae-acde-f866e095b0e6", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.0.5\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:37:06.895864Z\",\"terraform\":\"1.0.5\",\"type\":\"version\",\"ui\":\"0.1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Refreshing state... [id=arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:37:08.112383Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow\"},\"type\":\"refresh_start\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Refresh complete [id=arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:37:08.197666Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow\"},\"type\":\"refresh_complete\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:37:08.198356Z\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 0 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:37:08.203956Z\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:37:08.203989Z\",\"outputs\":{},\"type\":\"outputs\"}\n"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630258282080562e+09       DEBUG   provider-jet-aws        External resource is up to date {"controller": "managed/iam.aws.jet.crossplane.io/v1alpha2, kind=policy", "request": "/terrajetloadtestslow", "uid": "636bbd3e-94bb-40ae-acde-f866e095b0e6", "version": "573965832", "external-name": "arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow", "requeue-after": 1663025888.2080543}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630263118990378e+09       DEBUG   provider-jet-aws        Reconciling     {"controller": "managed/iam.aws.jet.crossplane.io/v1alpha2, kind=policy", "request": "/terrajetloadtestslow"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630263132372632e+09       DEBUG   provider-jet-aws        refresh ended   {"workspace": "/tmp/636bbd3e-94bb-40ae-acde-f866e095b0e6", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.0.5\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:45:11.952075Z\",\"terraform\":\"1.0.5\",\"type\":\"version\",\"ui\":\"0.1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Refreshing state... [id=arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:45:13.135934Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow\"},\"type\":\"refresh_start\"}\n{\"@level\":\"info\",\"@message\":\"aws_iam_policy.terrajetloadtestslow: Refresh complete [id=arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:45:13.226661Z\",\"hook\":{\"resource\":{\"addr\":\"aws_iam_policy.terrajetloadtestslow\",\"module\":\"\",\"resource\":\"aws_iam_policy.terrajetloadtestslow\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_iam_policy\",\"resource_name\":\"terrajetloadtestslow\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow\"},\"type\":\"refresh_complete\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:45:13.227599Z\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 0 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:45:13.232695Z\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2022-09-12T23:45:13.232734Z\",\"outputs\":{},\"type\":\"outputs\"}\n"}
crossplane-provider-jet-aws-599c4ea494ca-55b9dd885f-zr2pb provider-jet-aws 1.6630263132466455e+09       DEBUG   provider-jet-aws        External resource is up to date {"controller": "managed/iam.aws.jet.crossplane.io/v1alpha2, kind=policy", "request": "/terrajetloadtestslow", "uid": "636bbd3e-94bb-40ae-acde-f866e095b0e6", "version": "573976982", "external-name": "arn:aws:iam::ACCOUNT_NUMBER:policy/terrajetloadtestslow", "requeue-after": 1663026373.2466433}

From these logs we can also see the time it took for the resource to be Ready. From 23:28:19 to 23:45:13, ~17 minutes. (using GMT time to make it easier)

It should also be noted that the resource is created in the cloud provider at 23:28:25 (12 seconds), when the logs say Successfully requested creation of external resource, but it takes a long time until the resource is marked as Ready in kubernetes

The reason why it takes so long seems to be that resource change events are not given any priority over the normal reconciliation loop. The controller stays in an endless loop reconciling every resource it knows about, which is necessary to override external changes in the providers, but any resource that is added or changed in kubernetes doesn't get immediately picked up. Rather it seems to be added to the queue and only seen after all existing reconcile loops are done, then the resource is picked up, and a creation flow starts, but the result will only be seen on the next reconciliation loop. Once that comes, terrajet will pick up the resource's status from the terraform state and requeue once more, only after this last requeue the status will be updated to Ready.

It's also worth noting that changing the MaxConcurrentReconciles configuration to something bigger, like 5 or 10 reduces the time by that factor, but the underlying behavior persists and all resources are reconciled before considering new resource or updates Ready.

What I'm having trouble understanding is how this terrajet workflow differs from the native provider-aws one, as both seem to use the same Reconcile function from crossplane-runtime.

Also, the UseAsync configuration for a resource doesn't seem to change anything. I've tried both for policies.iam and for repositories.ecr with the same behavior.

How can we reproduce it?

• Create a new kubernetes cluster (with kind or in the cloud).
• Install crossplane
• Install a terrajet provider (I'll use AWS because it can be easily compared with the native provider-aws)
• Create a handful of resources to be managed by terrajet (I'll use ECR because it is quickly created and dont incur costs)
• Wait until all resources are created and ready it will take some minutes, but a burst of resources is expected to take a bit. Althought it does take much longer than provider-aws for the same resource.
• Create one more resource to be managed by terrajet

The last step will take a long time, which is the problem this bug report is about.

# Create kind cluster
kind create cluster --name terrajet-load --image kindest/node:v1.23.10

# Install crossplane with helm
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm upgrade --install crossplane --namespace crossplane-system crossplane-stable/crossplane

# Install AWS terrajet provider
kubectl apply -f - <<YAML
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: crossplane-provider-jet-aws
spec:
  controllerConfigRef:
    name: config
  package: crossplane/provider-jet-aws:v0.5.0
---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: config
spec:
  args:
  - --debug
YAML
## use your aws credentials in the secret, if you have a custom way to interact with AWS, change the credentials key of this secret.
kubectl create secret generic -n crossplane-system --from-file=credentials=$HOME/.aws/credentials aws-credentials
kubectl apply -f - <<YAML
apiVersion: aws.jet.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  finalizers:
  - in-use.crossplane.io
  name: default
spec:
  credentials:
    secretRef:
      key: credentials
      name: aws-credentials
      namespace: crossplane-system
    source: Secret
YAML

# Create a handful of resources managed by terrajet. I chose policies because they are created near-instantly in the cloud and don't incur costs
## Note: seq on MacOS doesn't seem to support the -w flag, it can be removed safely below
for n in $(seq -w 200); do
  echo "---"
  sed "s/NUMBER/$n/" <<YAML
apiVersion: iam.aws.jet.crossplane.io/v1alpha2
kind: Policy
metadata:
  name: terrajetloadtestNUMBER
spec:
  forProvider:
    name: terrajetloadtestNUMBER
    policy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Deny",
            "Action": "*",
            "Resource": ["*"]
          }
        ]
      }
  providerConfigRef:
    name: default
YAML
done | kubectl apply -f -

# Wait until all are ready, it took about 15 minutes for me
kubectl get policies.iam.aws.jet.crossplane.io

# Create one more resource
kubectl apply -f - <<YAML
apiVersion: iam.aws.jet.crossplane.io/v1alpha2
kind: Policy
metadata:
  name: terrajetloadtestslow
spec:
  forProvider:
    name: terrajetloadtestslow
    policy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Deny",
            "Action": "*",
            "Resource": ["*"]
          }
        ]
      }
  providerConfigRef:
    name: default
YAML

# It takes a long time for the resource to become ready.
kubectl get policies.iam.aws.jet.crossplane.io terrajetloadtestslow

########

# Cleanup cloud resources
kubectl delete policy.iam.aws.jet $(kubectl get policy.iam.aws.jet | grep terrajetloadtest | awk '{print $1}')

# Delete kind cluster
kind destroy cluster --name terrajet-load

Possibly related discussions

• slack thread
• #55
• crossplane/crossplane#2477

Please let me know if I can help figure out what's happening.