Following readme to deploy cortex operatoer in GKE and I get the following eror

error: unable to recognize "cortex_operator_test.yaml": no matches for kind "Cortex" in version "cortex.opstrace.io/v1alpha1"

What could cause this, my yaml looks like this

apiVersion: cortex.opstrace.io/v1alpha1
kind: Cortex
metadata:
  name: cortex-jd-operator
spec:
  image: "cortexproject/cortex:v1.9.0"
  config:
    server:
      grpc_server_max_recv_msg_size: 41943040
      grpc_server_max_send_msg_size: 41943040
    memberlist:
      max_join_backoff: 1m
      max_join_retries: 20
      min_join_backoff: 1s
    query_range:
      split_queries_by_interval: 24h
    limits:
      compactor_blocks_retention_period: 192h
      ingestion_rate: 100000
      ingestion_rate_strategy: local
      ingestion_burst_size: 200000
      max_global_series_per_user: 10000000
      max_series_per_user: 5000000
      accept_ha_samples: true
      ha_cluster_label: prometheus
      ha_replica_label: prometheus_replica
      ruler_tenant_shard_size: 3
    ingester:
      lifecycler:
        join_after: 30s
        observe_period: 30s
        num_tokens: 512
    blocks_storage:
      tsdb:
        retention_period: 6h
      backend: gcs
      gcs:
        # https://cortexmetrics.io/docs/configuration/configuration-file/#blocks_storage_config
        # TODO: Pass in bucket name
        bucket_name: cortex-data
    configs:
      database:
        # TODO: Do we need this database or can we skip this? 
        # https://cortexmetrics.io/docs/configuration/configuration-file/#configs_config
        # URI where the database can be found (for dev you can use memory://)
        # CLI flag: -configs.database.uri
        uri: https://someuri.com
        # Path where the database migration files can be found
        # CLI flag: -configs.database.migrations-dir
        migrations_dir: /migrations
    alertmanager_storage:
      backend: gcs
      # TODO: Pass in bucket name 
      # https://cortexmetrics.io/docs/configuration/configuration-file/#alertmanager_config
      gcs:
        bucket_name: cortex-config
    ruler_storage:
      backend: gcs
      # TODO: Pass in bucket name 
      # https://cortexmetrics.io/docs/configuration/configuration-file/#ruler_config
      gcs:
        bucket_name: cortex-config

Cortex supports tracing using envvars to enable it on a per-component/pod basis. I don't know if there's anything else that use envvars. Another option could be to expose a tracing-specific section, but there's a lot of tracing client options for things like tags and auth, making it feel more straightforward to just expose an env option directly.

Also updates the sample config to exercise more fields.

NOTE: Haven't really tried this out yet, outside of checking that existing tests were happy with the new sample config.

Signed-off-by: Nick Parker [email protected]

The controller creates a config map for the runtime config in the same namespace as the cortex crd resource. The operator can edit it to apply the configuration overrides.

The reload period is set to 5s.

The config map is mounted as a file at /etc/cortex/runtime-config.yaml.

To set the imagePullSecrets with the cortex-operator you can set the .spec.service_account_spec.image_pull_secrets field.

Example:

apiVersion: cortex.opstrace.io/v1alpha1
kind: Cortex
metadata:
  name: cortex-sample
spec:
  service_account_spec:
    image_pull_secrets:
    - name: secret-name

• Removes defaulting via kubebuilder annotations since it was having issues with pointer fields.
• Make runtime_config field a runtime.RawExtention. More flexibility to set more config options via this field.
• Fix image name.

For pods to authenticate with Google Cloud they should have the iam.gke.io/gcp-service-account annotation set.

With this PR it's now possible to set the service account annotations. This makes it possible to deploy the operator on GCP using a workload identity.

Example:

apiVersion: cortex.opstrace.io/v1alpha1
kind: Cortex
metadata:
  name: cortex-sample
spec:
  service_account_spec:
    annotations:
      foo: "bar"

A few unrelated changes to add features to integrate this operator in Opstrace:

• Rename cortex configmap
• Allow configuring the statefulsets storage class
• Enable auth

The defaulting step was concatenating the list of members in the memberlist gossip ring each time the CRD resource was updated. This was a byproduct of the defaulting step mutating the config in the resource.

The create and update validation webhooks now merge the user config with the opinionated defaults.

Small example to configure the ingester and the results Memcached cache:

apiVersion: cortex.opstrace.io/v1alpha1
kind: Cortex
metadata:
  name: cortex-sample
spec:
  image: "cortexproject/cortex:v1.9.0"

  ingester_spec:
    replicas: 1
    datadir_size: 2Gi # default is 1Gi

  memcached:
    image: memcached:memcached-1.6.9-alpine
    results_cache_spec:
      replicas: 1
      memory_limit: 1024 # default is 4096
      max_item_size: "1m" # default is 2m

Adds a new field to the Cortex CRD to specify the desired Cortex configuration and removes the previous storage configuration options.

This is how the struct is defined to generate the CRD schema.

// CortexSpec defines the desired state of Cortex
type CortexSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Image of Cortex to deploy.
	Image string `json:"image,omitempty"`

	// Config accepts any object, meaning it accepts any valid Cortex config
	// yaml. Defaulting and Validation are done in the webhooks.
	// +kubebuilder:pruning:PreserveUnknownFields
	Config runtime.RawExtension `json:"config,omitempty"`
}

Using runtime.RawExtension allows us to accept any free form configuration in that field. This means we can now embed the Cortex configuration directly in the spec.config field. Here's an example:

apiVersion: cortex.opstrace.io/v1alpha1
kind: Cortex
metadata:
  name: cortex-sample
spec:
  image: "cortexproject/cortex:v1.9.0"
  config:
    server:
      grpc_server_max_recv_msg_size: 41943040
      grpc_server_max_send_msg_size: 41943040
    memberlist:
      max_join_backoff: 1m
      max_join_retries: 20
      min_join_backoff: 1s
    query_range:
      split_queries_by_interval: 24h
    limits:
      compactor_blocks_retention_period: 192h
      ingestion_rate: 100000
      ingestion_rate_strategy: local
      ingestion_burst_size: 200000
      max_global_series_per_user: 10000000
      max_series_per_user: 5000000
      accept_ha_samples: true
      ha_cluster_label: prometheus
      ha_replica_label: prometheus_replica
      ruler_tenant_shard_size: 3
    ingester:
      lifecycler:
        join_after: 30s
        observe_period: 30s
        num_tokens: 512
    blocks_storage:
      tsdb:
        retention_period: 6h
      backend: s3
      s3:
        bucket_name: cortex-operator-example-209f-data
        endpoint: s3.us-west-2.amazonaws.com
    configs:
      database:
        uri: https://someuri.com
        migrations_dir: /migrations
    alertmanager_storage:
      backend: s3
      s3:
        bucket_name: cortex-operator-example-209f-config
        endpoint: s3.us-west-2.amazonaws.com
    ruler_storage:
      backend: s3
      s3:
        bucket_name: cortex-operator-example-209f-config
        endpoint: s3.us-west-2.amazonaws.com

This PR also adds webhook validation and defaulting. Defaulting webhook, amongst other things, ensures the Cortex configuration has the correct addresses for the services. The Defaulting webhook merges the user-specified configuration with the operator defaults. The validation webhook checks the resulting configuration is valid by calling out the Validate method of the upstream Cortex configuration struct.