Manage large fleets of Kubernetes clusters

Related tags

DevOps Tools fleet
Overview

Introduction

Fleet is GitOps at scale. Fleet is designed to manage up to a million clusters. It's also lightweight enough that it works great for a single cluster too, but it really shines when you get to a large scale. By large scale we mean either a lot of clusters, a lot of deployments, or a lot of teams in a single organization.

Fleet can manage deployments from git of raw Kubernetes YAML, Helm charts, or Kustomize or any combination of the three. Regardless of the source all resources are dynamically turned into Helm charts and Helm is used as the engine to deploy everything in the cluster. This gives a high degree of control, consistency, and auditability. Fleet focuses not only on the ability to scale, but to give one a high degree of control and visibility to exactly what is installed on the cluster.

Quick Start

Who needs documentation, let's just run this thing!

Install

Get helm if you don't have it. Helm 3 is just a CLI and won't do bad insecure things to your cluster.

brew install helm

Install the Fleet Helm charts (there's two because we separate out CRDs for ultimate flexibility.)

VERSION=0.3.3
helm -n fleet-system install --create-namespace --wait \
    fleet-crd https://github.com/rancher/fleet/releases/download/v${VERSION}/fleet-crd-${VERSION}.tgz
helm -n fleet-system install --create-namespace --wait \
    fleet https://github.com/rancher/fleet/releases/download/v${VERSION}/fleet-${VERSION}.tgz

Add a Git Repo to watch

Change spec.repo to your git repo of choice. Kubernetes manifest files that should be deployed should be in /manifests in your repo.

cat > example.yaml << "EOF"
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: sample
  # This namespace is special and auto-wired to deploy to the local cluster
  namespace: fleet-local
spec:
  # Everything from this repo will be ran in this cluster. You trust me right?
  repo: "https://github.com/fleet-demo/simple"
EOF

kubectl apply -f example.yaml

Get Status

Get status of what fleet is doing

kubectl -n fleet-local get fleet

You should see something like this get created in your cluster.

kubectl get deploy frontend
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
frontend   3/3     3            3           116m

Enjoy and read the docs.

Comments
  • fleet-agent cannot find secret in local cluster in Rancher single-install setup

    fleet-agent cannot find secret in local cluster in Rancher single-install setup

    run rancher:master-0f691dc70f86bbda3d6563af11779300a6191584-head in the single-install mode

    Screen Shot 2020-09-14 at 5 17 43 PM Screen Shot 2020-09-14 at 5 17 51 PM Screen Shot 2020-09-14 at 5 18 30 PM

    The following line floods the log of the pod fleet-agent-7dfdfd5846-xjw96

    time="2020-09-15T00:18:30Z" level=info msg="Waiting for secret fleet-clusters-system/c-09ea1d541bf704218ec6fc9ab2d60c0392543af636c1c3a90793946522685 for request-2vz49: secrets \"c-09ea1d541bf704218ec6fc9ab2d60c0392543af636c1c3a90793946522685\" not found"
    

    gz#14319

    internal kind/bug 
    opened by jiaqiluo 30
  • gitjob:v0.1.11 in 0.3.2

    gitjob:v0.1.11 in 0.3.2

    The gitjob image v0.1.11 in 0.3.2 does not have a uid entry for 1000 in /etc/passwd This results in an error when the gitjob attempts to read the git repository.

    ~The workaround is to patch the gitjob deployment with a securityContext.runAsUser. It appears the nobody user works...~

      securityContext:
        runAsUser: 65534
    
    opened by boxboatmatt 29
  • fleet-agent cleanup continually deleting releases

    fleet-agent cleanup continually deleting releases

    The cleanup loop in fleet-agent isn't able to properly identify releases with bundles. For a given release, it looks for the bundle deployment associated to that release. If the release has a different name than fleet-agent expects (based on the bundle deployment) then fleet-agent deletes the release.

    The problem is that then a release with the exact same name is created and the next time the cleanup loop runs, fleet-agent will delete the release again.

    For example, a bundle deployment with name mcc-anupamafinalrcrke2-managed-system-upgrade creates a release with name mcc-anupamafinalrcrke2-managed-system-upgrade-c-407d2.

    It seems that this function should look at status.release as well when trying to determine the name of the release from a bundle deployment.

    [zube]: Done kind/bug 
    opened by thedadams 17
  • Add debug logging to fleet controller and agents

    Add debug logging to fleet controller and agents

    We need more debug logging in the code and the ability to enable it via charts. As is, we no logs or clear reason for failures such as chart changes that do not stick. Detailed debug logging is vital to the supportability of Fleet.

    internal status/dev-validate kind/docs team/area3 status/to-test 
    opened by Tejeev 15
  • Deploying the multi-cluster kustomize example results in bundle render error

    Deploying the multi-cluster kustomize example results in bundle render error

    Steps to reproduce:

    1. Label a cluster with "env=dev"
    2. Create a GitRepo via the UI pointing to the Multi-Cluster Kustomize Example folder.
    kind: GitRepo
    apiVersion: fleet.cattle.io/v1alpha1
    metadata:
      name: kustomize
      namespace: fleet-default
    spec:
      repo: https://github.com/rancher/fleet-examples
      paths:
      - multi-cluster/kustomize
      targets:
      - name: dev
        clusterSelector:
          matchLabels:
            env: dev
    

    Expected:

    App is deployed to the cluster labeled "env=dev"

    Actual: App is not deployed. Bundle is stuck in error.

    error while running post render on files: Object 'Kind' is missing in '{"resources":["frontend-deployment.yaml","frontend-service.yaml","redis-master-deployment.yaml","redis-master-service.yaml","redis-slave-deployment.yaml","redis-slave-service.yaml"]}'

    Environment: Rancher v2.5.5

    Export of failing bundle:

    apiVersion: fleet.cattle.io/v1alpha1
    kind: Bundle
    metadata:
      creationTimestamp: "2021-01-22T13:56:00Z"
      generation: 1
      labels:
        fleet.cattle.io/commit: c6e54d7a56565e52a63de8a2088997b46253c1fb
        fleet.cattle.io/repo-name: test123
      managedFields:
      - apiVersion: fleet.cattle.io/v1alpha1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:labels:
              .: {}
              f:fleet.cattle.io/commit: {}
              f:fleet.cattle.io/repo-name: {}
          f:spec:
            .: {}
            f:namespace: {}
            f:resources: {}
            f:targetRestrictions: {}
            f:targets: {}
        manager: fleet
        operation: Update
        time: "2021-01-22T13:56:00Z"
      - apiVersion: fleet.cattle.io/v1alpha1
        fieldsType: FieldsV1
        fieldsV1:
          f:status:
            .: {}
            f:conditions: {}
            f:display:
              .: {}
              f:readyClusters: {}
            f:maxUnavailable: {}
            f:maxUnavailablePartitions: {}
            f:observedGeneration: {}
            f:summary:
              .: {}
              f:desiredReady: {}
              f:ready: {}
            f:unavailable: {}
            f:unavailablePartitions: {}
        manager: fleetcontroller
        operation: Update
        time: "2021-01-22T13:56:01Z"
      name: test123-multi-cluster-kustomize
      namespace: fleet-default
      resourceVersion: "128834872"
      selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/fleet-default/bundles/test123-multi-cluster-kustomize
      uid: 777465f5-f2f0-4e02-a6d9-d917bc90a8f7
    spec:
      namespace: fleet-mc-kustomize-example
      resources:
      - content: |
          # Multi-Cluster Kustomize Example
    
          This example will deploy the [Kubernetes sample guestbook](https://github.com/kubernetes/examples/tree/master/guestbook/) application
          using kustomize. The app will be deployed into the `fleet-mc-kustomize-example` namespace.
    
          The application will be customized as follows per environment:
    
          * Dev clusters: Only the redis leader is deployed and not the followers.
          * Test clusters: Scale the front deployment to 3
          * Prod clusters: Scale the front deployment to 3 and set the service type to LoadBalancer
    
          ```yaml
          kind: GitRepo
          apiVersion: fleet.cattle.io/v1alpha1
          metadata:
            name: kustomize
            namespace: fleet-local
          spec:
            repo: https://github.com/rancher/fleet-examples
            paths:
            - multi-cluster/kustomize
            targets:
            - name: dev
              clusterSelector:
                matchLabels:
                  env: dev
    
            - name: test
              clusterSelector:
                matchLabels:
                  env: test
    
            - name: prod
              clusterSelector:
                matchLabels:
                  env: prod
          ```
        name: README.md
      - content: |
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: frontend
          spec:
            selector:
              matchLabels:
                app: guestbook
                tier: frontend
            replicas: 1
            template:
              metadata:
                labels:
                  app: guestbook
                  tier: frontend
              spec:
                containers:
                - name: php-redis
                  image: gcr.io/google-samples/gb-frontend:v4
                  resources:
                    requests:
                      cpu: 100m
                      memory: 100Mi
                  ports:
                  - containerPort: 80
        name: base/frontend-deployment.yaml
      - content: |
          apiVersion: v1
          kind: Service
          metadata:
            name: frontend
            labels:
              app: guestbook
              tier: frontend
          spec:
            type: NodePort
            ports:
            - port: 80
            selector:
              app: guestbook
              tier: frontend
        name: base/frontend-service.yaml
      - content: |
          resources:
          - frontend-deployment.yaml
          - frontend-service.yaml
          - redis-master-deployment.yaml
          - redis-master-service.yaml
          - redis-slave-deployment.yaml
          - redis-slave-service.yaml
        name: base/kustomization.yaml
      - content: |
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: redis-master
          spec:
            selector:
              matchLabels:
                app: redis
                role: master
                tier: backend
            replicas: 1
            template:
              metadata:
                labels:
                  app: redis
                  role: master
                  tier: backend
              spec:
                containers:
                - name: master
                  image: redis
                  resources:
                    requests:
                      cpu: 100m
                      memory: 100Mi
                  ports:
                  - containerPort: 6379
        name: base/redis-master-deployment.yaml
      - content: |
          apiVersion: v1
          kind: Service
          metadata:
            name: redis-master
            labels:
              app: redis
              role: master
              tier: backend
          spec:
            ports:
            - port: 6379
              targetPort: 6379
            selector:
              app: redis
              role: master
              tier: backend
        name: base/redis-master-service.yaml
      - content: |
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: redis-slave
          spec:
            selector:
              matchLabels:
                app: redis
                role: slave
                tier: backend
            replicas: 2
            template:
              metadata:
                labels:
                  app: redis
                  role: slave
                  tier: backend
              spec:
                containers:
                - name: slave
                  image: gcr.io/google_samples/gb-redisslave:v1
                  resources:
                    requests:
                      cpu: 100m
                      memory: 100Mi
                  ports:
                  - containerPort: 6379
        name: base/redis-slave-deployment.yaml
      - content: |
          apiVersion: v1
          kind: Service
          metadata:
            name: redis-slave
            labels:
              app: redis
              role: slave
              tier: backend
          spec:
            ports:
            - port: 6379
            selector:
              app: redis
              role: slave
              tier: backend
        name: base/redis-slave-service.yaml
      - content: |
          namespace: fleet-mc-kustomize-example
          targetCustomizations:
          - name: dev
            clusterSelector:
              matchLabels:
                env: dev
            kustomize:
              dir: overlays/dev
    
          - name: test
            clusterSelector:
              matchLabels:
                env: test
            kustomize:
              dir: overlays/test
    
          - name: prod
            clusterSelector:
              matchLabels:
                env: prod
            kustomize:
              dir: overlays/prod
        name: fleet.yaml
      - content: |
          resources:
          - ../../base
          patches:
          - redis-slave-deployment.yaml
          - redis-slave-service.yaml
        name: overlays/dev/kustomization.yaml
      - content: |
          kind: Deployment
          apiVersion: apps/v1
          metadata:
            name: redis-slave
          spec:
            replicas: 0
        name: overlays/dev/redis-slave-deployment.yaml
      - content: |
          kind: Service
          apiVersion: v1
          metadata:
            name: redis-slave
          spec:
            selector:
              role: master
        name: overlays/dev/redis-slave-service.yaml
      - content: |
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: frontend
          spec:
            replicas: 3
        name: overlays/prod/frontend-deployment.yaml
      - content: |
          kind: Service
          apiVersion: v1
          metadata:
            name: frontend
          spec:
            type: LoadBalancer
        name: overlays/prod/frontend-service.yaml
      - content: |
          resources:
          - ../../base
          patches:
          - frontend-deployment.yaml
          - frontend-service.yaml
        name: overlays/prod/kustomization.yaml
      - content: |
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: frontend
          spec:
            replicas: 3
        name: overlays/test/frontend-deployment.yaml
      - content: |
          resources:
          - ../../base
          patches:
          - frontend-deployment.yaml
        name: overlays/test/kustomization.yaml
      targetRestrictions:
      - clusterSelector: {}
      targets:
      - clusterSelector:
          matchLabels:
            env: dev
        kustomize:
          dir: overlays/dev
        name: dev
      - clusterSelector:
          matchLabels:
            env: test
        kustomize:
          dir: overlays/test
        name: test
      - clusterSelector:
          matchLabels:
            env: prod
        kustomize:
          dir: overlays/prod
        name: prod
      - clusterSelector: {}
    status:
      conditions:
      - lastUpdateTime: "2021-01-22T13:56:01Z"
        message: 'error while running post render on files: Object ''Kind'' is missing
          in ''{"resources":["frontend-deployment.yaml","frontend-service.yaml","redis-master-deployment.yaml","redis-master-service.yaml","redis-slave-deployment.yaml","redis-slave-service.yaml"]}'''
        reason: Error
        status: "False"
        type: Processed
      display:
        readyClusters: 0/0
      maxUnavailable: 0
      maxUnavailablePartitions: 0
      observedGeneration: 0
      summary:
        desiredReady: 0
        ready: 0
      unavailable: 0
      unavailablePartitions: 0
    

    gz#14552 gz#14575

    internal 
    opened by janeczku 15
  • Secrets with vals

    Secrets with vals

    Parses helm charts with https://github.com/variantdev/vals to inject secrets from any of the supported backends.

    vals is a tool I use all the time to centralize secrets in a single location. I don't know if people will support this change but I personally would like it added.

    I've only done the helm deployer as I don't use any other deployment method but it is simple enough it can be quickly add it to all.

    Sample usage.

    helm:
      valuesFiles:
        - secrets.yaml
    

    and secrets.yaml would contain something like

    password: ref+vault://secret/rancher/password
    

    Make sure you have the right env vars for the backend configured

    opened by digiserg 14
  • Add content controller to purge orphaned contents

    Add content controller to purge orphaned contents

    Signed-off-by: Matthew DeVenny [email protected]

    Currently fleet does not cleanup the Content objects that are created for a Bundle. Over time this will result in a very large number of Content objects being stored on the fleet-controller cluster. Added a ContentController to purge orphans when bundleDeployments are updated.

    opened by boxboatmatt 13
  • Add helm.valueFiles string array to fleet.yaml

    Add helm.valueFiles string array to fleet.yaml

    Add helm.valueFiles string array to fleet.yaml for the following reasons:

    1. In-lining Helm values within fleet.yaml makes them inaccessible for traditional testing using the helm client binary (Note that using the fleet CLI for this purpose is currently undocumented, is missing a --dry-run option, and has some issues running stand-alone such as https://github.com/rancher/fleet/issues/186)
    2. In-lining large values within fleet.yaml makes this resource difficult to comprehend
    3. Eases migration from Argo-CD application.yaml spec.source.helm.valueFiles (see https://github.com/argoproj/argo-cd/blob/45b3e48dfc607194e0498a0d654e0df2a624060d/docs/operator-manual/application.yaml#L35)
    opened by ron1 13
  • Fix Fleet post renderer errors with non-trivial kustomize git repo

    Fix Fleet post renderer errors with non-trivial kustomize git repo

    Fleet reports post renderer errors with various non-trivial kustomize git repos such as this which includes modifications to the multi-cluster kustomize example to mimic various real-world kustomize repos.

    Steps to reproduce the problem:

    1. git clone -b expose-fleet-mc-kustomize-example-post-renderer-bug https://github.com/ron1/fleet-examples.git
    2. cd fleet-examples/multi-cluster/kustomize
    3. kustomize build overlays/dev - verifies that kustomize successfully builds the dev overlay in this repo
    4. fleet test -t dev - reports the following unexpected error:
    # Matched: dev
    FATA[0000] error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
        "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
          "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
        "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
              {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
                  "-util-image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
                "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
                "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
                : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                    "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                    "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                    "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node
    

    Note that when I define a GitRepo for this branch with path multi-cluster/kustomize, its status.resourceErrors is updated as listed below.

    status:
      resourcesErrors:
      - |-
        error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
        "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
          "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
        "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
              {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
                  "-util-image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
                "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
                "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
                : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                    "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                    "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                    "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node
      - |-
        error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
        "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
          "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
        "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
              {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
                  "-util-image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
                "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
                "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
                : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                    "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                    "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                    "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node
      - |-
        error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
        "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
          "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
        "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
              {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
                  "-util-image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
                "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/[email protected]:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
                "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
                : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                    "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                    "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                    "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node
    
    status/has-dependency 
    opened by ron1 12
  • Request: Add overlay option from cluster secrets to handle secret strings in the values.yaml

    Request: Add overlay option from cluster secrets to handle secret strings in the values.yaml

    We need to be able to set secret strings on the values.yaml with a staged secret similar to Flux's helm release using valuesFrom.

    For example, Harbor has you set the database password in the values.yaml. We want to use Sealed Secrets to apply a secret, secret-values.yaml, via a separate bundle to the harbor namespace downstream with a key, values.yaml, of:

    database:
      external:
        password: areallybadpass
    

    Then call that overlay at the end of the fleet.yaml with something like this:

    defaultNamespace: harbor
    helm:
      takeOwnership: true
      releaseName: dev
      chart: "harbor"
      repo: "https://helm.goharbor.io"
      version: "1.6.0"
      force: false
      timeoutSeconds: 0
      values:
        stuff: andthangs
        database:
          external:
            password: ThisWillBeReplaced
      valuesFrom:
        - kind: Secret
          name: secret-values.yaml
          valuesKey: values.yaml
    

    The overlaying would happen just the same as overlay/values_patch.yaml would work.

    The fleet-agent on the downstream cluster would handle the final rendering of the bundle before applying the chart. This is to both prevent decrypted secrets from being in the bundle and allow each downstream cluster to have its own value for the secret-values.yaml.

    An alternative pattern could be to add something like vals to the downstream fleet-agent and allow vals type syntax in the values tree to reference secrets stored across multiple store types. Example:

    values:
        stuff: andthangs
        database:
          external:
            password: refs+vault://mycluster/harbor/database#/password
    

    Vals approach could require additional authentication secrets in each cluster's fleet namespace depending on the backend.

    Our current workaround is having fleet deploy helm-controller and source-controller then using fleet to apply helm-releases for charts that force this pattern. We'd like to consolidate to just fleet and drop the other controllers.

    QA/S kind/enhancement status/dev-validate 
    opened by ewokpoacher 11
  • Infinite Helm Release History

    Infinite Helm Release History

    I am seeing infinite helm release history with my fleet install.

    Somewhat related to rancher/fleet#233 but they appear to be different bugs. The MaxHistory 5 setting does not appear to be honored

    opened by boxboatmatt 11
  • Helm chart deployment using fleet bundle

    Helm chart deployment using fleet bundle

    Is there an existing issue for this?

    • [X] I have searched the existing issues

    Current Behavior

    I am trying to install rocketchat helm chart using a fleet bundle.

    Expected Behavior

    Helm chart installed

    Steps To Reproduce

    Try to install rocketchat using Fleet bundle.

    Working as manifes if using a helmchart object with same values (see bellow).

    Environment

    - Architecture: AMD64
    - Fleet Version: 0.3.11
    - Cluster: 
      - Provider: RKE2
      - Options: RKE2 with Rancher that deploy a "child" cluster using vmware provisionning 
      - Kubernetes Version: 1.24.4
    

    Logs

    Fleet.yaml asked:

    defaultNamespace: rocketchat
    
    helm:
      chart: rocketchat
      repo: https://rocketchat.github.io/helm-charts
      releaseName: "rocketchat"
      version: "5.1.0"
      namespace: rocketchat
    
    targetCustomizations:
    - name: mydeploiement
      helm:
        values:
          replica: 1
          mongodb:
            auth:
              rootPassword: "rootpassword"
              username: "username"
              password: "password"
      clusterGroup: myclustergroup
    

    Bundle results:

    status:
      conditions:
      - lastUpdateTime: "2022-09-23T12:45:55Z"
        message: 'execution error at (rocketchat/templates/secret.yaml:31:19): Please
          specify a MongoDB password'
        reason: Error
        status: "False"
        type: Processed
    

    This is working using this manifest instead of a fleet bundle :

    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: rocketchat
      namespace: kube-system
    spec:
      chart: rocketchat
      repo: https://rocketchat.github.io/helm-charts
      version: "5.1.0"
      targetNamespace: rocketchat
      valuesContent: |-
        persistence:
          enabled: true
        mongodb:
          auth:
            rootPassword: "rootpassword"
            username: "username"
            password: "password"
    
    

    Anything else?

    Thanks for your help!

    [zube]: To Triage kind/bug area/fleet team/fleet 
    opened by Leopol123 1
  • Support cluster repo change

    Support cluster repo change

    Refers to #948

    More information in the commit messages

    Tested this manually with manual and managed downstream agents. Changing the cluster resource, updates the agent with the configured repo.

    Refers to https://github.com/rancher/rancher/pull/38950

    opened by manno 0
  • Fleet is not upgrading properly on windows nodes in a cluster with windows workers

    Fleet is not upgrading properly on windows nodes in a cluster with windows workers

    Is there an existing issue for this?

    • [X] I have searched the existing issues

    Current Behavior

    After upgrading rancher with an existing cluster that is running rke2 with a windows node from 2.6.8 -> 2.6.9-rc1, (fleet v0.3.11 -> v0.4.0-rc3) , fleet fails to upgrade as it is consistently trying to run on the windows node.

    Expected Behavior

    Expecting fleet to work in a windows cluster after upgrade

    Steps To Reproduce

    1. Deploy a windows cluster on rancher v2.6.8, k8s v1.23.10+rke2r1
    2. upgrade rancher
    3. wait for fleet in the downstream windows cluster to finish upgrading

    Environment

    - Architecture:
    - Fleet Version: v0.3.11
    - Cluster: rke2
      - Provider: vsphere
      - Options: with windows worker pool
      - Kubernetes Version: v1.23.10+rke2r1
    

    Logs

    MountVolume.SetUp failed for volume "kube-api-access-9w88q" : chown c:\var\lib\kubelet\pods\bfdcae51-9200-440e-89c9-e55b9fb69ef7\volumes\kubernetes.io~projected\kube-api-access-9w88q\..2022_09_20_16_52_32.1316767762\token: not supported by windows
    

    Anything else?

    No response

    [zube]: To Triage kind/bug area/fleet team/fleet 
    opened by slickwarren 0
  • Implement Helm Chart Capability Check

    Implement Helm Chart Capability Check

    Fix #399

    This pull request resolves #399 by moving all Helm processing to fleet agents. To do this, the fleet agents report resources from the installs back to the fleet agent using the status of bundle deployments. The fleet controller then updates the corresponding bundle, and gitrepo with the resource keys.

    Test

    To test this pull request, I used the following gitrepo:

    apiVersion: fleet.cattle.io/v1alpha1
    kind: GitRepo
    metadata:
      name: fleet-examples-mc-helm-external
      namespace: fleet-default
    
    spec:
      branch: master
      insecureSkipTLSVerify: false
      paths:
        - multi-cluster/helm-external
      repo: 'https://github.com/rancher/fleet-examples'
      targets:
        - clusterSelector:
            matchExpressions:
              - key: provider.cattle.io
                operator: NotIn
                values:
                  - harvester
    

    I tested this on a 3 cluster system(1 controller and 2 agents) with each running Microk8s. I plan to deploy this pull request to our semi-production systems in the morning to get a larger test.

    Additional Information

    Tradeoff

    This approach adds some latency to resources appearing in Rancher's UI. However, it removes potential issues that can be caused by resources not being known to the Controller's Helm calls.

    Other

    This pull request is a redo of #864 as the original code/solution wasn't as functional/refined. I redid things and closed the original.

    opened by romejoe 0
  • Feature Request: Better control of when helmSecretName is used

    Feature Request: Better control of when helmSecretName is used

    Is your feature request related to a problem?

    Yes.

    We specify GitRepo definitions with helmSecretName to provide (reference to) credentials for accessing private chart registries. In these GitRepo definitions, we refer to several different paths to monitor, each containing its own "fleet.yaml".

    This works as long as each "fleet.yaml" refers to the same intended (private) registry source, but it is too easy to make a mistake and include a chart reference to some other registry (public or private). If such a mistake is made, the credentials are automatically sent in the request to that registry, which is really bad from a security perspective.

    Solution you'd like

    Make it possible to specify applicability/scope for the credentials referred to by helmSecretName, e.g. a URL or hostname regex.

    This could be done by introducing another key next to helmSecretName, e.g. helmSecretScope or similar.

    When this has been defined, the chart fetching in Fleet (go-getter?) should only provide credentials if the source matches what has been specified. Otherwise, no credentials must be provided.

    Alternatives you've considered

    The scope (i.e. regex) for each set of credentials could also be defined via some custom annotations in the Secrets object, which would imply that no new key has to be introduced in the GitRepo definition.

    Anything else?

    No response

    [zube]: To Triage kind/enhancement area/fleet team/fleet 
    opened by uivraeus 0
  • Watch Command Returns invalid JSON

    Watch Command Returns invalid JSON

    Is there an existing issue for this?

    • [X] I have searched the existing issues

    Current Behavior

    When watching BundleDeployments via a get request. Fleet will send multiple JSON objects representing different events such as modified, created, and deleted. Each object on its own is valid JSON, but when combined into one GET response, it is an invalid JSON list of objects. This causes issues when the client that sent the request tries to decode the entire response body since the response header specifies 'Content-Encoding: application/json' but the response body is invalid JSON.

    Current Output (... for brevity)

    {
        "type": "MODIFIED",
        "object": {
            "apiVersion": "fleet.cattle.io/v1alpha1",
            "kind": "BundleDeployment",
            "metadata": {...},
            "spec": {...},
            "status": {...}
            }
    }
    {
        "type": "MODIFIED",
        "object": {
            "apiVersion": "fleet.cattle.io/v1alpha1",
            "kind": "BundleDeployment",
            "metadata": {...},
            "spec": {...},
            "status": {...}
            }
    }
    {
        "type": "BOOKMARK",
        "object": {
            "apiVersion": "fleet.cattle.io/v1alpha1",
            "kind": "BundleDeployment",
            "metadata": {
                "resourceVersion": "6255"
            }
        }
    }
    {
        "type": "BOOKMARK",
        "object": {
            "apiVersion": "fleet.cattle.io/v1alpha1",
            "kind": "BundleDeployment",
            "metadata": {
                "resourceVersion": "6255"
            }
        }
    }
    {
        "type": "BOOKMARK",
        "object": {
            "apiVersion": "fleet.cattle.io/v1alpha1",
            "kind": "BundleDeployment",
            "metadata": {
                "resourceVersion": "6255"
            }
        }
    }
    
    

    Expected Behavior

    The watch parameter returns a valid JSON list. This could be done by pretending and appending [ and ] to the returned response.

    Desired Output (... for brevity)

    [
        {
            "type": "MODIFIED",
            "object": {
                "apiVersion": "fleet.cattle.io/v1alpha1",
                "kind": "BundleDeployment",
                "metadata": {...},
                "spec": {...},
                "status": {...}
                }
        },
        {
            "type": "MODIFIED",
            "object": {
                "apiVersion": "fleet.cattle.io/v1alpha1",
                "kind": "BundleDeployment",
                "metadata": {...},
                "spec": {...},
                "status": {...}
                }
        },
        {
            "type": "BOOKMARK",
            "object": {
                "apiVersion": "fleet.cattle.io/v1alpha1",
                "kind": "BundleDeployment",
                "metadata": {
                    "resourceVersion": "6255"
                }
            }
        },
        {
            "type": "BOOKMARK",
            "object": {
                "apiVersion": "fleet.cattle.io/v1alpha1",
                "kind": "BundleDeployment",
                "metadata": {
                    "resourceVersion": "6255"
                }
            }
        },
        {
            "type": "BOOKMARK",
            "object": {
                "apiVersion": "fleet.cattle.io/v1alpha1",
                "kind": "BundleDeployment",
                "metadata": {
                    "resourceVersion": "6255"
                }
            }
        }
    ]
    

    Steps To Reproduce

    Perform a request for bundleddeployments with watch=true and a timeout that allows for multiple object events. Example Watch Command from Rancher: fleet.cattle.io/v1alpha1/namespaces/cluster-fleet-default-c-8xpp6-eca429edeba1/bundledeployments?allowWatchBookmarks=true&resourceVersion=6251&timeout=10s&timeoutSeconds=10&watch=true

    Environment

    - Architecture: arm64
    - Fleet Version: 0.3.11
    - Cluster: 
      - Provider: Rancher 
      - Options: Downstream digital ocean RKE1
      - Kubernetes Version: 1.24
    

    Logs

    No response

    Anything else?

    Causes the following issues with rancher audit logs https://github.com/rancher/rancher/issues/38846

    [zube]: To Triage kind/bug area/fleet team/fleet 
    opened by KevinJoiner 1
Releases(v0.5.0-rc2)
Owner
Rancher
Rancher
Crossplane provider to provision and manage Kubernetes objects on (remote) Kubernetes clusters.

provider-kubernetes provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters ty

Crossplane Contrib 59 Sep 26, 2022
PolarDB Stack is a DBaaS implementation for PolarDB-for-Postgres, as an operator creates and manages PolarDB/PostgreSQL clusters running in Kubernetes. It provides re-construct, failover swtich-over, scale up/out, high-available capabilities for each clusters.

PolarDB Stack开源版生命周期 1 系统概述 PolarDB是阿里云自研的云原生关系型数据库,采用了基于Shared-Storage的存储计算分离架构。数据库由传统的Share-Nothing,转变成了Shared-Storage架构。由原来的N份计算+N份存储,转变成了N份计算+1份存储

null 22 Jul 18, 2022
🐶 Kubernetes CLI To Manage Your Clusters In Style!

K9s - Kubernetes CLI To Manage Your Clusters In Style! K9s provides a terminal UI to interact with your Kubernetes clusters. The aim of this project i

Fernand Galiana 18k Sep 26, 2022
Simple Tools to help manage non-production Kubernetes Clusters

SecondMate.io A tool to help your nonProduction Kubernetes Clusters running clean. The goal of this tool is to add some features to non production clu

Corey McGalliard 1 Feb 21, 2022
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds

Website • Quickstart • Documentation • Blog • Twitter • Slack vcluster - Virtual Clusters For Kubernetes Lightweight & Low-Overhead - Based on k3s, bu

Loft Labs 2k Oct 2, 2022
Kubernetes IN Docker - local clusters for testing Kubernetes

kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Kubernetes SIGs 10.5k Sep 30, 2022
provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters

provider-kubernetes provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters ty

International Business Machines 2 Jan 5, 2022
Kubernetes IN Docker - local clusters for testing Kubernetes

Please see Our Documentation for more in-depth installation etc. kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Kaisen Linux 0 Feb 14, 2022
Deploy, manage, and secure applications and resources across multiple clusters using CloudFormation and Shipa

CloudFormation provider Deploy, secure, and manage applications across multiple clusters using CloudFormation and Shipa. Development environment setup

Shipa 1 Feb 12, 2022
Large-scale Kubernetes cluster diagnostic tool.

English | 简体中文 KubeProber What is KubeProber? KubeProber is a diagnostic tool designed for large-scale Kubernetes clusters. It is used to perform diag

Erda 98 Sep 21, 2022
The OCI Service Operator for Kubernetes (OSOK) makes it easy to connect and manage OCI services from a cloud native application running in a Kubernetes environment.

OCI Service Operator for Kubernetes Introduction The OCI Service Operator for Kubernetes (OSOK) makes it easy to create, manage, and connect to Oracle

Oracle 23 Sep 16, 2022
PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.

GalaxyKube -- PolarDB-X Operator PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes. It follo

null 63 Sep 8, 2022
kubequery is a Osquery extension that provides SQL based analytics for Kubernetes clusters

kubequery powered by Osquery kubequery is a Osquery extension that provides SQL based analytics for Kubernetes clusters kubequery will be packaged as

Uptycs Inc 78 Sep 17, 2022
Validation of best practices in your Kubernetes clusters

Best Practices for Kubernetes Workload Configuration Fairwinds' Polaris keeps your clusters sailing smoothly. It runs a variety of checks to ensure th

Fairwinds 2.7k Sep 27, 2022
A best practices checker for Kubernetes clusters. 🤠

Clusterlint As clusters scale and become increasingly difficult to maintain, clusterlint helps operators conform to Kubernetes best practices around r

DigitalOcean 486 Sep 19, 2022
Kubernetes operator to autoscale Google's Cloud Bigtable clusters

Bigtable Autoscaler Operator Bigtable Autoscaler Operator is a Kubernetes Operator to autoscale the number of nodes of a Google Cloud Bigtable instanc

RD Station 22 Nov 5, 2021
Nebula Operator manages NebulaGraph clusters on Kubernetes and automates tasks related to operating a NebulaGraph cluster

Nebula Operator manages NebulaGraph clusters on Kubernetes and automates tasks related to operating a NebulaGraph cluster. It evolved from NebulaGraph Cloud Service, makes NebulaGraph a truly cloud-native database.

vesoft inc. 52 Aug 17, 2022
Client extension for interacting with Kubernetes clusters from your k6 tests.

⚠️ This is a proof of concept As this is a proof of concept, it won't be supported by the k6 team. It may also break in the future as xk6 evolves. USE

k6 19 Sep 8, 2022
Managing your Kubernetes clusters (including public, private, edge, etc) as easily as visiting the Internet

Clusternet Managing Your Clusters (including public, private, hybrid, edge, etc) as easily as Visiting the Internet. Clusternet (Cluster Internet) is

Clusternet 954 Sep 24, 2022