Storage Orchestration for Kubernetes

Overview

Rook

CNCF Status GitHub release Docker Pulls Go Report Card CII Best Practices Security scanning Slack Twitter Follow

What is Rook?

Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.

Rook turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.

Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience.

For more details about the storage solutions currently supported by Rook, please refer to the project status section below. We plan to continue adding support for other storage systems and environments based on community demand and engagement in future releases. See our roadmap for more details.

Rook is hosted by the Cloud Native Computing Foundation (CNCF) as a graduated level project. If you are a company that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how Rook plays a role, read the CNCF announcement.

Getting Started and Documentation

For installation, deployment, and administration, see our Documentation.

Contributing

We welcome contributions. See Contributing to get started.

Report a Bug

For filing bugs, suggesting improvements, or requesting new features, please open an issue.

Reporting Security Vulnerabilities

If you find a vulnerability or a potential vulnerability in Rook please let us know immediately at [email protected]. We'll send a confirmation email to acknowledge your report, and we'll send an additional email when we've identified the issues positively or negatively.

For further details, please see the complete security release process.

Contact

Please use the following to reach members of the community:

Community Meeting

A regular community meeting takes place every other Tuesday at 9:00 AM PT (Pacific Time). Convert to your local timezone.

Any changes to the meeting schedule will be added to the agenda doc and posted to Slack #announcements and the rook-dev mailing list.

Anyone who wants to discuss the direction of the project, design and implementation reviews, or general questions with the broader community is welcome and encouraged to join.

Project Status

The status of each storage provider supported by Rook can be found in the table below. Each API group is assigned its own individual status to reflect their varying maturity and stability. More details about API versioning and status in Kubernetes can be found on the Kubernetes API versioning page, but the key difference between the statuses are summarized below:

  • Alpha: The API may change in incompatible ways in a later software release without notice, recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
  • Beta: Support for the overall features will not be dropped, though details may change. Support for upgrading or migrating between versions will be provided, either through automation or manual steps.
  • Stable: Features will appear in released software for many subsequent versions and support for upgrading between versions will be provided with software automation in the vast majority of scenarios.
Name Details API Group Status
Ceph Ceph is a distributed storage system that provides file, block and object storage and is deployed in large scale production clusters. ceph.rook.io/v1 Stable

This repo is for the Ceph storage provider. The Cassandra and NFS storage providers moved to a separate repo to allow for each storage provider to have an independent development and release schedule.

Official Releases

Official releases of Rook can be found on the releases page. Please note that it is strongly recommended that you use official releases of Rook, as unreleased versions from the master branch are subject to changes and incompatibilities that will not be supported in the official releases. Builds from the master branch can have functionality changed and even removed at any time without compatibility support and without prior notice.

Licensing

Rook is under the Apache 2.0 license.

FOSSA Status

Issues
  • Very high CPU usage on Ceph OSDs (v1.0, v1.1)

    Very high CPU usage on Ceph OSDs (v1.0, v1.1)

    I am not sure where the problem is but I am seeing very high CPU usage since I started using v1.0.0. With three small clusters load average skyrockets to the 10s quite quickly making the nodes unusable. This happens while copying quite a bit of data to a volume mapped on the host bypassing k8s (to restore data from an existing non-k8s server). Nothing else is happening with the clusters at all. I am using low specs servers (2 cores, 8 GB of RAM) but I didn't see any of these high load issues with 0.9.3 on same-specs servers. Has something changed about Ceph or else that might explain this? I've also tried with two providers, Hetzner Cloud and UpCloud. Same issue when actually using a volume.

    Is it just me or is it happening to others as well? Thanks!

    bug ceph ceph-osd 
    opened by vitobotta 229
  • The rook-ceph-csi-config cm disappeared after host reboot

    The rook-ceph-csi-config cm disappeared after host reboot

    Is this a bug report or feature request?

    • Bug Report

    Deviation from expected behavior: After the host reboot, the configmap rook-ceph-csi-config disappeared. All the ceph-csi pods were in "ContainerCreating" state.

    Expected behavior: The configmap rook-ceph-csi-config should not be deleted in this condition or should be re-created if not found.

    How to reproduce it (minimal and precise):

    Run "reboot" on the one of the host in the cluster. The issue didn't surface each time but had occurred several times in our testing. Below is the failure condition.

    # knc get pods | egrep -v "Run|Com"
    NAME                                                            READY   STATUS              RESTARTS   AGE
    csi-cephfsplugin-6cb75                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-9whpq                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-bpn88                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-gd6kk                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-hbjkj                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-jt48j                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-mlj6w                                          0/3     ContainerCreating   0          15h
    csi-cephfsplugin-provisioner-67cdf965c6-764bx                   0/5     ContainerCreating   0          15h
    csi-cephfsplugin-provisioner-67cdf965c6-pq4wm                   0/5     ContainerCreating   0          15h
    csi-cephfsplugin-rx599                                          0/3     ContainerCreating   0          15h
    csi-rbdplugin-9v8kb                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-bccpt                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-bqlpc                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-f2fb9                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-h8hbc                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-l2wbz                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-njtt7                                             0/3     ContainerCreating   0          15h
    csi-rbdplugin-provisioner-78d6f54775-dq47m                      0/6     ContainerCreating   0          15h
    csi-rbdplugin-provisioner-78d6f54775-hc9nb                      0/6     ContainerCreating   0          15h
    csi-rbdplugin-tfn52                                             0/3     ContainerCreating   0          15h
    rook-ceph-detect-version-kpc2p                                  0/1     Init:0/1            0          1s
    
    # knc describe pod csi-rbdplugin-provisioner-78d6f54775-hc9nb
    ...
    Events:
      Type     Reason       Age                  From                                        Message
      ----     ------       ----                 ----                                        -------
      Warning  FailedMount  40m (x57 over 14h)   kubelet, tesla-cb0434-csd1-csd1-control-03  Unable to attach or mount volumes: unmounted volumes=[ceph-csi-config], unattached volumes=[rook-csi-rbd-provisioner-sa-token-q4kbn host-dev host-sys lib-modules ceph-csi-config keys-tmp-dir socket-dir]: timed out waiting for the condition
      Warning  FailedMount  36m (x58 over 14h)   kubelet, tesla-cb0434-csd1-csd1-control-03  Unable to attach or mount volumes: unmounted volumes=[ceph-csi-config], unattached volumes=[socket-dir rook-csi-rbd-provisioner-sa-token-q4kbn host-dev host-sys lib-modules ceph-csi-config keys-tmp-dir]: timed out waiting for the condition
      Warning  FailedMount  15m (x62 over 15h)   kubelet, tesla-cb0434-csd1-csd1-control-03  Unable to attach or mount volumes: unmounted volumes=[ceph-csi-config], unattached volumes=[keys-tmp-dir socket-dir rook-csi-rbd-provisioner-sa-token-q4kbn host-dev host-sys lib-modules ceph-csi-config]: timed out waiting for the condition
      Warning  FailedMount  67s (x453 over 15h)  kubelet, tesla-cb0434-csd1-csd1-control-03  MountVolume.SetUp failed for volume "ceph-csi-config" : configmap "rook-ceph-csi-config" not found
    
    
    # cephstatus
      cluster:
        id:     79580ff1-adf9-4d6a-a4c6-9dc44fe784c5
        health: HEALTH_OK
     
      services:
        mon: 3 daemons, quorum b,c,d (age 15h)
        mgr: a(active, since 15h)
        mds: myfs:1 {0=myfs-b=up:active} 1 up:standby-replay
        osd: 6 osds: 6 up (since 15h), 6 in (since 15h)
        rgw: 1 daemon active (rook.ceph.store.a)
     
      task status:
        scrub status:
            mds.myfs-a: idle
            mds.myfs-b: idle
     
      data:
        pools:   10 pools, 208 pgs
        objects: 295 objects, 22 MiB
        usage:   6.9 GiB used, 53 GiB / 60 GiB avail
        pgs:     208 active+clean
     
      io:
        client:   852 B/s rd, 1 op/s rd, 0 op/s wr
    
    

    File(s) to submit:

    • Cluster CR (custom resource), typically called cluster.yaml, if necessary
    • Operator's logs, if necessary
    2020-08-25 04:05:57.736755 I | rookcmd: starting Rook v1.3.9 with arguments '/usr/local/bin/rook ceph operator'
    2020-08-25 04:05:57.737076 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-cephfs-provisioner-sts-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-sts.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --csi-rbd-provisioner-sts-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-sts.yaml, --enable-discovery-daemon=true, --enable-flex-driver=false, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --master=, --mon-healthcheck-interval=45s, --mon-out-timeout=5m0s, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule=
    2020-08-25 04:05:57.737087 I | cephcmd: starting operator
    2020-08-25 04:05:57.801061 I | op-discover: rook-discover daemonset already exists, updating ...
    2020-08-25 04:05:57.828608 I | operator: rook-provisioner ceph.rook.io/block started using ceph.rook.io flex vendor dir
    I0825 04:05:57.828776      10 leaderelection.go:242] attempting to acquire leader lease  rook-ceph/ceph.rook.io-block...
    2020-08-25 04:05:57.828838 I | operator: rook-provisioner rook.io/block started using rook.io flex vendor dir
    ...
    2020-08-25 04:05:59.546300 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (env var)
    2020-08-25 04:05:59.571085 E | ceph-block-pool-controller: failed to reconcile invalid pool CR "csireplpool" spec: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    : exit status 1
    2020-08-25 04:05:59.945701 I | op-mon: parsing mon endpoints: d=10.254.140.99:6789,b=10.254.27.205:6789,c=10.254.144.51:6789
    2020-08-25 04:06:00.229308 W | cephclient: failed to get ceph daemons versions, this likely means there is no cluster yet. failed to run 'ceph versions: exit status 1
    2020-08-25 04:06:00.345543 I | op-mon: parsing mon endpoints: d=10.254.140.99:6789,b=10.254.27.205:6789,c=10.254.144.51:6789
    2020-08-25 04:06:00.537510 E | ceph-file-controller: failed to reconcile invalid object filesystem "myfs" arguments: invalid metadata pool: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    : exit status 1
    2020-08-25 04:06:00.631651 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
    2020-08-25 04:06:00.632192 I | ceph-csi: detecting the ceph csi image version for image "bcmt-registry:5000/csi/cephcsi:v2.1.2"
    2020-08-25 04:06:00.821972 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="- effect: NoExecute\n  key: is_control\n  operator: Equal\n  value: \"true\"\n- effect: NoExecute\n  key: is_edge\n  operator: Equal\n  value: \"true\"\n- effect: NoExecute\n  key: is_storage\n  operator: Equal\n  value: \"true\"\n- effect: NoSchedule\n  key: node.cloudprovider.kubernetes.io/uninitialized\n  operator: Equal\n  value: \"true\"\n" (env var)
    2020-08-25 04:06:01.028714 W | cephclient: failed to get ceph daemons versions, this likely means there is no cluster yet. failed to run 'ceph versions: exit status 1
    2020-08-25 04:06:01.127681 E | ceph-block-pool-controller: failed to reconcile invalid pool CR "csireplpool" spec: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    : exit status 1
    2020-08-25 04:06:01.326682 E | ceph-object-controller: failed to reconcile invalid object store "rook-ceph-store" arguments: invalid metadata pool spec: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    : exit status 1
    2020-08-25 04:06:01.545112 I | op-mon: parsing mon endpoints: d=10.254.140.99:6789,b=10.254.27.205:6789,c=10.254.144.51:6789
    2020-08-25 04:06:01.545202 I | op-cluster: cluster info loaded for monitoring: &{FSID:79580ff1-adf9-4d6a-a4c6-9dc44fe784c5 MonitorSecret:AQDEckRfaObYIhAAfu5txBedGHfueBAZddUAzg== AdminSecret:AQDEckRfnro2LhAABI76dGcXtkM1BBtXpfHCDA== ExternalCred:{Username: Secret:} Name:rook-ceph Monitors:map[b:0xc00000c580 c:0xc00000c780 d:0xc00000c2c0] CephVersion:{Major:0 Minor:0 Extra:0 Build:0}}
    2020-08-25 04:06:01.545210 I | op-cluster: enabling cluster monitoring goroutines
    2020-08-25 04:06:01.545216 I | op-client: start watching client resources in namespace "rook-ceph"
    2020-08-25 04:06:02.146208 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (env var)
    2020-08-25 04:06:02.146238 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket"
    2020-08-25 04:06:02.147645 I | op-cluster: ceph status check interval is 60s
    I0825 04:06:02.147724      10 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner"  "name"="rook-ceph.ceph.rook.io/bucket"
    2020-08-25 04:06:02.356831 I | op-mon: parsing mon endpoints: d=10.254.140.99:6789,b=10.254.27.205:6789,c=10.254.144.51:6789
    2020-08-25 04:06:02.737032 E | ceph-block-pool-controller: failed to reconcile invalid pool CR "csireplpool" spec: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    : exit status 1
    2020-08-25 04:06:02.821955 E | op-cluster: failed to get ceph status. failed to get status. . Error initializing cluster client: ObjectNotFound('error calling conf_read_file',): exit status 1
    2020-08-25 04:06:02.852749 I | op-config: CephCluster "rook-ceph" status: "Failure". "Failed to configure ceph cluster"
    2020-08-25 04:06:02.939369 W | cephclient: failed to get ceph daemons versions, this likely means there is no cluster yet. failed to run 'ceph versions: exit status 1
    2020-08-25 04:06:02.945847 I | op-mon: parsing mon endpoints: d=10.254.140.99:6789,b=10.254.27.205:6789,c=10.254.144.51:6789
    2020-08-25 04:06:03.463716 W | cephclient: failed to get ceph daemons versions, this likely means there is no cluster yet. failed to run 'ceph versions: exit status 1
    2020-08-25 04:06:03.523955 E | ceph-file-controller: failed to reconcile invalid object filesystem "myfs" arguments: invalid metadata pool: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    : exit status 1
    2020-08-25 04:06:03.737853 I | ceph-spec: ceph-block-pool-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{"HEALTH_ERR" map["error":{"Urgent" "failed to get status. . Error initializing cluster client: ObjectNotFound('error calling conf_read_file',): exit status 1"}] "2020-08-25T04:06:02Z" "2020-08-25T04:06:02Z" "HEALTH_OK"}
    2020-08-25 04:06:03.755237 E | ceph-object-controller: failed to reconcile invalid object store "rook-ceph-store" arguments: invalid metadata pool spec: failed to get crush map: failed to get crush map. Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
    
    • Crashing pod(s) logs, if necessary

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read Github documentation if you need help.

    Environment:

    • OS (e.g. from /etc/os-release): Rhel 7.8
    • Kernel (e.g. uname -a): Linux tesla-cb0434-csd1-csd1-control-01 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Wed Feb 26 03:08:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    • Cloud provider or hardware configuration: openstack
    • Rook version (use rook version inside of a Rook Pod): v1.3.9
    • Storage backend version (e.g. for ceph do ceph -v): v14.2.10
    • Kubernetes version (use kubectl version): v1.18.8
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Tectonic
    • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK
    bug 
    opened by yanchicago 76
  • Operator not happy after upgrade

    Operator not happy after upgrade

    Is this a bug report or feature request?

    • Bug Report

    Deviation from expected behavior: When restarted, operator checks all OSDs are running, but at some point fails with:

    2018-08-17 01:22:22.635203 I | op-k8sutil: updating deployment rook-ceph-osd-id-130
    2018-08-17 01:22:24.666490 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-id-130
    2018-08-17 01:22:24.666509 I | op-osd: started deployment for osd 130 (dir=false, type=bluestore)
    2018-08-17 01:22:24.669040 I | op-osd: osd orchestration status for node ps-100g.sdsu.edu is starting
    2018-08-17 01:22:24.669056 I | op-osd: osd orchestration status for node siderea.ucsc.edu is starting
    2018-08-17 01:22:24.669061 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:24.670869 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:24.770976 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:24.771848 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:24.811933 I | op-provisioner: Deleting volume pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994
    2018-08-17 01:22:24.811952 I | exec: Running command: rbd rm rbd/pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994 --cluster= --conf=/var/lib/rook/.config --keyring=/v
    ar/lib/rook/client.admin.keyring
    2018-08-17 01:22:24.812100 I | op-provisioner: Deleting volume pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994
    2018-08-17 01:22:24.812118 I | exec: Running command: rbd rm rbd/pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994 --cluster= --conf=/var/lib/rook/.config --keyring=/v
    ar/lib/rook/client.admin.keyring
    2018-08-17 01:22:24.812127 I | op-provisioner: Deleting volume pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994
    2018-08-17 01:22:24.812144 I | exec: Running command: rbd rm rbd/pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994 --cluster= --conf=/var/lib/rook/.config --keyring=/v
    ar/lib/rook/client.admin.keyring
    E0817 01:22:24.825364       7 controller.go:1044] Deletion of volume "pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994" failed: Failed to delete rook block image rbd/
    pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994: failed to delete image pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994 in pool rbd: Failed to complete '': exit status 1. g
    lobal_init: unable to open config file from search list /var/lib/rook/.config
    . output:
    E0817 01:22:24.825422       7 goroutinemap.go:165] Operation for "delete-pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994[6aec4ddb-8ea0-11e8-92ef-0cc47a6be994]" faile
    d. No retries permitted until 2018-08-17 01:24:26.825404385 +0000 UTC m=+497.173393014 (durationBeforeRetry 2m2s). Error: Failed to delete rook block image rb
    d/pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994: failed to delete image pvc-5b9e9df4-8ea0-11e8-92ef-0cc47a6be994 in pool rbd: Failed to complete '': exit status 1.
     global_init: unable to open config file from search list /var/lib/rook/.config
    . output:
    E0817 01:22:24.825753       7 controller.go:1044] Deletion of volume "pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994" failed: Failed to delete rook block image rbd/pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994: failed to delete image pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994 in pool rbd: Failed to complete '': exit status 1. global_init: unable to open config file from search list /var/lib/rook/.config
    . output:
    E0817 01:22:24.825791       7 goroutinemap.go:165] Operation for "delete-pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994[5ffa47ae-a0d0-11e8-8e30-0cc47a6be994]" failed. No retries permitted until 2018-08-17 01:24:26.825780946 +0000 UTC m=+497.173769575 (durationBeforeRetry 2m2s). Error: Failed to delete rook block image rbd/pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994: failed to delete image pvc-584f3e18-a0d0-11e8-8e30-0cc47a6be994 in pool rbd: Failed to complete '': exit status 1. global_init: unable to open config file from search list /var/lib/rook/.config
    . output:
    E0817 01:22:24.825958       7 controller.go:1044] Deletion of volume "pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994" failed: Failed to delete rook block image rbd/pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994: failed to delete image pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994 in pool rbd: Failed to complete '': exit status 1. global_init: unable to open config file from search list /var/lib/rook/.config
    . output:
    E0817 01:22:24.825996       7 goroutinemap.go:165] Operation for "delete-pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994[119e6a37-9f46-11e8-92ef-0cc47a6be994]" failed. No retries permitted until 2018-08-17 01:24:26.825984549 +0000 UTC m=+497.173973182 (durationBeforeRetry 2m2s). Error: Failed to delete rook block image rbd/pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994: failed to delete image pvc-0b1bb4e5-9f46-11e8-92ef-0cc47a6be994 in pool rbd: Failed to complete '': exit status 1. global_init: unable to open config file from search list /var/lib/rook/.config
    . output:
    2018-08-17 01:22:24.871944 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:24.872769 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:24.972851 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:24.973845 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.073929 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.074814 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.174891 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.175758 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.275835 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.276647 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.376740 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.377564 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.477646 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.478442 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.578526 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.579283 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.679364 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.680127 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.780216 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.781114 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.881202 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.882004 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:25.982083 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:25.982835 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:26.082912 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:26.083701 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:26.183800 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:26.184550 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:26.284634 I | op-osd: 12/14 node(s) completed osd provisioning
    2018-08-17 01:22:26.285400 I | op-osd: orchestration status config map result channel closed, will restart watch.
    2018-08-17 01:22:26.385449 I | op-osd: 12/14 node(s) completed osd provisioning
    

    After this it keeps printing this message about channel closed.

    Expected behavior: Operator function normally after the upgrade 0.7.1->0.8.1

    Environment:

    • OS (e.g. from /etc/os-release): CentOS 7.5
    • Kernel (e.g. uname -a): 4.14.14-1.el7.elrepo.x86_64
    • Cloud provider or hardware configuration: BAremetal
    • Rook version (use rook version inside of a Rook Pod): 0.8.1
    • Kubernetes version (use kubectl version): 1.11.2
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm
    • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_WARN noscrub,nodeep-scrub flag(s) set
    ceph operator 
    opened by dimm0 74
  • cephfs storageclass does not work

    cephfs storageclass does not work

    Is this a bug report or feature request?

    • Bug Report

    Deviation from expected behavior: wp-pv-claim and mysql-pv-claim is Bound, but the cephfs-pvc is Pending

    Expected behavior: The cephfs-pvc works

    How to reproduce it (minimal and precise):

    # microk8s  v1.14.1 on Ubuntu 16.04.5 LTS (Xenial Xerus)
    kubectl apply -f ceph/common.yaml
    kubectl apply -f ceph/operator.yaml
    kubectl apply -f ceph/cluster-test.yaml
    kubectl apply -f ceph/toolbox.yaml
    kubectl apply -f ceph/csi/rbd/storageclass-test.yaml
    kubectl apply -f . # install mysql.yaml wordpress.yaml
    kubectl apply -f ceph/object-test.yaml
    kubectl apply -f ceph/object-user.yaml
    kubectl apply -f ceph/filesystem-test.yaml
    kubectl apply -f ceph/csi/cephfs/storageclass.yaml
    kubectl apply -f ceph/csi/cephfs/kube-registry.yaml
    

    File(s) to submit:

    • Cluster CR (custom resource), typically called cluster.yaml, if necessary
    • Operator's logs, if necessary
    • Crashing pod(s) logs, if necessary
    # kubectl -n kube-system describe pvc cephfs-pvc
    Name:          cephfs-pvc
    Namespace:     kube-system
    StorageClass:  csi-cephfs
    Status:        Pending
    Volume:        
    Labels:        <none>
    Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cephfs-pvc","namespace":"kube-system"},"spec":{"acc...
                   volume.beta.kubernetes.io/storage-provisioner: rook-ceph.cephfs.csi.ceph.com
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      
    Access Modes:  
    VolumeMode:    Filesystem
    Mounted By:    kube-registry-5b9c9854c5-psdsv
                   kube-registry-5b9c9854c5-r2g4l
                   kube-registry-5b9c9854c5-rmqms
    Events:
      Type     Reason                Age                   From                                                                                                             Message
      ----     ------                ----                  ----                                                                                                             -------
      Warning  ProvisioningFailed    5m10s (x11 over 38m)  rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-f64c4574b-mvb7p_cc13f063-e0f5-11e9-813c-56d3e713ca9f  failed to provision volume with StorageClass "csi-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
      Normal   ExternalProvisioning  72s (x162 over 41m)   persistentvolume-controller                                                                                      waiting for a volume to be created, either by external provisioner "rook-ceph.cephfs.csi.ceph.com" or manually created by system administrator
      Normal   Provisioning          10s (x12 over 41m)    rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-f64c4574b-mvb7p_cc13f063-e0f5-11e9-813c-56d3e713ca9f  External provisioner is provisioning volume for claim "kube-system/cephfs-pvc"
    
    
    # kubectl -n rook-ceph logs csi-cephfsplugin-provisioner-f64c4574b-mvb7p -c csi-provisioner
    I0927 08:03:15.230702       1 connection.go:183] GRPC response: {}
    I0927 08:03:15.231643       1 connection.go:184] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
    I0927 08:03:15.231740       1 controller.go:979] Final error received, removing PVC 264abaa4-e0f7-11e9-bead-ac1f6b84bde2 from claims in progress
    W0927 08:03:15.231762       1 controller.go:886] Retrying syncing claim "264abaa4-e0f7-11e9-bead-ac1f6b84bde2", failure 11
    E0927 08:03:15.231801       1 controller.go:908] error syncing claim "264abaa4-e0f7-11e9-bead-ac1f6b84bde2": failed to provision volume with StorageClass "csi-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
    I0927 08:03:15.231842       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"kube-system", Name:"cephfs-pvc", UID:"264abaa4-e0f7-11e9-bead-ac1f6b84bde2", APIVersion:"v1", ResourceVersion:"17010356", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
    
    # ceph -s    
      cluster:
        id:     9acd086e-493b-4ebd-a39f-2be2cce80080
        health: HEALTH_OK
     
      services:
        mon: 1 daemons, quorum a (age 62m)
        mgr: a(active, since 61m)
        mds: myfs:1 {0=myfs-a=up:active} 1 up:standby-replay
        osd: 1 osds: 1 up (since 61m), 1 in (since 61m)
        rgw: 1 daemon active (my.store.a)
     
      data:
        pools:   9 pools, 72 pgs
        objects: 407 objects, 457 MiB
        usage:   1.1 TiB used, 611 GiB / 1.7 TiB avail
        pgs:     72 active+clean
     
      io:
        client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
    

    Environment:

    • OS (e.g. from /etc/os-release): microk8s v1.14.1 on Ubuntu 16.04.5 LTS (Xenial Xerus)
    • Kernel (e.g. uname -a): Linux ubun 4.15.0-62-generic #69~16.04.1-Ubuntu SMP Fri Sep 6 02:43:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
    • Cloud provider or hardware configuration:
    • Rook version (use rook version inside of a Rook Pod): rook: v1.1.1
    • Storage backend version (e.g. for ceph do ceph -v): ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
    • Kubernetes version (use kubectl version):Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
    • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):
    bug wontfix ceph 
    opened by veezhang 65
  • Recovering Rook cluster after Kubernetes cluster loss

    Recovering Rook cluster after Kubernetes cluster loss

    Is this a bug report or feature request?

    • Feature Request

    Feature Request

    Are there any similar features already existing: n/a

    What should the feature do: Assuming I have backup of rook directory from nodes and I lost my kubernetes cluster I would like to be able to restore persistent volumes after kubernetes cluster redeployment making sure at the same time that existing PVs will be claimed by pods in new cluster

    What would be solved through this feature: Make possible to not lose all the data with lost of the kubernetes cluster

    Does this have an impact on existing features:

    Environment:

    • OS (e.g. from /etc/os-release):
    • Kernel (e.g. uname -a):
    • Cloud provider or hardware configuration:
    • Rook version (use rook version inside of a Rook Pod):
    • Kubernetes version (use kubectl version):
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare Metal/(VMs) with CoreOS
    • Ceph status (use ceph health in the Rook toolbox):
    docs reliability troubleshooting 
    opened by michalzxc 62
  • OSDs crashlooping after being OOMKilled: bind unable to bind

    OSDs crashlooping after being OOMKilled: bind unable to bind

    Is this a bug report or feature request?

    • Bug Report

    Deviation from expected behavior: Updated ceph from v14.2.2-20190722 to v14.2.4-20190917, which seems to have made some changes in memory management, and nodes started getting system OOMKilles followed by OSDs crashlooping.

    2019-09-26 01:20:10.118 7f70aa104dc0 -1 Falling back to public interface
    2019-09-26 01:20:10.128 7f70aa104dc0 -1  Processor -- bind unable to bind to v2:10.244.15.18:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-09-26 01:20:10.128 7f70aa104dc0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds
    2019-09-26 01:20:15.137 7f70aa104dc0 -1  Processor -- bind unable to bind to v2:10.244.15.18:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-09-26 01:20:15.137 7f70aa104dc0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds
    2019-09-26 01:20:20.144 7f70aa104dc0 -1  Processor -- bind unable to bind to v2:10.244.15.18:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-09-26 01:20:20.144 7f70aa104dc0 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
    

    How to reproduce it (minimal and precise): Get an OSD OOMKilled by system

    • Rook version (use rook version inside of a Rook Pod): 1.1.1
    • Storage backend version (e.g. for ceph do ceph -v): v14.2.4-20190917
    • Kubernetes version (use kubectl version): 1.15.1
    bug 
    opened by dimm0 54
  • OSD Processor -- bind unable to bind to IP on any port in range 6800-7300: (99) Cannot assign requested address

    OSD Processor -- bind unable to bind to IP on any port in range 6800-7300: (99) Cannot assign requested address

    Is this a bug report or feature request?

    • Bug Report

    Deviation from expected behavior:

    I'm running OpenShift v3.11.82 on 3 nodes using RHEL7 with FLEXVOLUME_DIR_PATH set to /usr/libexec/kubernetes/kubelet-plugins/volume/exec(no other changes to *.yml files). I've encountered OSD error "Processor -- bind unable to bind to v2:IP:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address" on all my 3 OSD.

    Expected behavior:

    Bind to address and start to listen on port. rook+ceph should work as expexted.

    How to reproduce it (minimal and precise):

    Environment:

    • OS (e.g. from /etc/os-release): RHEL7
    • Kernel (e.g. uname -a): Linux node2.system10.vlan124.mcp 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GN
    • Cloud provider or hardware configuration: On-prem (VMVare based)
    • Rook version (use rook version inside of a Rook Pod): 1.0.0
    • Kubernetes version (use kubectl version): OpenShift v.3.11.82 (aka. K8s v1.11)
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): OpenShift
    • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

    LOGS

    rook-ceph-osd-0

    2019-05-08 17:01:38.770518 I | rookcmd: starting Rook v1.0.0-13.g05b0166 with arguments '/rook/rook ceph osd start -- --foreground --id 0 --osd-uuid c05a22dc-97e5-4463-b4e6-ebd9b00d0f2e --conf /var/lib/rook/osd0/rook-ceph.config --cluster ceph --default-log-to-file false'
    2019-05-08 17:01:38.776349 I | rookcmd: flag values: --help=false, --log-flush-frequency=5s, --log-level=INFO, --osd-id=0, --osd-store-type=bluestore, --osd-uuid=c05a22dc-97e5-4463-b4e6-ebd9b00d0f2e
    2019-05-08 17:01:38.776480 I | op-mon: parsing mon endpoints: 
    2019-05-08 17:01:38.776577 W | op-mon: ignoring invalid monitor 
    2019-05-08 17:01:38.784163 I | exec: Running command: stdbuf -oL ceph-volume lvm activate --no-systemd --bluestore 0 c05a22dc-97e5-4463-b4e6-ebd9b00d0f2e
    2019-05-08 17:01:39.683863 I | Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
    2019-05-08 17:01:40.012259 I | Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-0
    2019-05-08 17:01:40.312664 I | Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
    2019-05-08 17:01:40.648273 I | Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-aaa9dfa1-6146-4a9d-9b19-781bf0f71abe/osd-data-06f7e1bb-336c-4b7f-860f-39ce2217ef5c --path /var/lib/ceph/osd/ceph-0 --no-mon-config
    2019-05-08 17:01:41.200711 I | Running command: /bin/ln -snf /dev/ceph-aaa9dfa1-6146-4a9d-9b19-781bf0f71abe/osd-data-06f7e1bb-336c-4b7f-860f-39ce2217ef5c /var/lib/ceph/osd/ceph-0/block
    2019-05-08 17:01:41.508309 I | Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
    2019-05-08 17:01:41.813803 I | Running command: /bin/chown -R ceph:ceph /dev/mapper/ceph--aaa9dfa1--6146--4a9d--9b19--781bf0f71abe-osd--data--06f7e1bb--336c--4b7f--860f--39ce2217ef5c
    2019-05-08 17:01:42.097564 I | Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
    2019-05-08 17:01:42.388105 I | --> ceph-volume lvm activate successful for osd ID: 0
    2019-05-08 17:01:42.400127 I | exec: Running command: ceph-osd --foreground --id 0 --osd-uuid c05a22dc-97e5-4463-b4e6-ebd9b00d0f2e --conf /var/lib/rook/osd0/rook-ceph.config --cluster ceph --default-log-to-file false
    2019-05-08 17:01:43.178290 I | 2019-05-08 17:01:43.177 7fe1fd660d80 -1 Falling back to public interface
    2019-05-08 17:01:43.829540 I | 2019-05-08 17:01:43.829 7fe1fd660d80 -1 osd.0 69 log_to_monitors {default=true}
    2019-05-08 17:01:43.847126 I | 2019-05-08 17:01:43.846 7fe1efef6700 -1 osd.0 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
    2019-05-08 17:01:54.275215 I | 2019-05-08 17:01:54.274 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:01:54.275251 I | 2019-05-08 17:01:54.274 7fe1e5ee2700 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
    2019-05-08 17:01:59.291147 I | 2019-05-08 17:01:59.290 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:01:59.291193 I | 2019-05-08 17:01:59.290 7fe1e5ee2700 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
    2019-05-08 17:02:04.308557 I | 2019-05-08 17:02:04.308 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:04.308596 I | 2019-05-08 17:02:04.308 7fe1e5ee2700 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
    2019-05-08 17:02:04.320883 I | 2019-05-08 17:02:04.320 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:04.320920 I | 2019-05-08 17:02:04.320 7fe1e5ee2700 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
    2019-05-08 17:02:09.332874 I | 2019-05-08 17:02:09.332 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:09.332911 I | 2019-05-08 17:02:09.332 7fe1e5ee2700 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
    2019-05-08 17:02:14.345173 I | 2019-05-08 17:02:14.344 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:14.345214 I | 2019-05-08 17:02:14.344 7fe1e5ee2700 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
    2019-05-08 17:02:14.359483 I | 2019-05-08 17:02:14.358 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:14.359524 I | 2019-05-08 17:02:14.358 7fe1e5ee2700 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
    2019-05-08 17:02:19.372082 I | 2019-05-08 17:02:19.371 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:19.372114 I | 2019-05-08 17:02:19.371 7fe1e5ee2700 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
    2019-05-08 17:02:24.385045 I | 2019-05-08 17:02:24.384 7fe1e5ee2700 -1  Processor -- bind unable to bind to v2:10.130.0.1:7300/2027 on any port in range 6800-7300: (99) Cannot assign requested address
    2019-05-08 17:02:24.385079 I | 2019-05-08 17:02:24.384 7fe1e5ee2700 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
    2019-05-08 17:02:24.385338 I | 2019-05-08 17:02:24.384 7fe1f3786700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
    2019-05-08 17:02:24.385351 I | 2019-05-08 17:02:24.384 7fe1f3786700 -1 osd.0 74 *** Got signal Interrupt ***
    

    osd-0 config

    [global]
    fsid                      = e9f14792-30db-427d-8537-73e5b73a91ac
    run dir                   = /var/lib/rook/osd0
    mon initial members       = c a b
    mon host                  = v1:172.30.246.255:6789,v1:172.30.152.231:6789,v1:172.30.117.179:6789
    public addr               = 10.130.0.19
    cluster addr              = 10.130.0.19
    mon keyvaluedb            = rocksdb
    mon_allow_pool_delete     = true
    mon_max_pg_per_osd        = 1000
    debug default             = 0
    debug rados               = 0
    debug mon                 = 0
    debug osd                 = 0
    debug bluestore           = 0
    debug filestore           = 0
    debug journal             = 0
    debug leveldb             = 0
    filestore_omap_backend    = rocksdb
    osd pg bits               = 11
    osd pgp bits              = 11
    osd pool default size     = 1
    osd pool default min size = 1
    osd pool default pg num   = 100
    osd pool default pgp num  = 100
    osd objectstore           = filestore
    crush location            = root=default host=node2-system10-vlan124-mcp
    rbd_default_features      = 3
    fatal signal handlers     = false
    
    [osd.0]
    keyring              = /var/lib/ceph/osd/ceph-0/keyring
    bluestore block path = /var/lib/ceph/osd/ceph-0/block
    
    bug ceph 
    opened by grzechukol 52
  • mgr pod in CrashLoop in 0.8.x

    mgr pod in CrashLoop in 0.8.x

    Is this a bug report or feature request? Bug Report

    Deviation from expected behavior: After rescheduling the mgr pod, it goes into a CrashLoop with the following:

    2018-09-30 19:26:48.956022 I | ceph-mgr: 2018-09-30 19:26:48.955809 7f0adebf4700  1 mgr send_beacon active
    2018-09-30 19:26:50.970860 I | ceph-mgr: 2018-09-30 19:26:50.970649 7f0adebf4700  1 mgr send_beacon active
    2018-09-30 19:26:52.985827 I | ceph-mgr: 2018-09-30 19:26:52.985611 7f0adebf4700  1 mgr send_beacon active
    2018-09-30 19:26:54.004538 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTING
    2018-09-30 19:26:54.004566 I | ceph-mgr: CherryPy Checker:
    2018-09-30 19:26:54.004575 I | ceph-mgr: The Application mounted at '' has an empty config.
    2018-09-30 19:26:54.004581 I | ceph-mgr: 
    2018-09-30 19:26:54.004588 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Started monitor thread '_TimeoutMonitor'.
    2018-09-30 19:26:54.004594 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTING
    2018-09-30 19:26:54.004600 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Started monitor thread '_TimeoutMonitor'.
    2018-09-30 19:26:54.004606 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Serving on :::7000
    2018-09-30 19:26:54.004611 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTED
    2018-09-30 19:26:54.004624 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Serving on :::9283
    2018-09-30 19:26:54.004630 I | ceph-mgr: [30/Sep/2018:19:26:47] ENGINE Bus STARTED
    2018-09-30 19:26:54.004636 I | ceph-mgr: terminate called after throwing an instance of 'std::out_of_range'
    2018-09-30 19:26:54.004644 I | ceph-mgr:   what():  map::at
    failed to run mgr. failed to start mgr: Failed to complete 'ceph-mgr': signal: aborted (core dumped).
    

    Expected behavior: No crash loop ;)

    How to reproduce it (minimal and precise): Personally I've experienced it in several test clusters, but haven't had time to dig into it until tonight. Another user mentioned this a week ago on slack and @galexrt pointed to this bug in Ceph that seems to be related: https://tracker.ceph.com/issues/24982

    In that issue, people mention multiple RGWs and I'm running RGWs as a daemonset for these clusters. So I tried to scale the amount of RGWs down to 1, using NodeAffinity, and nor the mgr was able to start up. After it's started, I can scale the RGWs back up to full count (5 on testing cluster) in one go and the mgr stays up. Without knowing this in depth, it seems to me the RGWs are building up a history of metrics to deliver to the mgr when they can't reach it. When the mgr starts again, these historic metrics overwhelms it and it gets startled, not to confuse with started.

    Environment:

    • OS (e.g. from /etc/os-release): CoreOS
    • Kernel (e.g. uname -a): Something new
    • Cloud provider or hardware configuration: Bare metal
    • Rook version (use rook version inside of a Rook Pod): v0.8.1 - 99% sure I saw it on 0.8.2 as well, but had to vacate that due to a different bug.
    • Kubernetes version (use kubectl version): 1.11.3
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Kubespray
    • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): Seems happy, but no dashboard and metrics obviously.
    wontfix ceph objectstore 
    opened by bjornryden 52
  • OpenShift: insufficient permission inside the containers

    OpenShift: insufficient permission inside the containers

    Bug Report

    What happened:

    When trying to create a cluster the operator fails with:

    op-cluster: failed to create cluster in namespace rook. failed to start the mons. failed to initialize ceph cluster info. failed to get cluster info. failed to create mon secret
    s. failed to create dir /var/lib/rook/rook. mkdir /var/lib/rook: permission denied
    

    What you expected to happen:

    Cluster creation should succeed.

    Additional information:

    OpenShift uses the following feature to get fewer user privileges on application development where the expected user is 'root', see https://blog.openshift.com/jupyter-on-openshift-part-6-running-as-an-assigned-user-id/

    How to reproduce it (minimal and precise):

    Simply run kubectl create -f rook-cluster.yml

    Environment:

    • OS (e.g. from /etc/os-release): CentOS Linux release 7.4.1708 (Core)
    • Kernel (e.g. uname -a): Linux k8s-master.example.com 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
    • Cloud provider or hardware configuration: VM 1 CPU, 4GB RAM
    • Rook version (use rook version inside of a Rook Pod): v0.6.0-80.g3dfb151
    • Kubernetes version (use kubectl version): v1.7.6+a08f5eeb62
    • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): OpenShift
    • Ceph status (use ceph health in the Rook toolbox): no cluster yet
    security 
    opened by leseb 49
  • rook cluster fails after restart k8s node

    rook cluster fails after restart k8s node

    i'm created operator, cluster and filesystem:

                  apiVersion: rook.io/v1alpha1
                  kind: Filesystem
                  metadata:
                    name: rookfs
                    namespace: rook-cluster
                  spec:
                    metadataPool:
                      replicated:
                        size: 3
                    dataPools:
                      - erasureCoded:
                         codingChunks: 1
                         dataChunks: 2
                    metadataServer:
                      activeCount: 1
                      activeStandby: true
    

    and try to connect to it:

    apiVersion: v1
    kind: Pod
    metadata:
      name: ceph-tools
      namespace: rook-cluster
    spec:
      containers:
      - name: ceph-tools
        image: nginx:1.13.5-alpine
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: ceph-store
            mountPath: /srv/ceph
      volumes:
        - name: ceph-store
          flexVolume:
            driver: rook.io/rook
            fsType: ceph
            options:
              fsName: rookfs
              clusterName: rook-cluster
              path: /
    

    and have such error:

      12m   1m      6       kubelet, 10.1.29.25             Warning FailedMount     Unable to mount volumes for pod "ceph-tools_rook-cluster(6ac879d0-d672-11e7-88d3-0050569d5b15)": timeout expired waiting for volumes to attach/mount for pod "rook-cluster"/"ceph-tools". list of unattached/unmounted volumes=[ceph-store]
      12m   1m      6       kubelet, 10.1.29.25             Warning FailedSync      Error syncing pod
      9m    13s     2       kubelet, 10.1.29.25             Warning FailedMount     MountVolume.SetUp failed for volume "ceph-store" : mount command failed, status: Failure, reason: failed to mount filesystem rookfs to /var/lib/kubelet/pods/6ac879d0-d672-11e7-88d3-0050569d5b15/volumes/rook.io~rook/ceph-store with monitor 10.3.183.65:6790,10.3.65.12:6790,10.3.131.61:6790:/ and options [name=admin secret=AQBNCiFa0oqjKhAA2WH+8VwHB0y17Irg6HCmIw== mds_namespace=rookfs]: mount failed: exit status 32
    Mounting command: mount
    Mounting arguments: 10.3.183.65:6790,10.3.65.12:6790,10.3.131.61:6790:/ /var/lib/kubelet/pods/6ac879d0-d672-11e7-88d3-0050569d5b15/volumes/rook.io~rook/ceph-store ceph [name=admin secret=AQBNCiFa0oqjKhAA2WH+8VwHB0y17Irg6HCmIw== mds_namespace=rookfs]
    Output: mount: mount 10.3.183.65:6790,10.3.65.12:6790,10.3.131.61:6790:/ on /var/lib/kubelet/pods/6ac879d0-d672-11e7-88d3-0050569d5b15/volumes/rook.io~rook/ceph-store failed: Connection timed out
    
    wontfix filesystem 
    opened by beatlejuse 48
  • cephosd: skipping device

    cephosd: skipping device "sdc" because it contains a filesystem "LVM2_member"

    Expected behaviour: My SSD disk get added as Ceph OSD.

    Reality: The Disk gets not added as OSD. Confirmed by looking at ceph status executed in the toolbox pod. FYI: The node has three disks: an 110GB SSD for the OS and a 7.3TB HDD and a 240GB SSD.

     cluster:
        id:     7634c163-d325-4e7d-8d7a-6c95ec997bf9
        health: HEALTH_WARN
                Reduced data availability: 1 pg inactive
                Degraded data redundancy: 1 pg undersized
                OSD count 1 < osd_pool_default_size 3
    
      services:
        mon: 3 daemons, quorum a,b,c (age 2h)
        mgr: a(active, since 11m)
        osd: 1 osds: 1 up (since 2h), 1 in (since 3h)
    
      data:
        pools:   1 pools, 1 pgs
        objects: 0 objects, 0 B
        usage:   1.0 GiB used, 7.3 TiB / 7.3 TiB avail
        pgs:     100.000% pgs not active
                 1 undersized+peered
    

    I inspected the Preparation Pod logs, which show that cephosd: skipping device "sdc" because it contains a filesystem "LVM2_member". I tried everything humanly possible to wipe the disk:

    • wipefs -a /dev/sdc
    • dd if=/dev/zero of=/dev/sdc bs=1MB count=250000
    • vgremove <volumegroup>
    • pvremove <pv>
    • sgdisk --zap-all /dev/sdc
    • gdisk /dev/sdc -> x -> z (zap all)
    • blkdiscard /dev/sdc After cleaning the disk I delete the Rook operator to reexectue the preparing pod. The LVM2_member type, as well as a volume group ceph-0a98cb91-3041-4841-9112-52a53571580f seems to magically reappear after some time (I deleted them several times).

    Kubernetes 1.20.1 Rook 1.5.4

    [email protected]:~$ lsblk
    NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    loop0                                                                                                   7:0    0    55M  1 loop /snap/core18/1880
    loop1                                                                                                   7:1    0  71.3M  1 loop /snap/lxd/16099
    loop2                                                                                                   7:2    0  29.9M  1 loop /snap/snapd/8542
    sda                                                                                                     8:0    0 119.2G  0 disk
    ├─sda1                                                                                                  8:1    0     1M  0 part
    ├─sda2                                                                                                  8:2    0     1G  0 part /boot
    └─sda3                                                                                                  8:3    0 118.2G  0 part
      └─ubuntu--vg-ubuntu--lv                                                                             253:1    0  59.1G  0 lvm  /
    sdb                                                                                                     8:16   0   7.3T  0 disk
    └─ceph--f28f9786--d8c6--49bb--a200--21508f37d267-osd--block--a13dd6d4--5581--4805--80cf--0c8aefdefc14 253:0    0   7.3T  0 lvm
    sdc
    
    [email protected]:~$ sudo vgs
      VG                                        #PV #LV #SN Attr   VSize    VFree
      ceph-0a98cb91-3041-4841-9112-52a53571580f   1   0   0 wz--n- <223.57g <223.57g
      ceph-f28f9786-d8c6-49bb-a200-21508f37d267   1   1   0 wz--n-   <7.28t       0
      ubuntu-vg
    
    [email protected]:~$ sudo pvs
      PV         VG                                        Fmt  Attr PSize    PFree
      /dev/sda3  ubuntu-vg                                 lvm2 a--  <118.24g  <59.12g
      /dev/sdb   ceph-f28f9786-d8c6-49bb-a200-21508f37d267 lvm2 a--    <7.28t       0
      /dev/sdc   ceph-50c0f515-41f1-4950-9394-dc306380cf4d lvm2 a--  <223.57g <223.57g
    
    wipefs /dev/sdc
    DEVICE OFFSET TYPE        UUID                                   LABEL
    sdc    0x218  LVM2_member oLvaRi-NlMY-oCjS-b7P3-W7ec-Ompt-cVJ28p
    

    Rook prepare pod log:

    [email protected]:~$ kubectl logs -f rook-ceph-osd-prepare-node1-tnwmw -n rook-ceph
    2020-12-20 19:00:55.783480 I | rookcmd: starting Rook v1.5.4 with arguments '/rook/rook ceph osd provision'
    2020-12-20 19:00:55.783588 I | rookcmd: flag values: --cluster-id=cf7e2823-d4d3-45d6-9438-6da53a4aafb8, --data-device-filter=all, --data-device-path-filter=, --data-devices=, --drive-groups=, --encrypted-device=false, --force-format=false, --help=false, --location=, --log-flush-frequency=5s, --log-level=DEBUG, --metadata-device=, --node-name=node1, --operator-image=, --osd-crush-device-class=, --osd-database-size=0, --osd-store=, --osd-wal-size=576, --osds-per-device=1, --pvc-backed-osd=false, --service-account=
    2020-12-20 19:00:55.783597 I | op-mon: parsing mon endpoints: a=10.110.220.201:6789,b=10.106.131.121:6789,c=10.96.247.66:6789
    2020-12-20 19:00:55.804891 I | op-osd: CRUSH location=root=default host=node1
    2020-12-20 19:00:55.804916 I | cephcmd: crush location of osd: root=default host=node1
    2020-12-20 19:00:55.804932 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/sbin/lvm --help
    2020-12-20 19:00:55.821629 I | cephosd: successfully called nsenter
    2020-12-20 19:00:55.821656 I | cephosd: binary "/usr/sbin/lvm" found on the host, proceeding with osd preparation
    2020-12-20 19:00:55.821665 D | exec: Running command: dmsetup version
    2020-12-20 19:00:55.824803 I | cephosd: Library version:   1.02.171-RHEL8 (2020-05-28)
    Driver version:    4.41.0
    2020-12-20 19:00:55.836122 D | cephclient: No ceph configuration override to merge as "rook-config-override" configmap is empty
    2020-12-20 19:00:55.836151 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
    2020-12-20 19:00:55.836394 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
    2020-12-20 19:00:55.836590 D | cephosd: config file @ /etc/ceph/ceph.conf: [global]
    fsid                = 7634c163-d325-4e7d-8d7a-6c95ec997bf9
    mon initial members = a b c
    mon host            = [v2:10.110.220.201:3300,v1:10.110.220.201:6789],[v2:10.106.131.121:3300,v1:10.106.131.121:6789],[v2:10.96.247.66:3300,v1:10.96.247.66:6789]
    public addr         = 10.244.166.187
    cluster addr        = 10.244.166.187
    
    [client.admin]
    keyring = /var/lib/rook/rook-ceph/client.admin.keyring
    
    2020-12-20 19:00:55.836608 I | cephosd: discovering hardware
    2020-12-20 19:00:55.836618 D | exec: Running command: lsblk --all --noheadings --list --output KNAME
    2020-12-20 19:00:55.844287 D | exec: Running command: lsblk /dev/loop0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.848091 W | inventory: skipping device "loop0". unsupported diskType loop
    2020-12-20 19:00:55.848116 D | exec: Running command: lsblk /dev/loop1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.850947 W | inventory: skipping device "loop1". unsupported diskType loop
    2020-12-20 19:00:55.850972 D | exec: Running command: lsblk /dev/loop2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.854021 W | inventory: skipping device "loop2". unsupported diskType loop
    2020-12-20 19:00:55.854054 D | exec: Running command: lsblk /dev/loop3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.857230 W | inventory: skipping device "loop3". diskType is empty
    2020-12-20 19:00:55.857262 D | exec: Running command: lsblk /dev/loop4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.859973 W | inventory: skipping device "loop4". diskType is empty
    2020-12-20 19:00:55.860006 D | exec: Running command: lsblk /dev/loop5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.863078 W | inventory: skipping device "loop5". diskType is empty
    2020-12-20 19:00:55.863157 D | exec: Running command: lsblk /dev/loop6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.866309 W | inventory: skipping device "loop6". diskType is empty
    2020-12-20 19:00:55.866347 D | exec: Running command: lsblk /dev/loop7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.869371 W | inventory: skipping device "loop7". diskType is empty
    2020-12-20 19:00:55.869397 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.873638 D | exec: Running command: sgdisk --print /dev/sda
    2020-12-20 19:00:55.877870 D | exec: Running command: udevadm info --query=property /dev/sda
    2020-12-20 19:00:55.887319 D | exec: Running command: lsblk --noheadings --pairs /dev/sda
    2020-12-20 19:00:55.895556 I | inventory: skipping device "sda" because it has child, considering the child instead.
    2020-12-20 19:00:55.895589 D | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.898570 D | exec: Running command: udevadm info --query=property /dev/sda1
    2020-12-20 19:00:55.909910 D | exec: Running command: lsblk /dev/sda2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.914529 D | exec: Running command: udevadm info --query=property /dev/sda2
    2020-12-20 19:00:55.926259 D | exec: Running command: lsblk /dev/sda3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.929424 D | exec: Running command: udevadm info --query=property /dev/sda3
    2020-12-20 19:00:55.938345 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.941775 D | exec: Running command: sgdisk --print /dev/sdb
    2020-12-20 19:00:55.945435 D | exec: Running command: udevadm info --query=property /dev/sdb
    2020-12-20 19:00:55.954729 D | exec: Running command: lsblk --noheadings --pairs /dev/sdb
    2020-12-20 19:00:55.962366 I | inventory: skipping device "sdb" because it has child, considering the child instead.
    2020-12-20 19:00:55.962400 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.966457 D | exec: Running command: sgdisk --print /dev/sdc
    2020-12-20 19:00:55.973838 D | exec: Running command: udevadm info --query=property /dev/sdc
    2020-12-20 19:00:55.983319 D | exec: Running command: lsblk --noheadings --pairs /dev/sdc
    2020-12-20 19:00:55.990427 D | exec: Running command: lsblk /dev/nbd0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.993344 W | inventory: skipping device "nbd0". diskType is empty
    2020-12-20 19:00:55.993370 D | exec: Running command: lsblk /dev/nbd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.996287 W | inventory: skipping device "nbd1". diskType is empty
    2020-12-20 19:00:55.996345 D | exec: Running command: lsblk /dev/nbd2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:55.999270 W | inventory: skipping device "nbd2". diskType is empty
    2020-12-20 19:00:55.999299 D | exec: Running command: lsblk /dev/nbd3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.002494 W | inventory: skipping device "nbd3". diskType is empty
    2020-12-20 19:00:56.002525 D | exec: Running command: lsblk /dev/nbd4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.005374 W | inventory: skipping device "nbd4". diskType is empty
    2020-12-20 19:00:56.005410 D | exec: Running command: lsblk /dev/nbd5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.008277 W | inventory: skipping device "nbd5". diskType is empty
    2020-12-20 19:00:56.008317 D | exec: Running command: lsblk /dev/nbd6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.011072 W | inventory: skipping device "nbd6". diskType is empty
    2020-12-20 19:00:56.011108 D | exec: Running command: lsblk /dev/nbd7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.014083 W | inventory: skipping device "nbd7". diskType is empty
    2020-12-20 19:00:56.014129 D | exec: Running command: lsblk /dev/dm-0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.017294 D | exec: Running command: sgdisk --print /dev/dm-0
    2020-12-20 19:00:56.021137 D | exec: Running command: udevadm info --query=property /dev/dm-0
    2020-12-20 19:00:56.030273 D | exec: Running command: lsblk /dev/dm-1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.033153 D | exec: Running command: sgdisk --print /dev/dm-1
    2020-12-20 19:00:56.036816 D | exec: Running command: udevadm info --query=property /dev/dm-1
    2020-12-20 19:00:56.045932 D | exec: Running command: lsblk /dev/nbd8 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.048851 W | inventory: skipping device "nbd8". diskType is empty
    2020-12-20 19:00:56.048881 D | exec: Running command: lsblk /dev/nbd9 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.051550 W | inventory: skipping device "nbd9". diskType is empty
    2020-12-20 19:00:56.051577 D | exec: Running command: lsblk /dev/nbd10 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.054601 W | inventory: skipping device "nbd10". diskType is empty
    2020-12-20 19:00:56.054631 D | exec: Running command: lsblk /dev/nbd11 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.057400 W | inventory: skipping device "nbd11". diskType is empty
    2020-12-20 19:00:56.057430 D | exec: Running command: lsblk /dev/nbd12 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.060136 W | inventory: skipping device "nbd12". diskType is empty
    2020-12-20 19:00:56.060198 D | exec: Running command: lsblk /dev/nbd13 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.062961 W | inventory: skipping device "nbd13". diskType is empty
    2020-12-20 19:00:56.062995 D | exec: Running command: lsblk /dev/nbd14 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.065801 W | inventory: skipping device "nbd14". diskType is empty
    2020-12-20 19:00:56.065837 D | exec: Running command: lsblk /dev/nbd15 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.068738 W | inventory: skipping device "nbd15". diskType is empty
    2020-12-20 19:00:56.068775 D | inventory: discovered disks are [0xc0001fd680 0xc0001c8000 0xc0001c90e0 0xc0001e66c0 0xc0001fcd80 0xc0001fd200]
    2020-12-20 19:00:56.068788 I | cephosd: creating and starting the osds
    2020-12-20 19:00:56.076950 D | cephosd: No Drive Groups configured.
    2020-12-20 19:00:56.077002 D | cephosd: desiredDevices are [{Name:all OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: IsFilter:true IsDevicePathFilter:false}]
    2020-12-20 19:00:56.077018 D | cephosd: context.Devices are [0xc0001fd680 0xc0001c8000 0xc0001c90e0 0xc0001e66c0 0xc0001fcd80 0xc0001fd200]
    2020-12-20 19:00:56.077029 D | exec: Running command: udevadm info --query=property /dev/sda1
    2020-12-20 19:00:56.085755 D | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
    2020-12-20 19:00:56.088783 D | exec: Running command: ceph-volume inventory --format json /dev/sda1
    2020-12-20 19:00:56.962453 I | cephosd: skipping device "sda1": ["Insufficient space (<5GB)"].
    2020-12-20 19:00:56.962489 I | cephosd: skipping device "sda2" because it contains a filesystem "ext4"
    2020-12-20 19:00:56.962500 I | cephosd: skipping device "sda3" because it contains a filesystem "LVM2_member"
    2020-12-20 19:00:56.962508 I | cephosd: skipping device "sdc" because it contains a filesystem "LVM2_member"
    2020-12-20 19:00:56.962516 I | cephosd: skipping 'dm' device "dm-0"
    2020-12-20 19:00:56.962523 I | cephosd: skipping 'dm' device "dm-1"
    2020-12-20 19:00:56.962687 I | cephosd: configuring osd devices: {"Entries":{}}
    2020-12-20 19:00:56.962706 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
    2020-12-20 19:00:56.962973 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list  --format json
    2020-12-20 19:00:57.651269 D | cephosd: {
        "0": [
            {
                "devices": [
                    "/dev/sdb"
                ],
                "lv_name": "osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14",
                "lv_path": "/dev/ceph-f28f9786-d8c6-49bb-a200-21508f37d267/osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14",
                "lv_size": "8001561821184",
                "lv_tags": "ceph.block_device=/dev/ceph-f28f9786-d8c6-49bb-a200-21508f37d267/osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14,ceph.block_uuid=szSYsx-AbEA-jNha-ddiK-U6PX-M3Ll-jMHAUa,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=7634c163-d325-4e7d-8d7a-6c95ec997bf9,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=a13dd6d4-5581-4805-80cf-0c8aefdefc14,ceph.osd_id=0,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0",
                "lv_uuid": "szSYsx-AbEA-jNha-ddiK-U6PX-M3Ll-jMHAUa",
                "name": "osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14",
                "path": "/dev/ceph-f28f9786-d8c6-49bb-a200-21508f37d267/osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14",
                "tags": {
                    "ceph.block_device": "/dev/ceph-f28f9786-d8c6-49bb-a200-21508f37d267/osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14",
                    "ceph.block_uuid": "szSYsx-AbEA-jNha-ddiK-U6PX-M3Ll-jMHAUa",
                    "ceph.cephx_lockbox_secret": "",
                    "ceph.cluster_fsid": "7634c163-d325-4e7d-8d7a-6c95ec997bf9",
                    "ceph.cluster_name": "ceph",
                    "ceph.crush_device_class": "None",
                    "ceph.encrypted": "0",
                    "ceph.osd_fsid": "a13dd6d4-5581-4805-80cf-0c8aefdefc14",
                    "ceph.osd_id": "0",
                    "ceph.osdspec_affinity": "",
                    "ceph.type": "block",
                    "ceph.vdo": "0"
                },
                "type": "block",
                "vg_name": "ceph-f28f9786-d8c6-49bb-a200-21508f37d267"
            }
        ]
    }
    2020-12-20 19:00:57.651477 I | cephosd: osdInfo has 1 elements. [{Name:osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14 Path:/dev/ceph-f28f9786-d8c6-49bb-a200-21508f37d267/osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14 Tags:{OSDFSID:a13dd6d4-5581-4805-80cf-0c8aefdefc14 Encrypted:0 ClusterFSID:7634c163-d325-4e7d-8d7a-6c95ec997bf9} Type:block}]
    2020-12-20 19:00:57.651493 I | cephosd: 1 ceph-volume lvm osd devices configured on this node
    2020-12-20 19:00:57.651528 I | cephosd: devices = [{ID:0 Cluster:ceph UUID:a13dd6d4-5581-4805-80cf-0c8aefdefc14 DevicePartUUID: BlockPath:/dev/ceph-f28f9786-d8c6-49bb-a200-21508f37d267/osd-block-a13dd6d4-5581-4805-80cf-0c8aefdefc14 MetadataPath: WalPath: SkipLVRelease:false Location: LVBackedPV:false CVMode:lvm Store:bluestore}]
    
    bug wontfix 
    opened by Knowledge91 45
  • build: Update go modules to build with go 1.18 and support go 1.19

    build: Update go modules to build with go 1.18 and support go 1.19

    Description of your changes:

    • Update from make go.mod.update to ensure we are consuming the latest stable packages available.
    • Support both go 1.18 and 1.19 and drop 1.17
    • Update to gofmt required for 1.19

    Checklist:

    • [ ] Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
    • [ ] Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
    • [ ] Reviewed the developer guide on Submitting a Pull Request
    • [ ] Pending release notes updated with breaking and/or notable changes for the next minor release.
    • [ ] Documentation has been updated, if necessary.
    • [ ] Unit tests have been added, if necessary.
    • [ ] Integration tests have been added, if necessary.
    opened by travisn 0
  • external: fix cluster id naming in import-external-cluster script (backport #10677)

    external: fix cluster id naming in import-external-cluster script (backport #10677)

    This is an automatic backport of pull request #10677 done by Mergify.


    Mergify commands and options

    More conditions and actions can be found in the documentation.

    You can also trigger Mergify actions by commenting on this pull request:

    • @Mergifyio refresh will re-evaluate the rules
    • @Mergifyio rebase will rebase this PR on its base branch
    • @Mergifyio update will merge the base branch into this PR
    • @Mergifyio backport <destination> will backport this PR on <destination> branch

    Additionally, on Mergify dashboard you can:

    • look at your merge queues
    • generate the Mergify configuration with the config editor.

    Finally, you can contact us on https://mergify.com

    opened by mergify[bot] 0
  • Crash in lib-bucket-provisioner when updating object bucket claims (OBCs)

    Crash in lib-bucket-provisioner when updating object bucket claims (OBCs)

    Is this a bug report or feature request?

    • Bug Report Track lib-bucket-provisioner bug https://github.com/kube-object-storage/lib-bucket-provisioner/issues/219
    bug beginner object bucket claims (OBC) 
    opened by BlaineEXE 0
  • Investigate If we can support --restricted flag for upgraded external cluster

    Investigate If we can support --restricted flag for upgraded external cluster

    Is this a bug report or feature request?

    • Feature Request Investigate If we can support --restricted flag for upgraded external cluster #10678

    We would be needing to update StorageClass with new secretes,

    Also ODF operator need to change this

    What should the feature do:

    What is use case behind this feature:

    Environment:

    feature 
    opened by parth-gr 2
  • nfs: add support for running SSSD as a sidecar

    nfs: add support for running SSSD as a sidecar

    Description of your changes: Implement the SSSD sidecar design for NFS.

    Which issue is resolved by this Pull Request: Resolves #

    Checklist:

    • [ ] Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
    • [ ] Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
    • [ ] Reviewed the developer guide on Submitting a Pull Request
    • [ ] Pending release notes updated with breaking and/or notable changes for the next minor release.
    • [ ] Documentation has been updated, if necessary.
    • [ ] Unit tests have been added, if necessary.
    • [ ] Integration tests have been added, if necessary.
    opened by BlaineEXE 0
  • docs: SSSD-NFS design

    docs: SSSD-NFS design

    Description of your changes:

    Which issue is resolved by this Pull Request: Resolves #

    Checklist:

    • [ ] Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
    • [ ] Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
    • [ ] Reviewed the developer guide on Submitting a Pull Request
    • [ ] Pending release notes updated with breaking and/or notable changes for the next minor release.
    • [ ] Documentation has been updated, if necessary.
    • [ ] Unit tests have been added, if necessary.
    • [ ] Integration tests have been added, if necessary.
    skip-ci 
    opened by BlaineEXE 1
Releases(v1.9.8)
  • v1.9.8(Jul 26, 2022)

    Improvements

    Rook v1.9.8 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • osd: Support OSDs on logical volumes in host-based clusters (#7967, @satoru-takeuchi)
    • Report rbd pool stats properly (#10575, @shalevpenker97)
    • helm: Add labels and annotations to the Prometheus rule (#10583, @sathieu)
    • build: Set default Ceph version to v16.2.10 (#10632, @travisn)
    • docs: Clarify PG count actions and update pgcalc URL (#10610, @anthonyeleven)
    • core: Fix stuck reconciler with more checkpoints for cancelled context (#10523, @leseb)
    • docs: Add telemetry section to quickstart guide (#10602, @travisn)
    • core: Fix numerous spelling errors (#10484, @jsoref)
    • object: Network mode can be set separately for cephcluster and rgw (#10491, @zhucan)
    • osd: Add udev path support for a metadata device (#10494, @microyahoo)
    • csi: Set leader election timeouts for the csi driver (#10570, @subhamkrai)
    • mgr: Ensure service selector is initialized for services with label app=rook-ceph-mgr (#10577, @travisn)
    • helm: Add missing port for webhook in the helm chart (#10569, @subhamkrai)
    • external: Correctly name variables in import-external-cluster.sh script (#10590, @der-ali)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.7(Jul 7, 2022)

    Improvements

    Rook v1.9.7 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • helm: Use recommended value for helm managed-by label (#10558, @galexrt)
    • manifest: Use Recreate strategy for operator upgrade (#10547, @sp98)
    • external: Add support for subvolumegroup and rados namespace (#10459, @parth-gr)
    • docs: Refactor cluster crd doc for subtopics (#10506, @travisn)
    • helm: use correct namespace for provisioner in cephobjectstore template (#10154, @bitfisher)
    • mon: Disable insecure global ids immediately for new deployments (#10505, @travisn)
    • csi: Update staging path to support K8s 1.24+ (#10490, @pkalever)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.6(Jun 23, 2022)

    Improvements

    Rook v1.9.6 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • core: Delete v1beta1 cronjob only when v1 cronjob not found (#10480, @subhamkrai)
    • helm: Increase memory of osd prepare pods (#10496, @parth-gr)
    • file: Finalizer blocks deletion on existence of subvolume groups (#9915, @BlaineEXE)
    • core: Increase liveness probe timeout to 2s (#10460, @subhamkrai)
    • csi: Add RBAC to create token for RBD CSI ClusterRole (#10442, @Rakshith-R)
    • doc: Update ceph external cluster flags (#10234, @parth-gr)
    • osd: Add lvm flag for activation of legacy lvm OSDs (#10445, @leseb)
    • ceph: update CSI-Addons side-car to latest (v0.4.0) release (#10454, @nixpanic)
    • csi: Update storage class for csi-users in external cluster (#10278, @parth-gr)
    • mon: Update node labels as per new key value (#10405, @humblec)
    • test: Add tests for NFS CSI pvc (#10354, @Rakshith-R)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.5(Jun 9, 2022)

    Improvements

    Rook v1.9.5 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • csi: Update Ceph CSI from 3.6.1 to 3.6.2 (#10424, @Madhu-1)
    • osd: Fix encrypted OSDs on latest Ceph releases and add test for encrypted osd (#10333, @satoru-takeuchi)
    • core: Report basic cluster and version for Ceph v17.2.1 or newer if telemetry is enabled (#10341, @travisn)
    • csi: Update sidecar to latest release (#10306, @Madhu-1)
    • ceph: Enable holder pod if csi host networking is disabled (#10342, @Madhu-1)
    • osd: Ignore the region label when it is the same as zone label in topology (#10383, @xu-cheng)
    • csi: Change sidecar images repo to registry.k8s.io (#10116, @Madhu-1)
    • osd: Improved errors for encrypted OSD provisioning failures (#10373, @leseb)
    • csi: Do not block the reconcile if all clusters are not ready (#10380, @leseb)
    • rgw: Convert FQDN RGW to an IP in external script (#10309, @parth-gr)
    • pool: Delete undlerying ceph pool when blockpool CR is deleted (#10362, @travisn)
    • docs: Fix broken doc links since the docs redesign (#10349, @subhamkrai)
    • docs: Updated links to CRD objects in helm chart (#10348, @denis256)
    • csi: Add --extra-create-metadata arg to csi sidecars (#10346, @pkalever)
    • external: Format python files using tool black (#10422, @subhamkrai)
    • external-script: Remove signature import (#10421, @subhamkrai)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.10(Jun 9, 2022)

    Improvements

    Rook v1.8.10 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • core: Improve detection of filesystem properties for disk in use (#10230, @leseb)
    • osd: Remove broken argument for upgraded OSDs on PVCs in legacy lvm mode (#10298, @leseb)
    • osd: Allow the osd to take two hours to start in case of ceph maintenance (#10250, @travisn)
    • operator: Report telemetry 'rook/version' in mon store (#10161, @BlaineEXE)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.4(May 26, 2022)

    Improvements

    Rook v1.9.4 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • build: Update examples and base operator image to use ceph v16.2.9 (#10275, @parth-gr)
    • test: Add canary integration test for osd with metadata device (#10286, @satoru-takeuchi)
    • csi: Update CephFS OSD caps for external clusters (#10325, @parth-gr)
    • docs: New doc framework with mkdocs and related build changes (#10321, @galexrt @travisn)
    • docs: Add ceph telemetry design doc (#10254, @BlaineEXE)
    • osd: Remove broken argument for upgraded lvm-based OSDs on PVCs (#10298, @leseb)
    • operator: Rename telemetry rook-version to rook/version (#10279, @BlaineEXE)
    • helm: Only copy relevant monitoring settings (#10274, @travisn)
    • rgw: Validate RGW endpoint IP for external clusters (#10006, @parth-gr)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.3(May 12, 2022)

    Improvements

    Rook v1.9.3 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • osd: Add additional checks so OSDs are not created on existing filesystems (#10230, @leseb)
    • osd: Allow the osd to take a long time to start (#10250, @travisn)
    • osd: Exit immediately if pvc block has not changed (#10246, @leseb)
    • osd: Close the encrypted disk after cleanup is done (#10191, @leseb)
    • operator: Increase memory limit of operator (#10195 #10199, @travisn)
    • operator: Set rook-version in mon k/v store (#10161, @BlaineEXE)
    • csi: Add the option to disable hostNetwork in the helm chart (#10253, @yuvalman)
    • operator: Update tolerations name from controlplane to control-plane (#10242, @ttyS0)
    • mon: quorumStatus.MonMap.Mons changed to quorumStatus.Quorum (#10172, @fengjiankui121)
    • build: Add psp for missing roles (#10244, @y1r)
    • webhook: Update the check-in cluster spec validation (#10226, @subhamkrai)
    • osd: Fix disk uuid management (#10212, @satoru-takeuchi)
    • test: Add canary integration tests for osd on device (#9931, @satoru-takeuchi)
    • rgw: Support FQDN for RGW endpoint (#10232, @parth-gr)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.2(Apr 28, 2022)

    Improvements

    Rook v1.9.2 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • core: Enable multus network configuration with the csi driver (#9925, @leseb)
    • build: Update minimum supported version to k8s 1.17 (#10123, @travisn)
    • csi: Update ceph-csi to v3.6.1 release (#10150, @leseb)
    • rgw: Fix dashboard admin creation for multiple object stores (#10137, @galexrt)
    • helm: Add ingress annotations for ssl (#9936, @slavonicsniper)
    • build: Update go-ceph version to v0.15 and remove ceph_preview (#10125, @subhamkrai)
    • ci: use stringData instead of data for secret (#10156, @leseb)
    • core: RGW realm pull spec is optional (#10139, @travisn)
    • build: Add support for go 1.18 (#10127, @subhamkrai)
    • csi: Refactor csi initialization code for setting parameters (#10124, @Madhu-1)
    • helm: ingressClassName was not available for networking.k8s.io/v1 (#10132, @casperbiering)
    • csi: Add/remove mon IP from csi config (#10135, @Madhu-1)
    • build: Add a placeholder rule to the RGW RBAC (#10142, @travisn)
    • build: Add placeholder NFS RBAC for csv generation (#10183, @Rakshith-R)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.1(Apr 21, 2022)

    Improvements

    Rook v1.9.1 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

    • monitoring: Disable new alerts that are not applicable to rook (#10109, @travisn)
    • core: Examples set default priority class names (#10096, @travisn)
    • nfs: use k8s.gcr.io for csi nfsplugin image (#10059, @Madhu-1)
    • csi: Add CSI encryption support (#9940, @Rakshith-R)
    • csi: Make GRPC timeout configurable (#10074, @Madhu-1)
    • helm: Add missing default resources in rook components (#10089, @yuvalman)
    • osd: Enable Quincy osd release version (#10088, @travisn)
    • core: Update controller-runtime version to v0.11.2 (#10023, @subhamkrai)
    • block: Validate name updates to the BlockPool CR (#10099, @Madhu-1)
    • helm: Add ingressClassName field (#10093, @log1cb0mb)
    • csi: Enable csi nfs to generate nfs rbac for csv (#10071, @Rakshith-R)
    • docs: Update NFS docs including latest CSI work (#10014, @BlaineEXE)
    • build: Generate a separate NFS CSI RBAC manifest (#10058, @BlaineEXE)
    • docs: Add info about mounting NFS exports externally (#10078, @BlaineEXE)
    • csi: Add nfs nodeplugin & provisioner (#9927, @Rakshith-R)
    • build: update golang version to v1.17 (#10094, @subhamkrai)
    • build: Add a placeholder rgw role for csv generation (#10108, @travisn)
    • subvolumegroup: Refactor subvolumegroup controller (#10111, @Madhu-1)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.9(Apr 21, 2022)

    Improvements

    Rook v1.8.9 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • helm: Add ingressClassName field (#10093, @log1cb0mb)
    • monitoring: Only set prometheus rules ownerref in same namespace (#10028, @travisn)
    • osd: only set kek to env var on encryption scenario (#10035, @leseb)
    • docs: Update the s3 client example for accessing RGW (#9968, @thotz)
    • osd: Add NixOS specific PATHs to check for lvm2 (#9967, @nazarewk)
    Source code(tar.gz)
    Source code(zip)
  • v1.9.0(Apr 13, 2022)

    Upgrade Guide

    To upgrade from previous versions of Rook, see the Ceph upgrade guide.

    Breaking Changes

    • The MDS liveness and startup probes are now configured by the CephFilesystem CR instead of the CephCluster CR. To apply the MDS probes, they need to be specified in the CephFilesystem CR. See the CephFilesystem doc for more details.
    • In the Helm charts, all Ceph components now have default values for the pod resources. The values can be modified or removed in values.yaml depending on cluster requirements.
    • Prometheus rules are installed by the cluster Helm chart. If you were relying on the CephCluster CR setting monitoring.enabled to create the prometheus rules, they now need to be enabled by setting monitoring.createPrometheusRules in the cluster Helm chart values.
    • Removed the obsolete cross build container, now unused by the CI

    Features

    • The number of mgr daemons for example clusters is increased to 2, resulting in a standby mgr daemon. If the active mgr goes down, Ceph will update the passive mgr to be active, and rook will update all the services with the label app=rook-ceph-mgr to direct traffic to the new active mgr.
    • Network encryption is configurable with settings in the CephCluster CR. Requires the 5.11 kernel or newer.
    • Network compression is configurable with settings in the CephCluster CR. Requires Ceph Quincy (v17) and a newer kernel as also required for encryption.
    • Support for custom ceph.conf for csi pods.
    • Ceph prometheus rules added and updated as recommended from the main Ceph project. The rules can be created with the cluster helm chart by setting the monitoring.createPrometheusRules, or see the monitoring documentation if not using helm.
    • The service account rook-ceph-rgw is added to the RGW pods.
    • A CephBlockPoolRadosNamespace CRD is provided to create RADOS namespaces in a pool.
    Source code(tar.gz)
    Source code(zip)
  • v1.8.8(Mar 31, 2022)

    Improvements

    Rook v1.8.8 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • core: Cluster CR status was not being refreshed after updating the cluster CR (#9962, @leseb)
    • core: GetLogCollectorResources to get the right resources (#9898, @yuvalman)
    • object: Remove unnecessary region option from the OBC StorageClass (#9906, @thotz)
    • core: Add Phase in additionalPrinterColumns for all CRs (#9910, @subhamkrai)
    • test: Avoid potential data inconsistency on zapping disk (#9930, @satoru-takeuchi)
    • ci: Add pylint in ci (#9879, @subhamkrai)
    • core: Incorrect join command in external cluster script (#9862, @vavuthu)
    • core: Rework usage of ReportReconcileResult (#9873, @BlaineEXE)
    • csi: Populate mon endpoints even if csi driver not enabled (#9878, @travisn)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.7(Mar 10, 2022)

    Improvements

    Rook v1.8.7 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • build: Update ceph base image to v16.2.7-20220216 (#9814, @travisn)
    • csi: default to ReadWriteOnceWithFSType for cephfs (#9729, @humblec)
    • mon: Disable startup probe on canary pods (#9888, @travisn)
    • core: Add Ceph FSID on the cephcluster CR status (#9847, @parth-gr)
    • csi: Properly apply CSI resource requests and limits (#9868, @TomHellier)
    • helm: Add resource requests and limits to the toolbox pod (#9856, @TomHellier)
    • helm: Remove obsolete .Values.image.prefix (#9863, @kahirokunn)
    • osd: Clarify vault auth error message (#9884, @leseb)
    • nfs: Remove secret and configmap when downscaling NFS daemons (#9859, @BlaineEXE)
    • helm: Handle empty StorageClass parameters for object, rbd, and cephfs in the helm chart (#9854, @Zempashi)
    • helm: Remove obsolete setting for enabling multiple filesystems (#9841, @travisn)
    • osd: Use lvm mode to create multiple OSDs per device (#9842, @BlaineEXE)
    • helm: Add filesystem pool name to the storage class (#9838, @mtt0)
    • docs: Document that the rook-ceph-operator-config ConfigMap is required (#9821, @matthiasr)
    • core: Suppress verbose disruption controller log messages (#9834, @travisn)
    • osd: Purge job will remove all pvcs for the osd, not just the data pvc (#9804, @travisn)
    • osd: Remove osd with purge instead of destroy (#9807, @travisn)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.6(Feb 24, 2022)

    Improvements

    Rook v1.8.6 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • osd: Enable raw mode for OSDs for Ceph v16.2.7 and newer (#9701, @satoru-takeuchi)
    • file: Allow filesystem removal even with failures (#9799, @travisn)
    • helm: Allow setting priority class on operator and toolbox (#9790, @travisn)
    • docs: Warn in quickstart guide about using a vm (#9777, @travisn)
    • helm: Fix the enabled flag for csiAddons (#9645, @peng225)
    • osd: Return error if fail to list osds in prepare job (#9746, @travisn)
    • core: various external mode enhancements (#9760, @leseb)
    • ceph: remove rbd_default_features setting (#9758, @Madhu-1)
    • docs: clarify teardown disk zap section (#9765, @BlaineEXE)
    • docs: Add multus known limitations (#9511, @leseb)
    • csi: Modify upgrade flag in external cluster (#9609, @parth-gr)
    • object: Fix backend path for transit engine for rgw kms (#9728, @thotz)
    • osd: Write log message when node is being skipped (#9718, @travisn)
    • csi: Cleanup csi driver resources when zero cephclusters exist (#9713, @Rakshith-R)
    • object: do not check for upgrade on external mode (#9724, @leseb)

    Also try out the new v0.1.0 release of the Rook Krew Plugin!

    Source code(tar.gz)
    Source code(zip)
  • v1.8.5(Feb 10, 2022)

    Improvements

    Rook v1.8.5 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • core: Operator was crashing when the mon endpoints were updated (#9715, @leseb)
    • csi: Bump registar image to v2.5.0 (#9702, @sathieu)
    • osd: Add namespace in vault client (#9720, @leseb)
    • doc: Add documentation for deploying the csi-addons (#9624, @yati1998)
    • object: Update object pool properties during reconcile (#9696, @travisn)
    • csi: Create separate csi users in external cluster configuration (#9703, @parth-gr)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.4(Feb 3, 2022)

    Improvements

    Rook v1.8.4 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • csi: Skip validation when installing snapshotter (#9672, @Madhu-1)
    • subvolumegroup: Handle external mode (#9694, @leseb)
    • rgw: Gateway deployment strategy RollingUpdate on Pacific (#9578, @micalgenus)
    • core: Dereference pointer before trying to compare with deepequal (#9663, @leseb)
    • osd: Merge all osd resources including custom (#9658, @travisn)
    • csi: bump csi snapshotter image to v5 (#9665, @Madhu-1)
    • helm: add annotations and labels for VolumeSnapshotClasses (#9661, @Madhu-1)
    • csi: Bump csi resizer and volume replication images (#9651, @sathieu)
    • osd: Allow injecting extra OSD environment variables via ConfigMap (#9647, @rzarzynski)
    • pool: Check for pool application already being set (#9649, @travisn)
    • helm: Update CI to the latest helm and fix the linter (#9642, @travisn)
    • rgw: Inject tls certs for bucket notification and topic operations (#9565, @thotz)
    • rgw: Handle the non-existence of the bucket correctly when removing bucket (#9603, @cupnes)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.3(Jan 21, 2022)

    Improvements

    Rook v1.8.3 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • helm: Apply operator settings in configmap instead of deployment (#9074, @travisn)
    • csi: Update cephcsi to 3.5.1 release (#9627, @Madhu-1)
    • csi: Update to the ceph csi release 3.5 image and sidecars (#9581, @humblec)
    • csi: Change rook-ceph-csi-config to expose clusterID for subvolume (#9484, @leseb)
    • core: Reconcile csi driver even if the operator configmap does not exist (#9613, @leseb)
    • core: Support priority class for the crashcollector daemon (#9502, @satoru-takeuchi)
    • csi: Add support to create a VolumeSnapshotClass (#9586, @Madhu-1)
    • mon: Support annotations for critical metadata secrets and configmaps for backup (#9547, @subhamkrai)
    • csi: Add osd blocklist capabilities to the external cephcluster (#9571, @parth-gr)
    • csi: Add support for the csi-addons sidecar, though disabled by default (#9534, @yati1998)
    • build: Set LC_ALL on the make target (#9594, @leseb)
    • osd: Correct bluestore compression min blob size for ssd (#9582, @subhamkrai)
    • osd: Add KMS support for IBM KeyProtect (#9545, @leseb)
    • mgr: Update services with the label app=rook-ceph-mgr when the active Ceph mgr changes (#9467, @travisn)
    • helm: Use operatorNamespace if exists for monitoring RBAC (#9554, @henryzhangsta)
    • osd: Use cluster name when fetching the cephcluster (#9560, @leseb)
    • security: Apdate auth permissions for external cluster (#9411, @parth-gr)
    • core: Limit use of wildcards in cluster-scoped rbac (#9485, @BlaineEXE)
    Source code(tar.gz)
    Source code(zip)
  • v1.7.11(Jan 21, 2022)

    Improvements

    Rook v1.7.11 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • mgr: Update services with the app=rook-ceph-mgr label when the active Ceph mgr changes (#9467, @travisn)
    • osd: Correct bluestore compression min blob size for ssd (#9582, @subhamkrai)
    • build: Update to go v1.16.12 (#9478, @BlaineEXE)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.2(Jan 6, 2022)

    Improvements

    Rook v1.8.2 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • subvolumegroup: Add CephFilesystemSubVolumeGroup CRD (#9384, @leseb)
    • core: Run the discover daemonset as root uid (#9512, @leseb)
    • core: Allow specifying daemon startup probes (#9468, @BlaineEXE)
    • rgw: Startup probe uses same as the readiness probe instead of liveness probe (#9533, @satoru-takeuchi)
    • rgw: Add events to the CephBucketNotification CR (#9400, @yuvalif)
    • object: Update to the latest lib-bucket-provisioner (#9529, @BlaineEXE)
    • build: Use the same csv name for all platforms (#9515, @leseb)
    • object: Fix handling for notifications for OBC (#9365, @thotz)
    • osd: Handle removal of encrypted osd deployment (#9434, @leseb)
    • core: Re-enable check to disallow for multiple clusters in the same namespace (#9457, @leseb)
    • operator: Report events more consistently for cluster status (#9427, @BlaineEXE)
    • rgw: Add permissions to update finalizers on OBCs (#9441, @crd1985)
    • docs: Recommend primary affinity over hybrid pools (#9465, @travisn)
    • docs: Add jumbo frames note to common issues (#9453, @BlaineEXE)
    • security: Add dry run mode for external cluster script (#9189, @subhamkrai)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.1(Dec 16, 2021)

    Improvements

    Rook v1.8.1 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • file: Allow CephFS data pools to be named. NOTE For existing filesystems, do not change the names of existing pools or new pools will also be added to the filesystem (#9296, @degorenko)
    • pool: Allow configuration of the built-in pools: device_health_metrics and .nfs (#9363, @travisn)
    • osd: Add CAP_MKNOD privilege to the PSP for OSDs daemons (#9439, @y1r)
    • mds: Skip sanity check during upgrades to 16.2.7 to avoid mon crash (#9418, @travisn)
    • mgr: Start the watch-active container with correct privs when multiple mgr daemons are running (#9386, @leseb)
    • helm: Add optional monitoring RBAC to operator chart (#9388, @BlaineEXE)
    • pool: Add rbd ec pool support in external cluster (#9276, @subhamkrai)
    • build: Remove tini command from the direct mount toolbox (#9423, @parth-gr)
    • pool: Clean up health checkers for filesystem and object store forced deletion (#9417, @BlaineEXE)
    • security: Use correct osd tags for restricted caps with external cluster script (#9410, @CO-lhagman)
    • core: Add context parameter to various k8sutil helpers (#8700, @y1r)
    • osd: Run the cleanup job using the root UID (#9380, @leseb)

    Full Changelog: https://github.com/rook/rook/compare/v1.8.0...v1.8.1

    Source code(tar.gz)
    Source code(zip)
  • v1.7.10(Dec 16, 2021)

    Improvements

    Rook v1.7.10 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • mds: Skip MDS sanity check during upgrades to Ceph 16.2.7 to avoid mon crash (#9418, @travisn)
    • pool: Clean up health checkers during filesystem and object store deletion (#9094, @BlaineEXE)
    • core: On failure initializing the cluster, return the error instead of continuing (#9347, @leseb)
    • osd: Truncate name of the osd prepare job to avoid pod generation failure on K8s 1.22 (#9312, @travisn)
    • mon: Set stretch tiebreaker reliably during mon failover (#9282, @travisn)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.0(Dec 9, 2021)

    K8s Version Support

    K8s 1.16 or newer

    The minimum supported version of Kubernetes is now 1.16. If running on an older version it is recommended to update to a newer version before updating to Rook v1.8.

    Upgrade Guide

    To upgrade from previous versions of Rook, see the Ceph upgrade guide.

    Breaking Changes

    • Support for Ceph Nautilus is removed. Clusters with Nautilus must upgrade to Octopus or Pacific before upgrading to v1.8.
    • The Rook Flex driver is fully deprecated. If you are still using flex volumes, it is required to convert them to csi volumes before upgrading to v1.8. To convert your flex volumes, see the flex conversion tool.
    • The directory structure of the example manifests has changed. The manifests have moved to deploy/examples. (The directory in previous releases was cluster/examples/kubernetes/ceph.)

    Features

    • Support added for Kubernetes Authentication when using HashiCorp Vault Key Management Service.
    • The failure domain of a pool can be applied on the CephBlockPool instead of requiring manual commands run via the toolbox.
    • The Rook Operator and the toolbox run under a rook user rather than root.
    • The Rook image includes the s5cmd binary to simplify testing with S3 gateways from the toolbox.
    • Rook adds a finalizer to a secret and configmap critical to the Ceph cluster so that the resources will not be accidentally deleted.
    • The Rook Operator does not use "tini" as an init process. Instead, the rook binary handles signals on its own.
    • Common K8s labels are added to the Rook and Ceph pods.
    • The helm chart is the source of truth for generating crds.yaml, common.yaml, and the csv for integration with operatorhub.

    Experimental Features

    Source code(tar.gz)
    Source code(zip)
  • v1.7.9(Dec 6, 2021)

    Improvements

    Rook v1.7.9 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    This release includes the new migration tool from flex to CSI. The conversion is required before clusters can be upgraded to v1.8 where flex volumes will not be supported.

    • docs: Guide for migrating from flex to csi volumes (#9222, @subhamkrai)
    • build: Add pv-migrator tool to rook image in 1.7.x branch (#9274, @subhamkrai)
    • helm: Addition of mountOptions into storage class configuration (#9287, @TomHellier)
    • osd: Update existing OSDs with deviceClass (#9259, @parth-gr)
    • cephfs-mirror: Various fixes for random bootstrap peer import errors (#9264, @leseb)
    • helm: Allow configurability of ingress version (#9237, @TomHellier)
    • osd: Check if osd is safe-to-destroy before removal from the osd purge job (#9230, @leseb)
    • object: Configure object store with correct zone name for multisite (#9249, @olivierbouffet)
    • monitoring: Update label on prometheus resources (#9243, @LittleFox94)
    • core: Add expected capabilities to OpenShift security context (#9236, @leseb)
    • nfs: Only set the pool size when it exists and always run default pool creation for NFS (#9224, @leseb)
    • monitoring: Allow overriding monitoring labels (#9240, @LittleFox94)
    • helm: Use a more explicit value for ingress for K8s 1.18 (#9280, @TomHellier)
    • object: Create system user with correct realm for external cluster (#9143, @olivierbouffet)
    • docs: Add details about HPA via KEDA (#9202, @thotz)
    Source code(tar.gz)
    Source code(zip)
  • v1.8.0-beta.0(Dec 1, 2021)

  • v1.7.8(Nov 19, 2021)

    Improvements

    Rook v1.7.8 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • docs: add OMAP quick fix warning to the upgrade guide (#9187, @BlaineEXE)
    • mon: Set cluster name to mon cluster (#9203, @parth-gr)
    • osd: Add privileged support for OpenShift to blkdevmapper securityContext (#9191, @Omar007)
    • osd: Set reduced blkdevmapper capabilities (#9158, @Omar007)
    • helm: Set correct ingress endpoint protocol for https and http (#9181, @travisn)
    • rgw: Allow downgrade of rgw daemon (#9137, @travisn)
    • helm: Enable cephfs volume expansion by default (#9128, @travisn)
    • core: Fail if config dir creation fails (#9122, @leseb)
    • osd: Increase wait timeout for osd prepare cleanup (#9116, @travisn)
    • pool: Allow more data chunks in an EC pool (#9107, @travisn)
    Source code(tar.gz)
    Source code(zip)
  • v1.6.11(Nov 19, 2021)

    Improvements

    Rook v1.6.11 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • rgw: Allow reconcile to complete even during a downgrade (#9137, @travisn)
    • docs: Add OMAP quick fix warning to the upgrade guide (#9187, @BlaineEXE)
    • multus: Do not build all the args to remote exec cmd (#8860, @leseb)
    • multus: do not fail on keys deletion (#8868, @leseb)
    Source code(tar.gz)
    Source code(zip)
  • v1.7.7(Nov 4, 2021)

    Improvements

    Rook v1.7.7 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • docs: Support ephemeral volumes with Ceph CSI RBD and CephFS driver (#9055, @humblec)
    • core: Allow downgrade of all daemons consistently (#9098, @travisn)
    • core: Reconcile once instead of multiple times after the cluster CR is edited (#9091, @leseb)
    • nfs: Add pool setting CR option (#9040, @leseb)
    • ceph: Trigger 'CephMonQuorumLost' alert when mon quorum is down (#9068, @aruniiird)
    • rgw: Updated livenessProbe and readinessProbe (#9080, @satoru-takeuchi)
    • mgr: Do not set the balancer mode on pacific (#9063, @leseb)
    • helm: Add appVersion property to the charts (#9051, @travisn)
    • rgw: Read tls secret hint for insecure tls (#9020, @leseb)
    • ceph: Ability to set labels on the crash collector (#9044, @leseb)
    • core: Treat cluster as not existing if the cleanup policy is set (#9041, @travisn)
    • docs: Document failover and failback scenarios for applications (#8411, @Yuggupta27)
    • ceph: Update endpoint with IP for external RGW server (#9010, @thotz)

    Full Changelog: https://github.com/rook/rook/compare/v1.7.6...v1.7.7

    Source code(tar.gz)
    Source code(zip)
  • v1.7.6(Oct 20, 2021)

    Improvements

    Rook v1.7.6 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • core: only merge stderr on error (#8995, @leseb)
    • nfs: remove RADOS options from CephNFS and use .nfs pool (#8501, @josephsawaya)
    • csi: fix comment for the provisioner and clusterID (#8990, @Madhu-1)
    • mon: Enable mon failover for the arbiter in stretch mode (#8984, @travisn)
    • monitoring: fixing the queries for alerts 'CephMgrIsAbsent' and 'CephMgrIsMissingReplicas' (#8985, @aruniiird)
    • osd: fix kms auto-detection when full TLS (#8867, @leseb)
    • csi: add affinity to csi version check job (#8965, @Rakshith-R)
    • pool: remove default value for pool compression (#8966, @leseb)
    • monitoring: handle empty ceph_version in ceph_mon_metadata to avoid raising misleading alert (#8947, @GowthamShanmugam)
    • nfs: remove RADOS options from CephNFS and use .nfs pool (#8501, @josephsawaya)
    • osd: print the c-v output when inventory command fails (#8971, @leseb)
    • helm: remove chart content not in common.yaml (#8884, @BlaineEXE)
    • rgw: replace period update --commit with function (#8911, @BlaineEXE)
    • rgw: fixing ClientID of log-collector for RGW instance (#8889, @parth-gr)
    • mon: run ceph commands to mon with timeout (#8939, @leseb)
    • osd: do not hide errors (#8933, @leseb)
    • rgw: use trace logs for RGW admin HTTP info (#8937, @BlaineEXE)

    Full Changelog: https://github.com/rook/rook/compare/v1.7.5...v1.7.6

    Source code(tar.gz)
    Source code(zip)
  • v1.7.5(Oct 7, 2021)

    Improvements

    Rook v1.7.5 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • Update csi sidecar references to the latest versions (#8820, @humblec)
    • No longer install the VolumeReplication CRDs from Rook (#8845, @travisn)
    • Initialize rbd block pool after creation (#8923, @Rakshith-R)
    • Close stdoutPipe for the discovery daemon (#8917, @subhamkrai)
    • Add documentation to recover a pod from a lost node (#8742, @subhamkrai)
    • Increasing the auto-resolvable alerts delay to 15m (#8896, @aruniiird)
    • Change CephAbsentMgr to use 'up' query (#8882, @aruniiird)
    • Adding 'namespace' field to the needed ceph queries (#8901, @aruniiird)
    • Update period if period does not exist (#8828, @BlaineEXE)
    • Do not fail on KMS keys deletion (#8868, @leseb)
    • Do not build all the multus args to remote exec cmd (#8860, @leseb)
    • Fix external script when passing monitoring list (#8807, @leseb)
    • Use insecure TLS for bucket health check (#8712, @leseb)
    • Add PVC privileges to the rook-ceph-purge-osd service account (#8833, @ashangit)
    • Fix the example of local PVC-based cluster (#8846, @satoru-takeuchi)
    • Add signal handling for log collector (#8806, @leseb)
    • Prometheus rules format changes (#8774, @aruniiird)
    • Add namespace to ceph node down query (#8793, @aruniiird)
    Source code(tar.gz)
    Source code(zip)
  • v1.7.4(Sep 24, 2021)

    Improvements

    Rook v1.7.4 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

    • Add missing error type check to exec (#8751, @BlaineEXE)
    • Raise minimum supported version of Ceph-CSI to v3.3.0 (#8803, @humblec)
    • Set the Ceph v16.2.6 release as the default version (#8743, @leseb)
    • Pass region to newS3agent() (#8766, @thotz)
    • Remove unnecessary CephFS provisioner permission (#8739, @Madhu-1)
    • Configurable csi provisioner replica count (#8801, @Madhu-1)
    • Allow setting the default storageclass for a filesystem in the helm chart (#8771, @kubealex)
    • Retry object health check if creation fails (#8708, @BlaineEXE)
    • Use the admin socket for the mgr liveness probe (#8721, @jmolmo)
    • Correct the CephFS mirroring documentation (#8732, @leseb)
    • Reconcile OSD PDBs if allowed disruption is 0 (#8698, @sp98)
    • Add peer spec migration to upgrade doc (#8435, @BlaineEXE)
    • Fix lvm osd db device check (#8267, @lyind)
    • Refactor documentation to simplify for the Ceph provider (#8693, @travisn)
    • Emphasize unit tests in the development guide (#8685, @BlaineEXE)
    Source code(tar.gz)
    Source code(zip)
Owner
Rook
Open Cloud-Native Storage for Kubernetes
Rook
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr

rclone 33.9k Aug 1, 2022
QingStor Object Storage service support for go-storage

go-services-qingstor QingStor Object Storage service support for go-storage. Install go get github.com/minhjh/go-service-qingstor/v3 Usage import ( "

minhjh 0 Dec 13, 2021
Rook is an open source cloud-native storage orchestrator for Kubernetes

Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.

Rook 25 Jun 24, 2022
High Performance, Kubernetes Native Object Storage

MinIO Quickstart Guide MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Ama

Multi-Cloud Object Storage 34.5k Aug 4, 2022
Cloud-Native distributed storage built on and for Kubernetes

Longhorn Build Status Engine: Manager: Instance Manager: Share Manager: Backing Image Manager: UI: Test: Release Status Release Version Type 1.1 1.1.2

Longhorn.io 4k Aug 4, 2022
An encrypted object storage system with unlimited space backed by Telegram.

TGStore An encrypted object storage system with unlimited space backed by Telegram. Please only upload what you really need to upload, don't abuse any

The golang.design Initiative 69 Jul 24, 2022
Storj is building a decentralized cloud storage network

Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

Storj 2.5k Aug 1, 2022
tstorage is a lightweight local on-disk storage engine for time-series data

tstorage is a lightweight local on-disk storage engine for time-series data with a straightforward API. Especially ingestion is massively opt

Ryo Nakao 785 Aug 3, 2022
storage interface for local disk or AWS S3 (or Minio) platform

storage interface for local disk or AWS S3 (or Minio) platform

Bo-Yi Wu 14 Apr 19, 2022
SFTPGo - Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

SFTPGo - Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

Nicola Murino 4.6k Aug 9, 2022
Terraform provider for the Minio object storage.

terraform-provider-minio A Terraform provider for Minio, a self-hosted object storage server that is compatible with S3. Check out the documenation on

Refaktory 8 Jan 29, 2022
A Redis-compatible server with PostgreSQL storage backend

postgredis A wild idea of having Redis-compatible server with PostgreSQL backend. Getting started As a binary: ./postgredis -addr=:6380 -db=postgres:/

Ivan Elfimov 1 Nov 8, 2021
CSI for S3 compatible SberCloud Object Storage Service

sbercloud-csi-obs CSI for S3 compatible SberCloud Object Storage Service This is a Container Storage Interface (CSI) for S3 (or S3 compatible) storage

Vitaly 2 Feb 17, 2022
Void is a zero storage cost large file sharing system.

void void is a zero storage cost large file sharing system. License Copyright © 2021 Changkun Ou. All rights reserved. Unauthorized using, copying, mo

Changkun Ou 6 Nov 22, 2021
This is a simple file storage server. User can upload file, delete file and list file on the server.

Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a

BH_Lin 0 Jan 19, 2022
Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.

Perkeep is your personal storage system. It's a way to store, sync, share, import, model, and back up content. Keep your stuff for life. For more, see

Perkeep (née Camlistore) 6k Aug 6, 2022
s3git: git for Cloud Storage. Distributed Version Control for Data.

s3git: git for Cloud Storage. Distributed Version Control for Data. Create decentralized and versioned repos that scale infinitely to 100s of millions of files. Clone huge PB-scale repos on your local SSD to make changes, commit and push back. Oh yeah, it dedupes too and offers directory versioning.

s3git 1.4k Aug 6, 2022
A High Performance Object Storage released under Apache License

MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag

null 1 Sep 30, 2021
Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration

Karmada Karmada: Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration Karmada (Kubernetes Armada) is a Kubernetes management system that enables

null 2.4k Aug 1, 2022
The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your container orchestrator

fortress-csi The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your co

Fortress 0 Jan 23, 2022
Zeebe.io - Workflow Engine for Microservices Orchestration

Distributed Workflow Engine for Microservices Orchestration

null 2.4k Aug 8, 2022
Declarative penetration testing orchestration framework

Decker - Penetration Testing Orchestration Framework Purpose Decker is a penetration testing orchestration framework. It leverages HashiCorp Configura

Steven Aldinger 271 Jul 15, 2022
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Cadence Visit cadenceworkflow.io to learn about Cadence. This repo contains the source code of the Cadence server. To implement workflows, activities

Uber Open Source 6.2k Jul 29, 2022
A programmable, observable and distributed job orchestration system.

?? Overview Odin is a programmable, observable and distributed job orchestration system which allows for the scheduling, management and unattended bac

James McDermott 439 Jul 14, 2022
Serverless SOAR (Security Orchestration, Automation and Response) framework for automatic inspection and evaluation of security alert

DeepAlert DeepAlert is a serverless framework for automatic response of security alert. Overview DeepAlert receives a security alert that is event of

null 37 Aug 5, 2022
Easegress (formerly known as EaseGateway)is an all-rounder traffic orchestration system

Easegress (formerly known as EaseGateway)is an all-rounder traffic orchestration system

MegaEase 4.6k Aug 9, 2022