Cloud-Native distributed storage built on and for Kubernetes

Last update: Jan 1, 2023

Overview

Longhorn

Build Status

Engine:
Manager:
Instance Manager:
Share Manager:
Backing Image Manager:
UI:
Test:

Release Status

Release	Version	Type
1.1	1.1.2	Latest & Stable
1.2	1.2.2	Latest

Overview

Longhorn is a distributed block storage system for Kubernetes. Longhorn is cloud native storage because it is built using Kubernetes and container primitives.

Longhorn is lightweight, reliable, and powerful. You can install Longhorn on an existing Kubernetes cluster with one kubectl apply command or using Helm charts. Once Longhorn is installed, it adds persistent volume support to the Kubernetes cluster.

Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes. Here are some notable features of Longhorn:

Enterprise-grade distributed storage with no single point of failure
Incremental snapshot of block storage
Backup to secondary storage (NFSv4 or S3-compatible object storage) built on efficient change block detection
Recurring snapshot and backup
Automated non-disruptive upgrade. You can upgrade the entire Longhorn software stack without disrupting running volumes!
Intuitive GUI dashboard

You can read more technical details of Longhorn here.

Get Involved

Community Meeting and Office Hours!: Hosted by the core maintainers of Longhorn: 2nd Friday of the every month at 09:00 Pacific Time (PT)/12:00 Eastern Time (ET) on Zoom: http://bit.ly/longhorn-community-meeting. Gcal event: http://bit.ly/longhorn-events

Longhorn Mailing List!: Stay up to date on the latest news and events: https://lists.cncf.io/g/cncf-longhorn

You can read more about the community and its events here: https://github.com/longhorn/community

Current status

The latest release of Longhorn is

Source code

Longhorn is 100% open source software. Project source code is spread across a number of repos:

Component	What it does	GitHub repo
Longhorn Backing Image Manager	Backing image download, sync, and deletion in a disk	longhorn/backing-image-manager
Longhorn Engine	Core controller/replica logic	longhorn/longhorn-engine
Longhorn Instance Manager	Controller/replica instance lifecycle management	longhorn/longhorn-instance-manager
Longhorn Manager	Longhorn orchestration, includes CSI driver for Kubernetes	longhorn/longhorn-manager
Longhorn Share Manager	NFS provisioner that exposes Longhorn volumes as ReadWriteMany volumes	longhorn/longhorn-share-manager
Longhorn UI	The Longhorn dashboard	longhorn/longhorn-ui

Requirements

For the installation requirements, refer to the Longhorn documentation.

Installation

Longhorn can be installed on a Kubernetes cluster in several ways:

Documentation

The official Longhorn documentation is here.

Community

Longhorn is open source software, so contributions are greatly welcome. Please read Code of Conduct and Contributing Guideline before contributing.

Contributing code is not the only way of contributing. We value feedbacks very much and many of the Longhorn features are originated from users' feedback. If you have any feedbacks, feel free to file an issue and talk to the developers at the CNCF #longhorn slack channel.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Longhorn is a CNCF Sandbox Project

Comments

Feature Request: Support for ARM64 Architecture

If not already working I would like to request support for the ARM architecture in order to run a full Rancher toolset on baremetal ARM providers like Packet or Scaleway.
kind/enhancement area/engine kind/test area/install-upgrade highlight priority/0 require/doc require/manual-test-plan

opened by ecliptik 91
[BUG] Self-Hosted Minio-Backupstorage - timeout during Backup
Describe the bug I have longhorn 1.1.0 on Rancher 2.5.1 and a self-hosted minio backupstorage. When I do a backup in Longhorn, it sometimes happens that minio displays a timeout error. The "Snapshots and Backups Longhorn" view shows that a backup was carried out. When I click on "Backup" in the Longhorn UI, the last backup is not available.

To Reproduce Manual or automatic backup to minio.

Expected behavior The backup is created.

Log docker logs -f

API: PutObject(bucket=k8s-cluster01, object=backupstore/volumes/79/dd/pvc-dca02b3d-8845-4e35-b4ba-7e004238d70d/blocks/2f/c1/2fc17d80430fbb443f3d6432f3d3565078acb49be1c6eff98a756888fffcc945.blk) Time: 13:52:46 UTC 01/28/2021 DeploymentID: 60a01f5f-7567-48t6-a9f2-d86b7d8df3c6 RequestID: 165E6977AB3A804E RemoteHost: XXX.XXX.XXX.XX Host: minio.domain.de UserAgent: aws-sdk-go/1.25.16 (go1.14.4; linux; amd64) Error: Operation timed out (cmd.OperationTimedOut) 3: cmd/fs-v1.go:1100:cmd.(*FSObjects).PutObject() 2: cmd/object-handlers.go:1565:cmd.objectAPIHandlers.PutObjectHandler() 1: net/http/server.go:2042:http.HandlerFunc.ServeHTTP()

Environment:

Longhorn version: 1.1.0

Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE

Number of management node in the cluster: 3

Number of worker node in the cluster: 3

Node config

OS type and version: Ubuntu 20.04

CPU per node: 32

Memory per node: 256

Disk type(e.g. SSD/NVMe): SSD

Network bandwidth between the nodes: 1G

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal

Number of Longhorn volumes in the cluster: 25

kind/bug area/engine area/volume-backup-restore investigation-needed duplicated
opened by lucky4ever2 61

cannot format to ext4 error on various environment

Hi,

I can't get Longhorn to work on my kubernetes cluster. I'm using CentOS7 as a base OS, and the iscsi-initiator-utils package is installed, which should contain all the tools required for Longhorn. Kubernetes version 1.9.7.

Installed Longhorn as follows:

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/deploy/longhorn.yaml
kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/deploy/example-storageclass.yaml

All components come up fine:

longhorn-system   longhorn-flexvolume-driver-deployer-85dd94b9bc-gcwrv   1/1       Running             0          12m
longhorn-system   longhorn-flexvolume-driver-td9hk                       1/1       Running             0          12m
longhorn-system   longhorn-flexvolume-driver-xs62n                       1/1       Running             0          12m
longhorn-system   longhorn-manager-2p9hp                                 1/1       Running             0          12m
longhorn-system   longhorn-manager-2xf4g                                 1/1       Running             0          12m
longhorn-system   longhorn-ui-599694bf-zhndh                             1/1       Running             0          12m

Then tried one of the examples:

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/examples/pvc.yaml

This doesn't work, the container gets stuck in the "creating" phase:

default volume-test 0/1 ContainerCreating 0 10m

However, the pvc is bound:

NAME                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
longhorn-volv-pvc   Bound     pvc-8a69cce3-60ac-11e8-a060-02b050e3657a   2Gi        RWO            longhorn       11m

In the Longhorn webui, I saw the volume switching between attached and detached, and now it seems to have settled for "detached":

Detached | pvc-8a69cce3-60ac-11e8-a060-02b050e3657a |   |   | 2 Gi | 12 minutes ago
-- | -- | -- | -- | -- | --

I also noticed some other containers spun up, but they got destroyed:

longhorn-system   pvc-8a69cce3-60ac-11e8-a060-02b050e3657a-r-41479c98    0/1       Terminating         0          46s
longhorn-system   pvc-8a69cce3-60ac-11e8-a060-02b050e3657a-r-456c4516    0/1       Terminating         0          46s
longhorn-system   pvc-8a69cce3-60ac-11e8-a060-02b050e3657a-r-d1ffaba1    0/1       Terminating         0          46s

How can I fix this?

opened by jeroenjacobs79 53

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it

Hello, I updated my longhorn to version 1.3.0, but 1 node was not healthy and crashed some volumes, they are degraded, what can I do?

I tried to delete the node with problems, but it only has this state
kind/bug area/upgrade area/validation-mutation backport/1.3.1

opened by elton557 50

[Question] longhorn-driver-deployer can not start

kubectl get pods \
> --namespace longhorn-system
NAME                                       READY   STATUS             RESTARTS   AGE
engine-image-ei-eee5f438-s7lb4             1/1     Running            0          10m
instance-manager-e-2c134851                1/1     Running            0          10m
instance-manager-r-100de490                1/1     Running            0          10m
longhorn-driver-deployer-cd74cb75b-dlgvt   0/1     Init:0/1           0          10m
longhorn-manager-8g48d                     1/1     Running            0          10m
longhorn-ui-8486987944-r78hc               0/1     CrashLoopBackOff   6          10m

kubectl describe pod longhorn-driver-deployer-cd74cb75b-dlgvt   --namespace longhorn-system

Events:
  Type    Reason     Age        From                              Message
  ----    ------     ----       ----                              -------
  Normal  Scheduled  <unknown>  default-scheduler                 Successfully assigned longhorn-system/longhorn-driver-deployer-cd74cb75b-dlgvt to izj6cco39nfexbhvl3qk7oz
  Normal  Pulled     11m        kubelet, izj6cco39nfexbhvl3qk7oz  Container image "longhornio/longhorn-manager:v1.0.0" already present on machine
  Normal  Created    11m        kubelet, izj6cco39nfexbhvl3qk7oz  Created container wait-longhorn-manager
  Normal  Started    11m        kubelet, izj6cco39nfexbhvl3qk7oz  Started container wait-longhorn-manager

kubectl logs longhorn-driver-deployer-cd74cb75b-dlgvt   --namespace longhorn-system                                                                                                      
Error from server (BadRequest): container "longhorn-driver-deployer" in pod "longhorn-driver-deployer-cd74cb75b-dlgvt" is waiting to start: PodInitializing

kubectl logs longhorn-ui-8486987944-r78hc  --namespace longhorn-system
2020/07/04 09:17:33 [warn] 1#1: duplicate MIME type "text/html" in /etc/nginx/nginx.conf:7
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:7
2020/07/04 09:17:33 [emerg] 1#1: host not found in upstream "longhorn-backend" in /etc/nginx/nginx.conf:32
nginx: [emerg] host not found in upstream "longhorn-backend" in /etc/nginx/nginx.conf:32

kind/enhancement kind/question priority/2 require/knowledge-base

opened by hillbun 48

[BUG] helm upgrade won't apply customize default setting
Is your feature request related to a problem? Please describe.

The longhorn-default-setting ConfigMap sync to settings CR only when settings CR does not exist. This means that during helm upgrade if the user changes the setting, then it won't applies to settings CR (but applies to longhorn-default-setting ConfigMap only)

helm upgrade longhorn longhorn/longhorn -n longhorn-system \ --set defaultSettings.backupTarget=<new-backup-target> \ --set defaultSettings.backupTargetCredentialSecret=<new-backup-target-credential-secret>

Describe the solution you'd like

I think we could configure the settings CR directly. So, we don't have to write a setting controller to reconcile to longhorn-default-setting ConfigMap to settings CR. However, to accomplish it, we need to have a structural schema on CRDs and also, have an admission webhook to validate the input of these settings.

Describe alternatives you've considered

N/A

Additional context

Related issues

https://github.com/longhorn/longhorn/issues/2562#issuecomment-832413461

https://github.com/longhorn/longhorn/issues/2539#issuecomment-827290662

https://github.com/longhorn/longhorn/issues/2611

https://github.com/longhorn/longhorn/issues/2744

https://github.com/longhorn/longhorn/issues/2825

https://github.com/longhorn/longhorn/issues/3398

https://github.com/longhorn/longhorn/issues/3458

kind/enhancement area/manager area/upgrade area/install-upgrade highlight priority/1 require/doc
opened by jenting 46
[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy
Is your improvement request related to a feature? Please describe

PodSecurityPolicy has been deprecated and will be removed from K8s 1.25, so we need to find an alternative way to resolve the need for PSP in Longhorn to support 1.25.

Also, some deprecated resource versions are also removed from 1.25. Need to resolve this via https://github.com/longhorn/longhorn/issues/4239 or even consider to backport this to 1.3 & 1.2 via an adaptive way to determine the K8s version of the cluster to use which API resource version if possible (except PSP, because it's totally removed instead of version bump).

Cronjob v1beta1 -> v1

EndpointSlice v1beta1 -> v1

Event v1beta1 -> v1

HorizontalPodAutoscaler v2beta1 -> v2

PodDisruptionBudget v1beta1 -> removed

Note: client-go is backward compatible with K8s any version.

Compatibility: client-go <-> Kubernetes clusters Since Kubernetes is backwards compatible with clients, older client-go versions will work with many different Kubernetes cluster versions.

Describe the solution you'd like

Deprecate PSP if it's not needed. Otherwise, we need an alternative solution like https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/.

Describe alternatives you've considered

N/A

Additional context

https://www.kubernetes.dev/resources/release/#timeline https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-25 https://github.com/longhorn/longhorn/issues/4239
area/kubernetes highlight release/note priority/0 require/doc require/manual-test-plan kind/improvement
opened by innobead 42
[BUG] Backup - S3 Timeout
Describe the bug In the backup UI, I get an error and the backups are not listed.

error listing backups: error listing backup volumes: Timeout executing: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.0/longhorn [backup ls --volume-only s3://PATH/], output , stderr, , error <nil>

Number of Volumes: 39 Number of Volumes with backup enabled: 32

S3 Bucket Size: 2TB

Expected behavior Backups of Volumes are showed

Log time="2020-11-05T12:55:15Z" level=warning msg="backup store monitor: failed to list backup volumes in s3://S3PATH/: error listing backup volumes: Timeout executing: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.0/longhorn [backup ls --volume-only s3://longhorn-production@ch-dk-2/], output , stderr, , error <nil>"

Environment:

Longhorn version: 1.0.0

Kubernetes version: 1.18.3

Node OS type and version: Centos 7.7

Additional context On an empty bucket the backups are listed normally.

S3 Provider https://exoscale.com
kind/bug area/engine priority/1 area/volume-backup-restore
opened by timmy59100 40

[BUG] Corruption using XFS after node restart or pod scale

Describe the bug

Upon either restarting a kubernetes worker node hosting Longhorn replicas, or during pod scaling (e.g. scaling a StatefulSet to 0), I've sometimes experienced silent data corruption. Following the Longhorn docs for detecting and deleting a failed replica does find a different hash, but deleting the replica doesn't solve the problem. Ultimately restoring from a snapshot is the only fix.

Of note, I'm using XFS as the underlying filesystem, and all affected apps have used SQLite.

Unfortunately/fortunately, this problem seems to be non-deterministic. I've experienced it twice; once when restarting a node for maintenance, and once scaling an app down and up.

To Reproduce

Steps to reproduce the behavior:

Use StatefulSets with VolumeClaimTemplates to define PVCs.
Restart a node containing replicas, or scale StatefulSets utilizing the PVCs to 0 and back to n>0.

Expected behavior

The replica volumes to return to service with no corruption.

Log or Support bundle

Example dmesg from a longhorn-manager pod. Note that while this shows the recovery completing, others do not. I unfortunately don't have timestamps to match up a failed one, although I can confirm that I experienced issues with this linked PVC, requiring a snapshot recovery.


[ 1332.278329] XFS (sdc): Metadata CRC error detected at xfs_agfl_read_verify+0xa2/0xf0 [xfs], xfs_agfl block 0x3
[ 1332.278332] XFS (sdc): Unmount and run xfs_repair
[ 1332.278334] XFS (sdc): First 128 bytes of corrupted metadata buffer:
[ 1332.278343] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278344] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278345] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278346] 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278346] 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278347] 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278348] 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278348] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 1332.278368] XFS (sdc): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x3 len 1 error 74
[ 1332.365389] XFS (sdc): xfs_do_force_shutdown(0x8) called from line 446 of file fs/xfs/libxfs/xfs_defer.c. Return address = 0000000061a87018
[ 1332.365393] XFS (sdc): Corruption of in-memory data detected.  Shutting down filesystem
[ 1332.365394] XFS (sdc): Please unmount the filesystem and rectify the problem(s)
[ 1447.117969] cni0: port 13(veth6f28afba) entered disabled state
[ 1447.129033] device veth6f28afba left promiscuous mode
[ 1447.129059] cni0: port 13(veth6f28afba) entered disabled state
[ 1448.304291] XFS (sdc): Unmounting Filesystem
[ 1452.882691] sd 7:0:0:1: [sdc] Synchronizing SCSI cache
[ 1459.419198] scsi host7: iSCSI Initiator over TCP/IP
[ 1459.448202] scsi 7:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
[ 1459.450009] scsi 7:0:0:0: Attached scsi generic sg3 type 12
[ 1459.451601] scsi 7:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
[ 1459.452849] sd 7:0:0:1: Attached scsi generic sg4 type 0
[ 1459.453233] sd 7:0:0:1: Power-on or device reset occurred
[ 1459.455747] sd 7:0:0:1: [sdc] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
[ 1459.456106] sd 7:0:0:1: [sdc] Write Protect is off
[ 1459.456111] sd 7:0:0:1: [sdc] Mode Sense: 69 00 10 08
[ 1459.456794] sd 7:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 1460.586295] sd 7:0:0:1: [sdc] Attached SCSI disk
[ 1469.044275] XFS (sdc): Mounting V5 Filesystem
[ 1471.123131] XFS (sdc): Starting recovery (logdev: internal)
[ 1471.572637] XFS (sdc): Ending recovery (logdev: internal)
[ 1471.624639] xfs filesystem being mounted at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-81fb33a4-4b2e-425b-b6df-9dbd7af36034/globalmount supports timestamps until 2038 (0x7fffffff)
[ 1471.850475] xfs filesystem being remounted at /var/lib/kubelet/pods/f0f3ff6b-841c-4c4e-a0f9-28cbd9fbac80/volumes/kubernetes.io~csi/pvc-81fb33a4-4b2e-425b-b6df-9dbd7af36034/mount supports timestamps until 2038
(0x7fffffff)

/sys/devices/system/edac/mc/mc{0,1} on each node shows 0 corrected and 0 uncorrected errors.

Environment

Longhorn version: v1.2.3
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: K3s v1.21.5+k3s2
- Number of management node in the cluster: 3
- Number of worker node in the cluster: 3
Node config
- OS type and version: k3OS v0.21.5-k3s2r1
- CPU per node: E5-2650 v2
  - Manager: 4
  - Worker: 28
- Memory per node: DDR3-PC10600R
  - Manager: 8 Gi
  - Worker: 24 Gi
- Disk type(e.g. SSD/NVMe): SSD (Intel DC S3500 300GB)
- Network bandwidth between the nodes: 1 GBe
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM (Proxmox)
Number of Longhorn volumes in the cluster: 20

Additional context

Add any other context about the problem here.

kind/bug area/engine severity/1 reproduce/rare priority/0 require/automation-e2e require/manual-test-plan duplicated backport/1.3.2 area/data-integrity backport/1.2.6

opened by stephanGarland 39

Remove the first backup snapshot of a restored volume result in rebuild always fail
there is a problem with the other two replicas which wasn't rebuilding:

time="2018-12-27T07:00:05Z" level=error msg="Error in request: Replica tcp://10.42.0.51:9502's chain not equal to RW replica tcp://10.42.1.245:9502's chain"

And this caused the rebuilding replica was reopened repeatedly without actually being worked on.

time="2018-12-27T07:45:01Z" level=error msg="Error in request: Replica must be closed, Can not add in state: open"

See the comments starting at: https://github.com/rancher/longhorn/issues/253#issuecomment-443540620
kind/bug area/engine area/manager kind/test
opened by yasker 39

[BUG] Instance managers and Pods with attached volumes restarted every hour

Describe the bug On a number of different clusters I've had over the past few months (k3s of various versions, on various clouds and OKE (Oracle Cloud)) on 1.19 and 1.20, I've had an issue where all instance managers and Pods with attached volumes get restarted precisely every hour. Sometimes redploying the whole of Longhorn from scratch and restoring from backup resolves the issue. The nodes are all healthy in this scenario and the rest of the cluster is stable and unchanged. However like clockwork, every hour it restarts all my pods.

Related to: https://github.com/longhorn/longhorn/issues/2435

To Reproduce I need to do further testing to see if it happens with completely fresh cluster without my backups restored. However it is currently happening for me on a two node ARM-based OKE cluster on Kubernetes 1.20, with 6 small volumes (100-500MB) restored from S3 backups and 3 larger 1-10GB volumes which were created fresh.

Expected behavior To not have everything restart every hour.

Log There is nothing in the logs indicating this is about to happen. The only thing which gets logged is the recovery.

time="2021-07-24T08:53:13Z" level=debug msg="Polling backup store for new volume backups" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185
time="2021-07-24T08:53:13Z" level=debug msg="Refreshed all volumes last backup based on backup store information" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="Stop monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=error msg="error receiving next item in engine watch: rpc error: code = Canceled desc = context canceled" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=error msg="error receiving next item in engine watch: rpc error: code = Unavailable desc = transport is closing" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="Stop monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Cannot find the instance manager for the running instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031, will mark the instance as state ERROR"
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 state, old state running, new state error"
time="2021-07-24T09:00:48Z" level=warning msg="Try to get requested log for pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c on node 10.0.121.185"
time="2021-07-24T09:00:48Z" level=warning msg="cannot get requested log for instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c on node 10.0.121.185, error invalid Instance Manager instance-manager-r-07214e36, state: error, IP: "
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state error, new state stopped"
time="2021-07-24T09:00:48Z" level=warning msg="Try to get requested log for pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 on node 10.0.121.185"
time="2021-07-24T09:00:48Z" level=warning msg="cannot get requested log for instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 on node 10.0.121.185, error invalid Instance Manager instance-manager-r-07214e36, state: error, IP: "
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 state, old state error, new state stopped"
time="2021-07-24T09:00:48Z" level=warning msg="Try to get requested log for pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 on node 10.0.121.185"
time="2021-07-24T09:00:48Z" level=warning msg="cannot get requested log for instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 on node 10.0.121.185, error invalid Instance Manager instance-manager-r-07214e36, state: error, IP: "
time="2021-07-24T09:00:48Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state error, new state stopped"
time="2021-07-24T09:00:58Z" level=info msg="Created instance manager pod instance-manager-e-04771971 for instance manager instance-manager-e-04771971"
time="2021-07-24T09:00:58Z" level=info msg="Created instance manager pod instance-manager-r-07214e36 for instance manager instance-manager-r-07214e36"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c, will mark the instance as state ERROR"
time="2021-07-24T09:00:58Z" level=warning msg="The starting instance manager instance-manager-r-07214e36 shouldn't contain the running instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=debug msg="Start monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:00:59Z" level=debug msg="Start monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9, will mark the instance as state ERROR"
time="2021-07-24T09:00:59Z" level=warning msg="Cannot find the instance status in instance manager instance-manager-r-07214e36 for the running instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031, will mark the instance as state ERROR"
time="2021-07-24T09:01:02Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state error, new state stopped"
time="2021-07-24T09:01:04Z" level=debug msg="Instance handler updated instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 state, old state error, new state stopped"
time="2021-07-24T09:01:06Z" level=debug msg="Instance handler updated instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e state, old state error, new state stopped"
time="2021-07-24T09:01:07Z" level=debug msg="Instance handler updated instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e state, old state error, new state stopped"
time="2021-07-24T09:01:09Z" level=debug msg="Instance handler updated instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c state, old state error, new state stopped"
time="2021-07-24T09:01:13Z" level=debug msg="Instance handler updated instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 state, old state error, new state stopped"
time="2021-07-24T09:01:16Z" level=debug msg="Stop monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:01:16Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:01:16Z" level=error msg="error receiving next item in engine watch: rpc error: code = Canceled desc = context canceled" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
time="2021-07-24T09:01:17Z" level=debug msg="Stop monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:17Z" level=debug msg="removed the engine from imc.instanceManagerMonitorMap" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:17Z" level=error msg="error receiving next item in engine watch: rpc error: code = Canceled desc = context canceled" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:19Z" level=warning msg="Error syncing Longhorn replica longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9" controller=longhorn-replica error="fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP: " node=10.0.121.185
time="2021-07-24T09:01:19Z" level=warning msg="Error syncing Longhorn replica longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9" controller=longhorn-replica error="fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP: " node=10.0.121.185
E0724 09:01:19.450376       1 replica_controller.go:178] fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP:
time="2021-07-24T09:01:19Z" level=warning msg="Dropping Longhorn replica longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 out of the queue" controller=longhorn-replica error="fail to sync replica for longhorn-system/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9: invalid Instance Manager instance-manager-r-07214e36, state: error, IP: " node=10.0.121.185
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3453 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4564 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1?action=detach HTTP/1.1" 200 2229 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3525 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338?action=detach HTTP/1.1" 200 3362 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4568 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-7be8efab-347f-463a-b507-3875c8e369fc detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc?action=detach HTTP/1.1" 200 3366 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6?action=detach HTTP/1.1" 200 3376 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3465 "" "Go-http-client/1.1"
time="2021-07-24T09:01:25Z" level=info msg="Volume pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72?action=detach HTTP/1.1" 200 2324 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:25 +0000] "POST /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e?action=detach HTTP/1.1" 200 2244 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3454 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4584 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4568 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3526 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:27 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3466 "" "Go-http-client/1.1"
time="2021-07-24T09:01:28Z" level=info msg="Created instance manager pod instance-manager-r-07214e36 for instance manager instance-manager-r-07214e36"
time="2021-07-24T09:01:28Z" level=info msg="Created instance manager pod instance-manager-e-04771971 for instance manager instance-manager-e-04771971"
10.244.2.136 - - [24/Jul/2021:09:01:29 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3454 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:29 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3506 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:29 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3446 "" "Go-http-client/1.1"
time="2021-07-24T09:01:29Z" level=debug msg="Start monitoring instance manager instance-manager-e-04771971" controller=longhorn-instance-manager instance manager=instance-manager-e-04771971 node=10.0.121.185
time="2021-07-24T09:01:30Z" level=debug msg="Start monitoring instance manager instance-manager-r-07214e36" controller=longhorn-instance-manager instance manager=instance-manager-r-07214e36 node=10.0.121.185
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3434 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3526 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
time="2021-07-24T09:01:31Z" level=info msg="Volume pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "POST /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6?action=attach HTTP/1.1" 200 3376 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:31 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3466 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3454 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3506 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4558 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:33 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3466 "" "Go-http-client/1.1"
time="2021-07-24T09:01:34Z" level=debug msg="Prepare to create instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031"
time="2021-07-24T09:01:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031\", UID:\"908ed51e-b8a2-4e57-8bc0-c519bb40c3c9\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015760\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031"
time="2021-07-24T09:01:35Z" level=debug msg="Instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:35Z" level=debug msg="Instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 starts running, Port 10000"
time="2021-07-24T09:01:35Z" level=debug msg="Instance handler updated instance pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6-r-5878c031 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3506 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:35 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3449 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:36 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3429 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4548 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4544 "" "Go-http-client/1.1"
time="2021-07-24T09:01:37Z" level=info msg="Volume pvc-7be8efab-347f-463a-b507-3875c8e369fc attachment to 10.0.117.182 with disableFrontend false requested"
time="2021-07-24T09:01:37Z" level=info msg="Volume pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "POST /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338?action=attach HTTP/1.1" 200 3362 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "POST /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc?action=attach HTTP/1.1" 200 3366 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 4609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
time="2021-07-24T09:01:37Z" level=info msg="Volume pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:37 +0000] "POST /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72?action=attach HTTP/1.1" 200 3360 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4544 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4548 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:39 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
time="2021-07-24T09:01:40Z" level=debug msg="Prepare to create instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066"
time="2021-07-24T09:01:40Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066\", UID:\"6ad0043e-62ff-4ec5-b0d2-399c22c3b563\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015878\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066"
10.244.2.136 - - [24/Jul/2021:09:01:40 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:40 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3377 "" "Go-http-client/1.1"
time="2021-07-24T09:01:41Z" level=debug msg="Instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:41Z" level=debug msg="Instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 starts running, Port 10015"
time="2021-07-24T09:01:41Z" level=debug msg="Instance handler updated instance pvc-7be8efab-347f-463a-b507-3875c8e369fc-r-62b67066 state, old state stopped, new state running"
time="2021-07-24T09:01:41Z" level=debug msg="Prepare to create instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:01:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015897\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4524 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4599 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:41 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 4562 "" "Go-http-client/1.1"
time="2021-07-24T09:01:41Z" level=debug msg="Prepare to create instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e"
time="2021-07-24T09:01:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e\", UID:\"47166af1-dc55-4945-aabf-b68034792aa1\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015902\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e starts running, IP 10.244.2.205"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e starts running, Port 10045"
time="2021-07-24T09:01:42Z" level=debug msg="Instance handler updated instance pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338-r-af4a8f2e state, old state stopped, new state running"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:42Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, Port 10030"
time="2021-07-24T09:01:42Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 4595 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 4599 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:43 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3505 "" "Go-http-client/1.1"
time="2021-07-24T09:01:44Z" level=debug msg="Prepare to create instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:01:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015945\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:01:44Z" level=debug msg="Instance process pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c had been created, need to wait for instance manager update"
10.244.2.136 - - [24/Jul/2021:09:01:44 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:44 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 4462 "" "Go-http-client/1.1"
time="2021-07-24T09:01:44Z" level=info msg="Volume pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:44 +0000] "POST /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e?action=attach HTTP/1.1" 200 3280 "" "Go-http-client/1.1"
time="2021-07-24T09:01:45Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, IP 10.244.2.205"
time="2021-07-24T09:01:45Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, Port 10060"
time="2021-07-24T09:01:45Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state stopped, new state running"
time="2021-07-24T09:01:45Z" level=debug msg="Prepare to create instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:01:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015973\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3538 "" "Go-http-client/1.1"
time="2021-07-24T09:01:45Z" level=debug msg="Prepare to create instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e"
time="2021-07-24T09:01:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e\", UID:\"0b2f24bc-2883-4164-b608-25a150d1876a\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9015978\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3542 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3552 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:45 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3505 "" "Go-http-client/1.1"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e starts running, IP 10.244.2.205"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e starts running, Port 10090"
time="2021-07-24T09:01:46Z" level=debug msg="Instance handler updated instance pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e-r-2b7f722e state, old state stopped, new state running"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, IP 10.244.2.205"
time="2021-07-24T09:01:46Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, Port 10075"
time="2021-07-24T09:01:46Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:46 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 4533 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3613 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3623 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:47 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3576 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4450 "" "Go-http-client/1.1"
time="2021-07-24T09:01:48Z" level=info msg="Volume pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "POST /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1?action=attach HTTP/1.1" 200 3265 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:48 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 4533 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3609 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3623 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:49 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3588 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:50 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4450 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:50 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3476 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3621 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:51 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3588 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:52 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4450 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:52 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3621 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:53 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3588 "" "Go-http-client/1.1"
time="2021-07-24T09:01:53Z" level=debug msg="Prepare to create instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c"
time="2021-07-24T09:01:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c\", UID:\"c28e49bc-2bea-49fb-a581-076b13e8f67e\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016091\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c"
time="2021-07-24T09:01:54Z" level=debug msg="Instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c starts running, IP 10.244.2.205"
time="2021-07-24T09:01:54Z" level=debug msg="Instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c starts running, Port 10105"
time="2021-07-24T09:01:54Z" level=debug msg="Instance handler updated instance pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1-r-4a8c740c state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:01:54 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 4521 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:54 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3648 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3417 "" "Go-http-client/1.1"
time="2021-07-24T09:01:55Z" level=info msg="Volume pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3651 "" "Go-http-client/1.1"
time="2021-07-24T09:01:55Z" level=info msg="Volume pvc-65580647-e44a-4210-8545-2aff63ff0fe2 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "POST /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2?action=detach HTTP/1.1" 200 2316 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3682 "" "Go-http-client/1.1"
time="2021-07-24T09:01:55Z" level=info msg="Volume pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 detachment from node 10.0.117.182 requested"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "POST /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270?action=detach HTTP/1.1" 200 2320 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 3659 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:55 +0000] "POST /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa?action=detach HTTP/1.1" 200 2053 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:56 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3464 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:56 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3648 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3679 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3727 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 5456 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:57 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3462 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:58 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3535 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:58 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3692 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 3625 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3659 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3662 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3695 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 5456 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:01:59 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3442 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:00 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:00 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3559 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3692 "" "Go-http-client/1.1"
time="2021-07-24T09:02:01Z" level=debug msg="Prepare to create instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483"
time="2021-07-24T09:02:01Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483\", UID:\"acbef7c7-8e56-4687-933f-718fbac4b78a\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016207\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 5493 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3659 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 3706 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3695 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72 HTTP/1.1" 200 5636 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:01 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3791 "" "Go-http-client/1.1"
time="2021-07-24T09:02:02Z" level=debug msg="Instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 starts running, IP 10.244.2.205"
time="2021-07-24T09:02:02Z" level=debug msg="Instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 starts running, Port 10120"
time="2021-07-24T09:02:02Z" level=debug msg="Instance handler updated instance pvc-ee608cc7-bc48-4abc-97d2-c72caf3a0c72-r-553d4483 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:02:02 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:02 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3586 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 3692 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 5493 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3703 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3695 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:03 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3791 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:04 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:04 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3586 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 5489 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-7be8efab-347f-463a-b507-3875c8e369fc HTTP/1.1" 200 5842 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3703 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3624 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:05 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4182 "" "Go-http-client/1.1"
time="2021-07-24T09:02:06Z" level=debug msg="Prepare to delete instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016263\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:06Z" level=debug msg="Prepare to delete instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016264\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
10.244.2.136 - - [24/Jul/2021:09:02:06 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:06 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3630 "" "Go-http-client/1.1"
time="2021-07-24T09:02:07Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state running, new state stopped"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 5489 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4052 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-dd45f8fb-fc5e-4abb-bc15-62ed9d042dd6 HTTP/1.1" 200 5852 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3553 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:07 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4166 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:08 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3547 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:08 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 3630 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-ab37a40c-a031-4b0d-bb6d-a09f01bd3338 HTTP/1.1" 200 5838 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4052 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3553 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:09 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4046 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:10 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 3618 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:10 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 5427 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:11 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4443 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:11 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3429 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:11 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4046 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:12 +0000] "GET /v1/volumes/pvc-3775cc56-b35a-4d5f-8d4b-ed16880d19c1 HTTP/1.1" 200 5764 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:12 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 5427 "" "Go-http-client/1.1"
time="2021-07-24T09:02:13Z" level=debug msg="Prepare to delete instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016350\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:13Z" level=debug msg="Prepare to delete instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016353\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
10.244.2.136 - - [24/Jul/2021:09:02:13 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4427 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:13 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3358 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:13 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3235 "" "Go-http-client/1.1"
time="2021-07-24T09:02:14Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state running, new state stopped"
10.244.2.136 - - [24/Jul/2021:09:02:14 +0000] "GET /v1/volumes/pvc-de808e12-9704-413c-a4b9-86bcabe4ca6e HTTP/1.1" 200 5776 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4307 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 4538 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3164 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 4538 "" "Go-http-client/1.1"
time="2021-07-24T09:02:15Z" level=info msg="Volume pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:02:15 +0000] "POST /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270?action=attach HTTP/1.1" 200 3356 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:17 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3496 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:17 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3093 "" "Go-http-client/1.1"
time="2021-07-24T09:02:17Z" level=debug msg="Prepare to create instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
time="2021-07-24T09:02:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c\", UID:\"5b8f6e66-4d04-4064-90ea-00efa1de7df6\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016418\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c"
10.244.2.136 - - [24/Jul/2021:09:02:17 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 4538 "" "Go-http-client/1.1"
time="2021-07-24T09:02:17Z" level=debug msg="Prepare to delete instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016420\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:17Z" level=debug msg="Prepare to delete instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016421\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:18Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, IP 10.244.2.205"
time="2021-07-24T09:02:18Z" level=debug msg="Instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c starts running, Port 10030"
time="2021-07-24T09:02:18Z" level=debug msg="Instance handler updated instance pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270-r-d98ed33c state, old state stopped, new state running"
time="2021-07-24T09:02:18Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state running, new state stopped"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4534 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4273 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/nodes/10.0.117.182 HTTP/1.1" 200 2197 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 4534 "" "Go-http-client/1.1"
time="2021-07-24T09:02:19Z" level=info msg="Volume pvc-65580647-e44a-4210-8545-2aff63ff0fe2 attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 4273 "" "Go-http-client/1.1"
time="2021-07-24T09:02:19Z" level=info msg="Volume pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa attachment to 10.0.117.182 with disableFrontend false requested"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3635 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "POST /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2?action=attach HTTP/1.1" 200 3352 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:19 +0000] "POST /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa?action=attach HTTP/1.1" 200 3089 "" "Go-http-client/1.1"
time="2021-07-24T09:02:20Z" level=debug msg="Prepare to create instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9\", UID:\"f40af8c3-b27e-41f4-9df0-d8fa96b353f5\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016464\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9"
time="2021-07-24T09:02:20Z" level=debug msg="Prepare to create instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852\", UID:\"3cdcfb9c-c86d-477e-b682-d16e8ee24638\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"9016469\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, IP 10.244.2.205"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 starts running, Port 10075"
time="2021-07-24T09:02:21Z" level=debug msg="Instance handler updated instance pvc-65580647-e44a-4210-8545-2aff63ff0fe2-r-711d6852 state, old state stopped, new state running"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, IP 10.244.2.205"
time="2021-07-24T09:02:21Z" level=debug msg="Instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 starts running, Port 10060"
time="2021-07-24T09:02:21Z" level=debug msg="Instance handler updated instance pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa-r-d55a13c9 state, old state stopped, new state running"
10.244.2.136 - - [24/Jul/2021:09:02:21 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 3662 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:21 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3548 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:21 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3358 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:23 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:23 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 3658 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:23 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 3397 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:25 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:25 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 5499 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:25 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 5238 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:27 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 5503 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:27 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 5499 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:27 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 5238 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:29 +0000] "GET /v1/volumes/pvc-5c4f2a7f-97e0-4c23-8055-4d78c302c270 HTTP/1.1" 200 6243 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:29 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 5499 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:29 +0000] "GET /v1/volumes/pvc-75559a4f-b71b-4998-8ad7-1c76a1129afa HTTP/1.1" 200 5978 "" "Go-http-client/1.1"
10.244.2.136 - - [24/Jul/2021:09:02:31 +0000] "GET /v1/volumes/pvc-65580647-e44a-4210-8545-2aff63ff0fe2 HTTP/1.1" 200 6239 "" "Go-http-client/1.1"
time="2021-07-24T09:03:13Z" level=debug msg="Polling backup store for new volume backups" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185
time="2021-07-24T09:03:13Z" level=debug msg="Refreshed all volumes last backup based on backup store information" component=backup-store-monitor controller=longhorn-setting node=10.0.121.185

You can also attach a Support Bundle here. You can generate a Support Bundle using the link at the footer of the Longhorn UI. longhorn-support-bundle_d4963230-e637-4991-9edd-4526a0295afe_2021-07-22T18-24-07Z.zip

Environment:

Longhorn version: v1.1.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: OKE 1.20
- Number of management node in the cluster: Unknown. Managed.
- Number of worker node in the cluster: 2
Node config
- OS type and version: Oracle Linux 7.9
- CPU per node: 2 (ARM64)
- Memory per node: 12GB
- Disk type(e.g. SSD/NVMe): Network attached SSDs of some kind
- Network bandwidth between the nodes: Unknown but unlikely to be a factor
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Oracle Cloud OKE
Number of Longhorn volumes in the cluster: 9

This has happened to me on single node (Intel) k3s clusters in AWS, a 3 node (Intel) k3s cluster on Civo Cloud as well as my current two node ARM64 Oracle setup. All with replicas set appropriately for the number of nodes of course.

Additional context If the support bundle comes up with nothing useful I'll spin up a few different clusters and see if I can get any more idea about what triggers it.

FAO @joshimoo

kind/bug area/install-upgrade severity/3 backport/1.2.4 backport/1.1.4

opened by cablespaghetti 37

[BUG] RWX doesn't work with release 1.4.0

Describe the bug (🐛 if you encounter this issue)

I reinstall longhorn 1.4.0 with k3s 1.25.5, everything is fine but RWX volume mount is repeatedly failed

To Reproduce

Steps to reproduce the behavior:

Make a volume with RWX
Mount it with a pod

Expected behavior

RWX should be mounted, as was in 1.3.2

Log or Support bundle

Here is the log from share-manager-:

│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FILEHANDLE from NIV_EVENT to NIV_INFO                                                                       │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DISPATCH from NIV_EVENT to NIV_INFO                                                                         │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE from NIV_EVENT to NIV_INFO                                                                      │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE_LRU from NIV_EVENT to NIV_INFO                                                                  │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE from NIV_EVENT to NIV_INFO                                                                        │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE_CACHE from NIV_EVENT to NIV_INFO                                                                  │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DUPREQ from NIV_EVENT to NIV_INFO                                                                           │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_INIT from NIV_EVENT to NIV_INFO                                                                             │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_MAIN from NIV_EVENT to NIV_INFO                                                                             │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_IDMAPPER from NIV_EVENT to NIV_INFO                                                                         │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_READDIR from NIV_EVENT to NIV_INFO                                                                      │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_LOCK from NIV_EVENT to NIV_INFO                                                                      │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CONFIG from NIV_EVENT to NIV_INFO                                                                           │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CLIENTID from NIV_EVENT to NIV_INFO                                                                         │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_SESSIONS from NIV_EVENT to NIV_INFO                                                                         │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_PNFS from NIV_EVENT to NIV_INFO                                                                             │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RW_LOCK from NIV_EVENT to NIV_INFO                                                                          │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NLM from NIV_EVENT to NIV_INFO                                                                              │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RPC from NIV_EVENT to NIV_INFO                                                                              │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_TIRPC from NIV_EVENT to NIV_INFO                                                                            │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_CB from NIV_EVENT to NIV_INFO                                                                           │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_THREAD from NIV_EVENT to NIV_INFO                                                                           │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_ACL from NIV_EVENT to NIV_INFO                                                                       │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_STATE from NIV_EVENT to NIV_INFO                                                                            │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P from NIV_EVENT to NIV_INFO                                                                               │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P_DISPATCH from NIV_EVENT to NIV_INFO                                                                      │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FSAL_UP from NIV_EVENT to NIV_INFO                                                                          │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DBUS from NIV_EVENT to NIV_INFO                                                                             │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_MSK from NIV_EVENT to NIV_INFO                                                                          │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed                                                                                               │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] init_fds_limit :INODE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576.                                                                                             │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] init_server_pkgs :NFS STARTUP :INFO :State lock layer successfully initialized                                                                                                    │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] init_server_pkgs :NFS STARTUP :INFO :IP/name cache successfully initialized                                                                                                       │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.                                                                                                                     │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.                                                                                                         │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] nfs4_recovery_init :CLIENT ID :INFO :Recovery Backend Init for longhorn                                                                                                           │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] longhorn_recov_init :CLIENT ID :EVENT :Initialize recovery backend 'share-manager-shared-volume'                                                                                  │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90                                                                                                               │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] longhorn_read_recov_clids :CLIENT ID :EVENT :Read clients from recovery backend share-manager-shared-volume                                                                       │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] read_clids :CLIENT ID :EVENT :response={"actions":{},"clients":[],"hostname":"share-manager-shared-volume","id":"share-manager-shared-volume","links":{"self":"http://longhorn-re │
│                                                                                                                                                                                                                                                                             │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend                                                                                                    │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)                                                                                                   │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] longhorn_recov_end_grace :CLIENT ID :EVENT :End grace for recovery backend 'share-manager-shared-volume' version LUUZWL8T                                                         │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] http_call :CLIENT ID :EVENT :HTTP error: 500 (url=http://longhorn-recovery-backend:9600/v1/recoverybackend/share-manager-shared-volume, payload={"version": "LUUZWL8T"})          │
│ 31/12/2022 22:40:52 : epoch 63b0ba74 : share-manager-shared-volume : nfs-ganesha-29[main] longhorn_recov_end_grace :CLIENT ID :FATAL :HTTP call error: res=-1 ((null))                                                                                                      │
│ time="2022-12-31T22:40:52Z" level=error msg="NFS server exited with error" encrypted=false error="ganesha.nfsd failed with error: exit status 2, output: " volume=shared-volume                                                                                             │
│ W1231 22:40:52.523325       1 mount_helper_common.go:133] Warning: "/export/shared-volume" is not a mountpoint, deleting                                                                                                                                                    │
│ time="2022-12-31T22:40:52Z" level=debug msg="Device /dev/mapper/shared-volume is not an active LUKS device" error="failed to run cryptsetup args: [status shared-volume] output:  error: exit status 4"

Environment

Longhorn version: 1.4.0
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s 1.25.5
- Number of management node in the cluster: 2
- Number of worker node in the cluster: 2
Node config
- OS type and version: Ubuntu 20.04
- CPU per node: 64
- Memory per node: 384Gi
- Disk type(e.g. SSD/NVMe): SSD
- Network bandwidth between the nodes: 10G + 10G (link aggregated)
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): On-prem
Number of Longhorn volumes in the cluster: 7

Additional context

Add any other context about the problem here.

kind/bug area/manager investigation-needed area/volume-rwx area/scalability backport/1.4.1

opened by jinserk 18

[TASK] Update the test log instruction in the test repo.

What's the task? Please describe

The instruction to watch the test logs need to be updated in our test repo. https://github.com/longhorn/longhorn-tests/tree/master/manager/integration

Same has to be updated like below kubectl logs -f longhorn-test -c longhorn-test
kind/task severity/4

opened by khushboo-rancher 0
[TASK] Clarify if any upcoming K8s API deprecation/removal will impact Longhorn 1.4
What's the task? Please describe

After discussing with @PhanLe1010 again, we need to double clarify if the removed storage v1beta1 API from 1.27 really impacts us.

There is no v1beta1 storage.k8s.io used in our current code base (1.4 and master), because we already upgrade to v1.

https://kubernetes.io/docs/reference/using-api/deprecation-guide/#csistoragecapacity-v127 is not related to CSI snapshot CRD which is individual from K8s builtin.

Describe the items of the task (DoD, definition of done) you'd like

[ ] clarify if the upcoming API deprecation/removal will impact Longhorn 1.4

Additional context

cc @longhorn/dev
area/kubernetes priority/0 kind/task area/upstream area/compatibility
opened by innobead 0
[TEST] Create a testing guidance for feature/regression testing
What's the test to develop? Please describe

Usually, QA will rely on the test cases provided by engineering to do the feature testing or bug regression testing. However, it's the potential to ignore unhappy paths.

We should provide a non-programming framework or guidance to let QA members follow up on how to test a new feature or regression bug, not only rely on the test cases provided by engineering but also base on them and cover possible unhappy paths, especially involuntary factors/events like network partition, node reboot, node deletion, kubelet restart, etc. The framework and guidance can be like a cheat sheet.

Describe the items of the test development (DoD, definition of done) you'd like

[ ] Create a cheat sheet/framework/guidance doc for QA to guide them on how to test a feature or bug regression.

Additional context

A good example when testing online volume expansion, https://github.com/longhorn/longhorn/issues/1674#issuecomment-1366969645

cc @longhorn/qa
kind/test priority/0
opened by innobead 0

Releases(v1.4.0)

v1.4.0(Dec 30, 2022)
Release Note

v1.4.0 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Kubernetes 1.25 Support [doc] In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

ARM64 GA ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

RWX GA [lep][doc] RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

Volume Snapshot Checksum [lep][doc] Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

Volume Bit-rot Protection [lep][doc] When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

Volume Replica Rebuilding Speedup When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

Volume Trim [lep][doc] Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

Online Volume Expansion [doc] Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

Local Volume via Data Locality Strict Mode [lep][doc] Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

Volume Recurring Job Backup Restore [lep][doc] Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

Volume IO Metrics [doc] Longhorn enriches Volume metrics by providing real-time IO stats including IOPS, latency, and throughput of R/W IO. Users can set up a monotoning solution like Prometheus to monitor volume performance.

Longhorn System Backup & Restore [lep][doc] Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

Support Bundle Enhancement [lep] Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

Tunable Timeout between Engine and Replica [doc] In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.

The built-in CSI Snapshotter sidecar is upgraded to v5.0.1. The v1beta1 version of Volume Snapshot custom resource is deprecated but still supported. However, it will be removed after upgrading CSI Snapshotter to 6.1 or later versions in the future, so please start using v1 version instead before the deprecated version is removed.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[FEATURE] Reclaim/Shrink space of volume (836) - @yangchiu @derekbit @smallteeths @shuo-wu

[FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher

[FEATURE] Online volume expansion (1674) - @shuo-wu @chriscchien

[FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz

[FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien

[FEATURE] Support metrics for Volume IOPS, throughput and latency real time (2406) - @derekbit @roger-ryao

[FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien

[FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit

[FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Faster resync time for fresh replica rebuilding (4092) - @yangchiu @derekbit

[FEATURE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao

[FEATURE] Update K8s version support and component/pkg/build dependencies (4239) - @yangchiu @PhanLe1010

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Adjust the iSCSI timeout and the engine-to-replica timeout settings (4491) - @yangchiu @derekbit

[IMPROVEMENT] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @roger-ryao

[IMPROVEMENT] Speed up replica rebuilding by the metadata such as ctime of snapshot disk files (4783) - @yangchiu @derekbit

Enhancements

[FEATURE] Configure successfulJobsHistoryLimit of CronJobs (1711) - @weizhe0422 @chriscchien

[FEATURE] Allow customization of the cipher used by cryptsetup in volume encryption (3353) - @mantissahz @chriscchien

[FEATURE] New setting to limit the concurrent volume restoring from backup (4558) - @c3y1huang @chriscchien

[FEATURE] Make FS format options configurable in storage class (4642) - @weizhe0422 @chriscchien

Improvement

[IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien

[IMPROVEMENT] Improve 'recovery from longhorn backup without system installed' doc. (1522) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Dump NFS ganesha logs to pod stdout (2380) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Reduce read and write calls in liblonghorn (tgt <-> engine) (4133) - @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[DOC] Update Helm readme and document (4175) - @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chriscchien

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @yangchiu @PhanLe1010

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chriscchien

[IMPROVEMENT] Show non-JSON error from backup store (4336) - @c3y1huang

[IMPROVEMENT] Update nfs-ganesha to v4.0 (4351) - @derekbit

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

[IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien

[IMPROVEMENT] Add prep work for fixing the corrupted filesystem using fsck in KB (4440) - @derekbit

[IMPROVEMENT] Prevent users from accidentally uninstalling Longhorn (4509) - @yangchiu @PhanLe1010

[IMPROVEMENT] add possibility to use nodeSelector on the storageClass (4574) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @yangchiu @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

[IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien

[IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit

[IMPROVEMENT] Longhorn system backup/restore UI (4855) - @smallteeths

[IMPROVEMENT] Replace the modTime (mtime) with ctime in snapshot hash (4934) - @derekbit @chriscchien

[BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien

[IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien

[IMPROVEMENT] Allow users to change volume.spec.snapshotDataIntegrity on UI (4994) - @yangchiu @smallteeths

[IMPROVEMENT] Backup and restore recurring jobs on UI (5009) - @smallteeths @chriscchien

[IMPROVEMENT] Disable Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly for RWX volumes (5017) - @derekbit @chriscchien

[IMPROVEMENT] Enable fast replica rebuilding by default (5023) - @derekbit @roger-ryao

[IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit

[IMPROVEMENT] UI show error when backup target is empty for system backup (5056) - @smallteeths @khushboo-rancher

[IMPROVEMENT] System restore job name should be Longhorn prefixed (5057) - @c3y1huang @khushboo-rancher

[BUG] Error in logs while restoring the system backup (5061) - @c3y1huang @chriscchien

[IMPROVEMENT] Add warning message to when deleting the restoring backups (5065) - @smallteeths @khushboo-rancher @roger-ryao

[IMPROVEMENT] Inconsistent name convention across volume backup restore and system backup restore (5066) - @smallteeths @roger-ryao

[IMPROVEMENT] System restore should proceed to restore other volumes if restoring one volume keeps failing for a certain time. (5086) - @c3y1huang @khushboo-rancher @roger-ryao

[IMPROVEMENT] Support customized number of replicas of webhook and recovery-backend (5087) - @derekbit @chriscchien

[IMPROVEMENT] Simplify the page by placing some configuration items in the advanced configuration when creating the volume (5090) - @yangchiu @smallteeths

[IMPROVEMENT] Support replica sync client timeout setting to stabilize replica rebuilding (5110) - @derekbit @chriscchien

[IMPROVEMENT] Set a newly created volume's data integrity from UI to ignored rather than Fast-Check. (5126) - @yangchiu @smallteeths

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[TASK] RWX volume performance measurement and investigation (3665) - @derekbit

[TASK] Verify spinning disk/HDD via the current e2e regression (4182) - @yangchiu

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

[TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

[BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao

[BUG] Every other backup fails and crashes the volume (Segmentation Fault) (1768) - @olljanat @mantissahz

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

[BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang

[TASK] Disable tcmalloc in longhorn-engine and longhorn-instance-manager (5068) - @derekbit

Bugs

[BUG] Removing old instance records after the new IM pod is launched will take 1 minute (1363) - @mantissahz

[BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien

[BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @roger-ryao

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chriscchien

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chriscchien

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @roger-ryao

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao

[BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao

[BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien

[BUG] A IM-proxy connection not closed in full regression 1.3 (4113) - @c3y1huang @chriscchien

[BUG] Scale replica warning (4120) - @c3y1huang @chriscchien

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @chriscchien

[BUG] longhorn-manager cannot start while upgrading if the configmap contains volume sensitive settings (4160) - @derekbit @chriscchien

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @chriscchien

[BUG] Unable to use a TTY error when running environment_check.sh (4216) - @flkdnt @chriscchien

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @chriscchien

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @yangchiu @PhanLe1010

[BUG] the IP address is duplicate when using storage network and the second network is contronllerd by ovs-cni. (4281) - @mantissahz

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chriscchien

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chriscchien

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @roger-ryao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Update the Longhorn document in Using the Environment Check Script (4450) - @weizhe0422 @roger-ryao

[BUG] Unable to search 1.3.1 doc by algolia (4457) - @mantissahz @roger-ryao

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Flaky case test_autosalvage_with_data_locality_enabled (4489) - @weizhe0422

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Inconsistent system snapshots between replicas after rebuilding (4513) - @derekbit

[BUG] Prometheus metric for backup state (longhorn_backup_state) returns wrong values (4521) - @mantissahz @roger-ryao

[BUG] Longhorn accidentally schedule all replicas onto a worker node even though the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @roger-ryao

[BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz @roger-ryao

[BUG] Fix the share-manager deletion failure if the confimap is not existing (4648) - @derekbit @roger-ryao

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit @chriscchien

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @roger-ryao

[BUG] rebuilding the replica failed after upgrading from 1.2.4 to 1.3.2-rc2 (4705) - @derekbit @chriscchien

[BUG] Cannot re-run helm uninstallation if the first one failed and cannot fetch logs of failed uninstallation pod (4711) - @yangchiu @PhanLe1010 @roger-ryao

[BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien

[BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao

[BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien

[BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien

[BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao

[BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien

[BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien

[BUG] Failed to build longhorn-instance-manager master branch (4946) - @derekbit

[BUG] PVC only works with plural annotation volumes.kubernetes.io/storage-provisioner: driver.longhorn.io (4951) - @weizhe0422

[BUG] Failed to create a replenished replica process because of the newly adding option (4962) - @yangchiu @derekbit

[BUG] Incorrect log messages in longhorn-engine processRemoveSnapshot() (4980) - @derekbit

[BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher

[BUG] System backup should validate empty backup target (5055) - @c3y1huang @khushboo-rancher

[BUG] missing the restoreVolumeRecurringJob parameter in the VolumeGet API (5062) - @mantissahz @roger-ryao

[BUG] System restore stuck in restoring if pvc exists with identical name (5064) - @c3y1huang @roger-ryao

[BUG] No error shown on UI if system backup conf not available (5072) - @c3y1huang @khushboo-rancher

[BUG] System restore missing services (5074) - @yangchiu @c3y1huang

[BUG] In a system restore, PV & PVC are not restored if PVC was created with 'longhorn-static' (created via Longhorn GUI) (5091) - @c3y1huang @khushboo-rancher

[BUG][v1.4.0-rc1] image security scan CRITICAL issues (5107) - @yangchiu @mantissahz

[BUG] Snapshot trim wrong label in the volume detail page. (5127) - @smallteeths @chriscchien

[BUG] Filesystem on the volume with a backing image is corrupted after applying trim operation (5129) - @derekbit @chriscchien

[BUG] Error in uninstall job (5132) - @c3y1huang @chriscchien

[BUG] Uninstall job unable to delete the systembackup and systemrestore cr. (5133) - @c3y1huang @chriscchien

[BUG] Nil pointer dereference error on restoring the system backup (5134) - @yangchiu @c3y1huang

[BUG] UI option Update Replicas Auto Balance should use capital letter like others (5154) - @smallteeths @chriscchien

[BUG] System restore cannot roll out when volume name is different to the PV (5157) - @yangchiu @c3y1huang

[BUG] Online expansion doesn't succeed after a failed expansion (5169) - @derekbit @shuo-wu @khushboo-rancher

Misc

[DOC] RWX support for NVIDIA JETSON Ubuntu 18.4LTS kernel requires enabling NFSV4.1 (3157) - @yangchiu @derekbit

[DOC] Add information about encryption algorithm to documentation (3285) - @mantissahz

[DOC] Update the doc of volume size after introducing snapshot prune (4158) - @shuo-wu

[Doc] Update the outdated "Customizing Default Settings" document (4174) - @derekbit

[TASK] Refresh distro version support for 1.4 (4401) - @weizhe0422

[TASK] Update official document Longhorn Networking (4478) - @derekbit

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @roger-ryao

[TASK] Disable doc search for archived versions < 1.1 (4524) - @mantissahz

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

[TASK] Update base image of all components from BCI 15.3 to 15.4 (4617) - @yangchiu

[DOC] Update the Longhorn document in Install with Helm (4745) - @roger-ryao

[TASK] Create longhornio support-bundle-kit image (4911) - @yangchiu

[DOC] Add Recurring * Jobs History Limit to setting reference (4912) - @weizhe0422 @roger-ryao

[DOC] Add Failed Backup TTL to setting reference (4913) - @mantissahz

[TASK] Create longhornio liveness probe image (4945) - @yangchiu

[TASK] Make system managed components branch-based build (5024) - @yangchiu

[TASK] Remove unstable s390x from PR check for all repos (5040) -

[TASK] Update longhorn-share-manager's nfs-ganesha to V4.2.1 (5083) - @derekbit @mantissahz

[DOC] Update the Longhorn document in Setting up Prometheus and Grafana (5158) - @roger-ryao

Contributors

@FrankYang0529

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@flkdnt

@innobead

@joshimoo

@khushboo-rancher

@mantissahz

@olljanat

@roger-ryao

@shuo-wu

@smallteeths

@w13915984028

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.4.0-rc3(Dec 28, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.0-rc3 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Kubernetes 1.25 Support [doc] In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

ARM64 GA ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

RWX GA [lep][doc] RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

Volume Snapshot Checksum [lep][doc] Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

Volume Bit-rot Protection [lep][doc] When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

Volume Replica Rebuilding Speedup When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

Volume Trim [lep][doc] Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

Online Volume Expansion Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

Local Volume via Data Locality Strict Mode [lep][doc] Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

Volume Recurring Job Backup Restore [lep][doc] Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

Volume IO Metrics [doc] Longhorn enriches Volume metrics by providing real-time IO stats including IOPS, latency, and throughput of R/W IO. Users can set up a monotoning solution like Prometheus to monitor volume performance.

Longhorn System Backup & Restore [lep][doc] Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

Support Bundle Enhancement [lep] Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

Tunable Timeout between Engine and Replica [doc] In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

The live upgrade from v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[FEATURE] Reclaim/Shrink space of volume (836) - @yangchiu @derekbit @smallteeths @shuo-wu

[FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher

[FEATURE] Online volume expansion (1674) - @shuo-wu

[FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz

[FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien

[FEATURE] Support metrics for Volume IOPS, throughput and latency real time (2406) - @derekbit @roger-ryao

[FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien

[FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit

[FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Faster resync time for fresh replica rebuilding (4092) - @yangchiu @derekbit

[FEAUTRE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao

[FEATURE] Align with upstream K8s version support from Longhorn 1.4 and update component/pkg/build dependencies (4239) - @yangchiu @PhanLe1010

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Adjust the iSCSI timeout and the engine-to-replica timeout settings (4491) - @yangchiu @derekbit

[IMPROVEMENT] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @roger-ryao

[IMPROVEMENT] Speed up replica rebuilding by the metadata such as ctime of snapshot disk files (4783) - @yangchiu @derekbit

Enhancements

[FEATURE] Configure successfulJobsHistoryLimit of CronJobs (1711) - @weizhe0422 @chriscchien

[FEATURE] Allow customization of the cipher used by cryptsetup in volume encryption (3353) - @mantissahz @chriscchien

[FEATURE] New setting to limit the concurrent volume restoring from backup (4558) - @c3y1huang @chriscchien

[FEATURE] Make FS format options configurable in storage class (4642) - @weizhe0422 @chriscchien

Improvement

[IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien

[IMPROVEMENT] Improve 'recovery from longhorn backup without system installed' doc. (1522) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Dump NFS ganesha logs to pod stdout (2380) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Reduce read and write calls in liblonghorn (tgt <-> engine) (4133) - @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[DOC] Update Helm readme and document (4175) - @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chriscchien

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @yangchiu @PhanLe1010

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chriscchien

[IMPROVEMENT] Show non-JSON error from backup store (4336) - @c3y1huang

[IMPROVEMENT] Update nfs-ganesha to v4.0 (4351) - @derekbit

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

[IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien

[IMPROVEMENT] Add prep work for fixing the corrupted filesystem using fsck in KB (4440) - @derekbit

[IMPROVEMENT] Prevent users from accidentally uninstalling Longhorn (4509) - @yangchiu @PhanLe1010

[IMPROVEMENT] add possibility to use nodeSelector on the storageClass (4574) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @yangchiu @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

[IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien

[IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit

[IMPROVEMENT] Longhorn system backup/restore UI (4855) - @smallteeths

[IMPROVEMENT] Replace the modTime (mtime) with ctime in snapshot hash (4934) - @derekbit @chriscchien

[BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien

[IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien

[IMPROVEMENT] Allow users to change volume.spec.snapshotDataIntegrity on UI (4994) - @yangchiu @smallteeths

[IMPROVEMENT] Backup and restore recurring jobs on UI (5009) - @smallteeths @chriscchien

[IMPROVEMENT] Disable Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly for RWX volumes (5017) - @derekbit @chriscchien

[IMPROVEMENT] Enable fast replica rebuilding by default (5023) - @derekbit @roger-ryao

[IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit

[IMPROVEMENT] UI show error when backup target is empty for system backup (5056) - @smallteeths @khushboo-rancher

[IMPROVEMENT] System restore job name should be Longhorn prefixed (5057) - @c3y1huang @khushboo-rancher

[BUG] Error in logs while restoring the system backup (5061) - @c3y1huang @chriscchien

[IMPROVEMENT] Add warning message to when deleting the restoring backups (5065) - @smallteeths @khushboo-rancher @roger-ryao

[IMPROVEMENT] Inconsistent name convention across volume backup restore and system backup restore (5066) - @smallteeths @roger-ryao

[IMPROVEMENT] System restore should proceed to restore other volumes if restoring one volume keeps failing for a certain time. (5086) - @c3y1huang @khushboo-rancher @roger-ryao

[IMPROVEMENT] Support customized number of replicas of webhook and recovery-backend (5087) - @derekbit @chriscchien

[IMPROVEMENT] Simplify the page by placing some configuration items in the advanced configuration when creating the volume (5090) - @yangchiu @smallteeths

[IMPROVEMENT] Support replica sync client timeout setting to stabilize replica rebuilding (5110) - @derekbit @chriscchien

[IMPROVEMENT] Set a newly created volume's data integrity from UI to ignored rather than Fast-Check. (5126) - @yangchiu @smallteeths

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[TASK] RWX volume performance measurement and investigation (3665) - @derekbit

[TASK] Verify spinning disk/HDD via the current e2e regression (4182) - @yangchiu

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

[TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

[BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao

[BUG] Every other backup fails and crashes the volume (Segmentation Fault) (1768) - @olljanat @mantissahz

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

[BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang

[TASK] Disable tcmalloc in longhorn-engine and longhorn-instance-manager (5068) - @derekbit

Bugs

[BUG] Removing old instance records after the new IM pod is launched will take 1 minute (1363) - @mantissahz

[BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien

[BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @roger-ryao

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chriscchien

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chriscchien

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @roger-ryao

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao

[BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao

[BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien

[BUG] A IM-proxy connection not closed in full regression 1.3 (4113) - @c3y1huang @chriscchien

[BUG] Scale replica warning (4120) - @c3y1huang @chriscchien

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @chriscchien

[BUG] longhorn-manager cannot start while upgrading if the configmap contains volume sensitive settings (4160) - @derekbit @chriscchien

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @chriscchien

[BUG] Unable to use a TTY error when running environment_check.sh (4216) - @flkdnt @chriscchien

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @chriscchien

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @yangchiu @PhanLe1010

[BUG] the IP address is duplicate when using storage network and the second network is contronllerd by ovs-cni. (4281) - @mantissahz

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chriscchien

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chriscchien

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @roger-ryao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Update the Longhorn document in Using the Environment Check Script (4450) - @weizhe0422 @roger-ryao

[BUG] Unable to search 1.3.1 doc by algolia (4457) - @mantissahz @roger-ryao

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Flaky case test_autosalvage_with_data_locality_enabled (4489) - @weizhe0422

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Inconsistent system snapshots between replicas after rebuilding (4513) - @derekbit

[BUG] Prometheus metric for backup state (longhorn_backup_state) returns wrong values (4521) - @mantissahz @roger-ryao

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @roger-ryao

[BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz @roger-ryao

[BUG] Fix the share-manager deletion failure if the confimap is not existing (4648) - @derekbit @roger-ryao

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit @chriscchien

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @roger-ryao

[BUG] rebuilding the replica failed after upgrading from 1.2.4 to 1.3.2-rc2 (4705) - @derekbit @chriscchien

[BUG] Cannot re-run helm uninstallation if the first one failed and cannot fetch logs of failed uninstallation pod (4711) - @yangchiu @PhanLe1010 @roger-ryao

[BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien

[BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao

[BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien

[BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien

[BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao

[BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien

[BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien

[BUG] Failed to build longhorn-instance-manager master branch (4946) - @derekbit

[BUG] PVC only works with plural annotation volumes.kubernetes.io/storage-provisioner: driver.longhorn.io (4951) - @weizhe0422

[BUG] Failed to create a replenished replica process because of the newly adding option (4962) - @yangchiu @derekbit

[BUG] Incorrect log messages in longhorn-engine processRemoveSnapshot() (4980) - @derekbit

[BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher

[BUG] System backup should validate empty backup target (5055) - @c3y1huang @khushboo-rancher

[BUG] missing the restoreVolumeRecurringJob parameter in the VolumeGet API (5062) - @mantissahz @roger-ryao

[BUG] System restore stuck in restoring if pvc exists with identical name (5064) - @c3y1huang @roger-ryao

[BUG] No error shown on UI if system backup conf not available (5072) - @c3y1huang @khushboo-rancher

[BUG] System restore missing services (5074) - @yangchiu @c3y1huang

[BUG] In a system restore, PV & PVC are not restored if PVC was created with 'longhorn-static' (created via Longhorn GUI) (5091) - @c3y1huang @khushboo-rancher

[BUG][v1.4.0-rc1] image security scan CRITICAL issues (5107) - @yangchiu @mantissahz

[BUG] Snapshot trim wrong label in the volume detail page. (5127) - @smallteeths @chriscchien

[BUG] Filesystem on the volume with a backing image is corrupted after applying trim operation (5129) - @derekbit @chriscchien

[BUG] Error in uninstall job (5132) - @c3y1huang @chriscchien

[BUG] Uninstall job unable to delete the systembackup and systemrestore cr. (5133) - @c3y1huang @chriscchien

[BUG] Nil pointer dereference error on restoring the system backup (5134) - @yangchiu @c3y1huang @khushboo-rancher

[BUG] UI option Update Replicas Auto Balance should use capital letter like others (5154) - @smallteeths @chriscchien

Misc

[DOC] RWX support for NVIDIA JETSON Ubuntu 18.4LTS kernel requires enabling NFSV4.1 (3157) - @yangchiu @derekbit

[DOC] Add information about encryption algorithm to documentation (3285) - @mantissahz

[DOC] Update the doc of volume size after introducing snapshot prune (4158) - @shuo-wu

[Doc] Update the outdated "Customizing Default Settings" document (4174) - @derekbit

[TASK] Refresh distro version support for 1.4 (4401) - @weizhe0422

[TASK] Update official document Longhorn Networking (4478) - @derekbit

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @roger-ryao

[TASK] Disable doc search for archived versions < 1.1 (4524) - @mantissahz

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

[TASK] Update base image of all components from BCI 15.3 to 15.4 (4617) - @yangchiu

[DOC] Update the Longhorn document in Install with Helm (4745) - @roger-ryao

[TASK] Create longhornio support-bundle-kit image (4911) - @yangchiu

[DOC] Add Recurring * Jobs History Limit to setting referenence (4912) - @weizhe0422 @roger-ryao

[DOC] Add Failed Backup TTL to setting referenence (4913) - @mantissahz

[TASK] Create longhornio liveness probe image (4945) - @yangchiu

[TASK] Make system managed components branch-based build (5024) - @yangchiu

[TASK] Remove unstable s390x from PR check for all repos (5040) -

[TASK] Update longhorn-share-manager's nfs-ganesha to V4.2.1 (5083) - @derekbit @mantissahz

Contributors

@FrankYang0529

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@flkdnt

@innobead

@joshimoo

@khushboo-rancher

@mantissahz

@olljanat

@roger-ryao

@shuo-wu

@smallteeths

@w13915984028

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.4.0-rc2(Dec 21, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.0-rc2 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Kubernetes 1.25 Support [doc] In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

ARM64 GA ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

RWX GA [lep][doc] RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

Volume Snapshot Checksum [lep][doc] Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

Volume Bit-rot Protection [lep][doc] When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

Volume Replica Rebuilding Speedup When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

Volume Trim [lep][doc] Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

Online Volume Expansion Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

Local Volume via Data Locality Strict Mode [lep][doc] Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

Volume Recurring Job Backup Restore [lep][doc] Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

Volume IO Metrics Longhorn enriches Volume metrics by providing real-time IO stats including IOPS, latency, and throughput of R/W IO. Users can set up a monotoning solution like Prometheus to monitor volume performance.

Longhorn System Backup & Restore [lep] Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

Support Bundle Enhancement [lep] Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

Tunable Timeout between Engine and Replica [doc] In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

The live upgrade from v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[FEATURE] Reclaim/Shrink space of volume (836) - @smallteeths @shuo-wu @roger-ryao

[FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher

[FEATURE] Online volume expansion (1674) - @shuo-wu

[FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz

[FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien

[FEATURE] Support metrics for IOPS and bandwidth real time (2406) - @derekbit

[FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien

[FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit

[FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Faster resync time for fresh replica rebuilding (4092) - @derekbit

[FEAUTRE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao

[FEATURE] Align with upstream K8s version support from Longhorn 1.4 and update component/pkg/build dependencies (4239) - @yangchiu @PhanLe1010

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Adjust the iSCSI timeout and the engine-to-replica timeout settings (4491) - @yangchiu @derekbit

[IMPROVEMENT] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @roger-ryao

[IMPROVEMENT] Speed up replica rebuilding by the metadata such as ctime of snapshot disk files (4783) - @yangchiu @derekbit

Enhancements

[FEATURE] Configure successfulJobsHistoryLimit of CronJobs (1711) - @weizhe0422 @chriscchien

[FEATURE] Allow customization of the cipher used by cryptsetup in volume encryption (3353) - @mantissahz @chriscchien

[FEATURE] New setting to limit the concurrent volume restoring from backup (4558) - @c3y1huang @chriscchien

[FEATURE] Make FS format options configurable in storage class (4642) - @weizhe0422 @chriscchien

Improvement

[IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien

[IMPROVEMENT] Improve 'recovery from longhorn backup without system installed' doc. (1522) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Dump NFS ganesha logs to pod stdout (2380) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Reduce read and write calls in liblonghorn (tgt <-> engine) (4133) - @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[DOC] Update Helm readme and document (4175) - @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chriscchien

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @yangchiu @PhanLe1010

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chriscchien

[IMPROVEMENT] Show non-JSON error from backup store (4336) - @c3y1huang

[IMPROVEMENT] Update nfs-ganesha to v4.0 (4351) - @derekbit

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

[IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien

[IMPROVEMENT] Add prep work for fixing the corrupted filesystem using fsck in KB (4440) - @derekbit

[IMPROVEMENT] Prevent users from accidentally uninstalling Longhorn (4509) - @yangchiu @PhanLe1010

[IMPROVEMENT] add possibility to use nodeSelector on the storageClass (4574) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @yangchiu @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

[IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien

[IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit

[IMPROVEMENT] Longhorn system backup/restore UI (4855) - @smallteeths

[IMPROVEMENT] Replace the modTime (mtime) with ctime in snapshot hash (4934) - @derekbit @chriscchien

[BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien

[IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien

[IMPROVEMENT] Allow users to change volume.spec.snapshotDataIntegrity on UI (4994) - @yangchiu @smallteeths

[IMPROVEMENT] Backup and restore recurring jobs on UI (5009) - @smallteeths @chriscchien

[IMPROVEMENT] Disable Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly for RWX volumes (5017) - @derekbit @chriscchien

[IMPROVEMENT] Enable fast replica rebuilding by default (5023) - @derekbit @roger-ryao

[IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit

[IMPROVEMENT] UI show error when backup target is empty for system backup (5056) - @smallteeths @khushboo-rancher

[IMPROVEMENT] System restore job name should be Longhorn prefixed (5057) - @c3y1huang @khushboo-rancher

[BUG] Error in logs while restoring the system backup (5061) - @c3y1huang @chriscchien

[IMPROVEMENT] Add warning message to when deleting the restoring backups (5065) - @smallteeths @khushboo-rancher @roger-ryao

[IMPROVEMENT] System restore should proceed to restore other volumes if restoring one volume keeps failing for a certain time. (5086) - @c3y1huang @khushboo-rancher

[IMPROVEMENT] Support customized number of replicas of webhook and recovery-backend (5087) - @derekbit @chriscchien

[IMPROVEMENT] Simplify the page by placing some configuration items in the advanced configuration when creating the volume (5090) - @yangchiu @smallteeths

[IMPROVEMENT] Support replica sync client timeout setting to stabilize replica rebuilding (5110) - @derekbit

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[TASK] RWX volume performance measurement and investigation (3665) - @derekbit

[TASK] Verify spinning disk/HDD via the current e2e regression (4182) - @yangchiu

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

[TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

[BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao

[BUG] Every other backup fails and crashes the volume (Segmentation Fault) (1768) - @olljanat @mantissahz

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

[BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang

[TASK] Disable tcmalloc in longhorn-engine and longhorn-instance-manager (5068) - @derekbit

Bugs

[BUG] Removing old instance records after the new IM pod is launched will take 1 minute (1363) - @mantissahz

[BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien

[BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @roger-ryao

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chriscchien

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chriscchien

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @roger-ryao

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao

[BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao

[BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien

[BUG] A IM-proxy connection not closed in full regression 1.3 (4113) - @c3y1huang @chriscchien

[BUG] Scale replica warning (4120) - @c3y1huang @chriscchien

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @chriscchien

[BUG] longhorn-manager cannot start while upgrading if the configmap contains volume sensitive settings (4160) - @derekbit @chriscchien

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @chriscchien

[BUG] Unable to use a TTY error when running environment_check.sh (4216) - @flkdnt @chriscchien

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @chriscchien

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @yangchiu @PhanLe1010

[BUG] the IP address is duplicate when using storage network and the second network is contronllerd by ovs-cni. (4281) - @mantissahz

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chriscchien

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chriscchien

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @roger-ryao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Update the Longhorn document in Using the Environment Check Script (4450) - @weizhe0422 @roger-ryao

[BUG] Unable to search 1.3.1 doc by algolia (4457) - @mantissahz @roger-ryao

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Flaky case test_autosalvage_with_data_locality_enabled (4489) - @weizhe0422

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Inconsistent system snapshots between replicas after rebuilding (4513) - @derekbit

[BUG] Prometheus metric for backup state (longhorn_backup_state) returns wrong values (4521) - @mantissahz @roger-ryao

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @roger-ryao

[BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz @roger-ryao

[BUG] Fix the share-manager deletion failure if the confimap is not existing (4648) - @derekbit @roger-ryao

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit @chriscchien

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @roger-ryao

[BUG] rebuilding the replica failed after upgrading from 1.2.4 to 1.3.2-rc2 (4705) - @derekbit @chriscchien

[BUG] Cannot re-run helm uninstallation if the first one failed and cannot fetch logs of failed uninstallation pod (4711) - @yangchiu @PhanLe1010 @roger-ryao

[BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien

[BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao

[BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien

[BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien

[BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao

[BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien

[BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien

[BUG] Failed to build longhorn-instance-manager master branch (4946) - @derekbit

[BUG] PVC only works with plural annotation volumes.kubernetes.io/storage-provisioner: driver.longhorn.io (4951) - @weizhe0422

[BUG] Failed to create a replenished replica process because of the newly adding option (4962) - @yangchiu @derekbit

[BUG] Incorrect log messages in longhorn-engine processRemoveSnapshot() (4980) - @derekbit

[BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher

[BUG] System backup should validate empty backup target (5055) - @c3y1huang @khushboo-rancher

[BUG] missing the restoreVolumeRecurringJob parameter in the VolumeGet API (5062) - @mantissahz @roger-ryao

[BUG] System restore stuck in restoring if pvc exists with identical name (5064) - @c3y1huang @roger-ryao

[BUG] No error shown on UI if system backup conf not available (5072) - @c3y1huang @khushboo-rancher

[BUG] System restore missing services (5074) - @yangchiu @c3y1huang

[BUG] In a system restore, PV & PVC are not restored if PVC was created with 'longhorn-static' (created via Longhorn GUI) (5091) - @c3y1huang @khushboo-rancher

[BUG][v1.4.0-rc1] image security scan CRITICAL issues (5107) - @yangchiu @mantissahz

Misc

[DOC] Add information about encryption algorithm to documentation (3285) - @mantissahz

[DOC] Update the doc of volume size after introducing snapshot prune (4158) - @shuo-wu

[Doc] Update the outdated "Customizing Default Settings" document (4174) - @derekbit

[TASK] Refresh distro version support for 1.4 (4401) - @weizhe0422

[TASK] Update official document Longhorn Networking (4478) - @derekbit

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @roger-ryao

[TASK] Disable doc search for archived versions < 1.1 (4524) - @mantissahz

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

[TASK] Update base image of all components from BCI 15.3 to 15.4 (4617) - @yangchiu

[DOC] Update the Longhorn document in Install with Helm (4745) - @roger-ryao

[TASK] Create longhornio support-bundle-kit image (4911) - @yangchiu

[DOC] Add Recurring * Jobs History Limit to setting referenence (4912) - @weizhe0422 @roger-ryao

[DOC] Add Failed Backup TTL to setting referenence (4913) - @mantissahz

[TASK] Create longhornio liveness probe image (4945) - @yangchiu

[TASK] Make system managed components branch-based build (5024) - @yangchiu

[TASK] Remove unstable s390x from PR check for all repos (5040) -

[TASK] Update longhorn-share-manager's nfs-ganesha to V4.2.1 (5083) - @derekbit @mantissahz

Contributors

@FrankYang0529

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@flkdnt

@innobead

@joshimoo

@khushboo-rancher

@mantissahz

@olljanat

@roger-ryao

@shuo-wu

@smallteeths

@w13915984028

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.4.0-rc1(Dec 13, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.0-rc1 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Kubernetes 1.25 Support [doc] In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

ARM64 GA ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

RWX GA [lep][doc] RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

Volume Snapshot Checksum [lep][doc] Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

Volume Bit-rot Protection [lep][doc] When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

Volume Replica Rebuilding Speedup When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

Volume Trim [lep][doc] Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

Online Volume Expansion Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

Block Volume Encryption Besides supporting FS Volume Encryption in current versions, Longhorn supports Block Volume Encryption via CSI node driver.

Local Volume via Data Locality Strict Mode [lep][doc] Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

Volume Recurring Job Backup Restore [lep][doc] Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

Longhorn System Backup & Restore [lep] Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

Support Bundle Enhancement [lep] Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

Tunable Timeout between Engine and Replica [doc] In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

The live upgrade from v1.3.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[FEATURE] Reclaim/Shrink space of volume (836) - @smallteeths @shuo-wu @roger-ryao

[FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher

[FEATURE] Online volume expansion (1674) - @shuo-wu

[FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz

[FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien

[FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien

[FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit

[FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien

[IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien

[FEAUTRE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao

[FEATURE] Align with upstream K8s version support from Longhorn 1.4 and update component/pkg/build dependencies (4239) - @yangchiu @PhanLe1010

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Adjust the iSCSI timeout and the engine-to-replica timeout settings (4491) - @yangchiu @derekbit

[Improvement] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @roger-ryao

[IMPROVEMENT] Speed up replica rebuilding by the metadata such as ctime of snapshot disk files (4783) - @yangchiu @derekbit

[FEATURE] Support Encryption for VolumeMode Block (4883) - @joshimoo

Enhancements

[FEATURE] Configure successfulJobsHistoryLimit of CronJobs (1711) - @weizhe0422 @chriscchien

[FEATURE] Allow customization of the cipher used by cryptsetup in volume encryption (3353) - @mantissahz @chriscchien

[FEATURE] New setting to limit the concurrent volume restoring from backup (4558) - @c3y1huang @chriscchien

[FEATURE] Make FS format options configurable in storage class (4642) - @weizhe0422 @chriscchien

Improvement

[IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien

[IMPROVEMENT] Improve 'recovery from longhorn backup without system installed' doc. (1522) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Do not count the failure replica reuse failure caused by the disconnection (1923) - @mantissahz

[IMPROVEMENT] Dump NFS ganesha logs to pod stdout (2380) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chriscchien

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Reduce read and write calls in liblonghorn (tgt <-> engine) (4133) - @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[DOC] Update Helm readme and document (4175) - @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chriscchien

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @yangchiu @PhanLe1010

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chriscchien

[IMPROVEMENT] Show non-JSON error from backup store (4336) - @c3y1huang

[IMPROVEMENT] Update nfs-ganesha to v4.0 (4351) - @derekbit

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

[IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien

[IMPROVEMENT] Add prep work for fixing the corrupted filesystem using fsck in KB (4440) - @derekbit

[IMPROVEMENT] Prevent users from accidentally uninstalling Longhorn (4509) - @yangchiu @PhanLe1010

[IMPROVEMENT] add possibility to use nodeSelector on the storageClass (4574) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @roger-ryao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @yangchiu @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

[IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien

[IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit

[IMPROVEMENT] Longhorn system backup/restore UI (4855) - @smallteeths

[IMPROVEMENT] Replace the modTime (mtime) with ctime in snapshot hash (4934) - @derekbit @chriscchien

[BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien

[IMPROVEMENT] Allow users to change volume.spec.snapshotDataIntegrity on UI (4994) - @smallteeths

[IMPROVEMENT] Backup and restore recurring jobs on UI (5009) - @smallteeths

[IMPROVEMENT] Disable Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly for RWX volumes (5017) - @derekbit @chriscchien

[IMPROVEMENT] Enable fast replica rebuilding by default (5023) - @derekbit @roger-ryao

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[TASK] RWX volume performance measurement and investigation (3665) - @derekbit

[TASK] Verify spinning disk/HDD via the current e2e regression (4182) - @yangchiu

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

Stability

[BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao

[BUG] Every other backup fails and crashes the volume (Segmentation Fault) (1768) - @mantissahz

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

[BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang

Bugs

[BUG] Removing old instance records after the new IM pod is launched will take 1 minute (1363) - @mantissahz

[BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien

[BUG] A stopped replica on a removed node should not be counted as a healthy replica for the drain setting (2237) - @weizhe0422

[BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @roger-ryao

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chriscchien

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chriscchien

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @roger-ryao

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao

[BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao

[BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien

[BUG] A IM-proxy connection not closed in full regression 1.3 (4113) - @c3y1huang @chriscchien

[BUG] Scale replica warning (4120) - @c3y1huang @chriscchien

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @chriscchien

[BUG] longhorn-manager cannot start while upgrading if the configmap contains volume sensitive settings (4160) - @derekbit @chriscchien

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @chriscchien

[BUG] Unable to use a TTY error when running environment_check.sh (4216) - @flkdnt @chriscchien

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @chriscchien

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @yangchiu @PhanLe1010

[BUG] the IP address is duplicate when using storage network and the second network is contronllerd by ovs-cni. (4281) - @mantissahz

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chriscchien

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chriscchien

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @roger-ryao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Update the Longhorn document in Using the Environment Check Script (4450) - @weizhe0422 @roger-ryao

[BUG] Unable to search 1.3.1 doc by algolia (4457) - @mantissahz @roger-ryao

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Flaky case test_autosalvage_with_data_locality_enabled (4489) - @weizhe0422

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Inconsistent system snapshots between replicas after rebuilding (4513) - @derekbit

[BUG] Prometheus metric for backup state (longhorn_backup_state) returns wrong values (4521) - @mantissahz @roger-ryao

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @roger-ryao

[BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz @roger-ryao

[BUG] Fix the share-manager deletion failure if the confimap is not existing (4648) - @derekbit @roger-ryao

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit @chriscchien

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @roger-ryao

[BUG] rebuilding the replica failed after upgrading from 1.2.4 to 1.3.2-rc2 (4705) - @derekbit @chriscchien

[BUG] Cannot re-run helm uninstallation if the first one failed and cannot fetch logs of failed uninstallation pod (4711) - @yangchiu @PhanLe1010 @roger-ryao

[BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien

[BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao

[BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien

[BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @khushboo-rancher

[BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao

[BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien

[BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien

[BUG] Failed to build longhorn-instance-manager master branch (4946) - @derekbit

[BUG] Failed to create a replenished replica process because of the newly adding option (4962) - @yangchiu @derekbit

[BUG] Incorrect log messages in longhorn-engine processRemoveSnapshot() (4980) - @derekbit

[BUG] System backup showing wrong age (5047) - @c3y1huang

Misc

[TASK] Remove deprecated volume spec recurringJobs and storageClass recurringJobs field (2865) - @weizhe0422

[DOC] Add information about encryption algorithm to documentation (3285) - @mantissahz

[DOC] Update the doc of volume size after introducing snapshot prune (4158) - @shuo-wu

[Doc] Update the outdated "Customizing Default Settings" document (4174) - @derekbit

[TASK] Refresh distro version support for 1.4 (4401) - @weizhe0422

[TASK] Update official document Longhorn Networking (4478) - @derekbit

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @roger-ryao

[TASK] Disable doc search for archived versions < 1.1 (4524) - @mantissahz

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

[TASK] Update base image of all components from BCI 15.3 to 15.4 (4617) - @yangchiu

[DOC] Update the Longhorn document in Install with Helm (4745) - @roger-ryao

[TASK] Create longhornio support-bundle-kit image (4911) - @yangchiu

[DOC] Add Recurring * Jobs History Limit to setting referenence (4912) - @weizhe0422 @roger-ryao

[DOC] Add Failed Backup TTL to setting referenence (4913) - @mantissahz

[TASK] Create longhornio liveness probe image (4945) - @yangchiu

[TASK] Remove a deprecated feature disable-replica-rebuild from longhorn-manager (4997) - @weizhe0422

[TASK] Make system managed components branch-based build (5024) - @yangchiu

[TASK] Remove unstable s390x from PR check for all repos (5040) -

Contributors

@FrankYang0529

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@flkdnt

@innobead

@joshimoo

@khushboo-rancher

@mantissahz

@roger-ryao

@shuo-wu

@smallteeths

@w13915984028

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.2.6(Nov 4, 2022)
Release Note

v1.2.6 released! 🎆

This release introduces improvements and bug fixes as described below including stability, data correction, performance and so on.

There are two important fixes that will prevent rarely potential data corruption during replica rebuilding, and also improve write performance via the specific filesystem block size.

4354: Introduce Data Alignment Correction for existing volumes if the filesystem block size is less than 4096.

4594: Use the specific block size for the filesystem to avoid unnecessary Ready-Modify-Write operations between volume head and snapshots.

Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Note Please ensure the Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.2.6 from v1.1.x or v1.2.x. Only support upgrading from v1.1.x and v1.2.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[Improvement] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @roger-ryao

Improvement

[IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @roger-ryao

Bugs

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chriscchien

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @roger-ryao

[BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @roger-ryao

[BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao

[BUG] Unable to reuse existing failed replica causes test case test_allow_volume_creation_with_degraded_availability_restore failed (4791) - @yangchiu @mantissahz

Misc

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

Contributors

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@innobead

@mantissahz

@roger-ryao

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.2.6-rc1(Oct 20, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.2.6-rc1 released! 🎆

This release introduces improvements and bug fixes as described below including stability, performance, availability, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.1.* or v1.2.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[Improvement] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @Lichiehyao

Improvement

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @Lichiehyao

Bugs

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @Lichiehyao

[BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @Lichiehyao

[BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz

Misc

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@innobead

@mantissahz

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.2(Oct 14, 2022)
Release Note

v1.3.2 released! 🎆

This release introduces improvements and bug fixes as described below including stability, data correction, performance and so on.

There are two important fixes that will prevent rarely potential data corruption during replica rebuilding, and also improve write performance via the specific filesystem block size.

4354: Introduce Data Alignment Correction for existing volumes if the filesystem block size is less than 4096.

4594: Use the specific block size for the filesystem to avoid unnecessary Ready-Modify-Write operations between volume head and snapshots.

Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Note Please ensure the Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.3.2 from v1.2.x or v1.3.x. Only support upgrading from v1.2.x and v1.3.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[Improvement] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @Lichiehyao

Improvement

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @Lichiehyao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @yangchiu @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

Bugs

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @Lichiehyao

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @Lichiehyao

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @Lichiehyao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @Lichiehyao

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz @Lichiehyao

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit @chriscchien

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @Lichiehyao

[BUG] rebuilding the replica failed after upgrading from 1.2.4 to 1.3.2-rc2 (4705) - @derekbit @chriscchien

Misc

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @Lichiehyao

[TASK] Add K8s version upper limitation (< 1.25) to < 1.4 Longhorn chart (4525) - @weizhe0422 @chriscchien

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@innobead

@mantissahz

@smallteeths

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.2-rc2(Oct 11, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.2-rc2 released! 🎆

This release introduces improvements, and bug fixes as described below including stability, performance, availability, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* or v1.3.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[Improvement] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @Lichiehyao

Improvement

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @Lichiehyao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @yangchiu @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

Bugs

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @Lichiehyao

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @Lichiehyao

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @Lichiehyao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @yangchiu @mantissahz

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @Lichiehyao

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz @Lichiehyao

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit @chriscchien

[BUG] charts/longhorn/questions.yaml include oudated csi-image tags (4669) - @PhanLe1010 @Lichiehyao

Misc

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @Lichiehyao

[TASK] Add K8s version upper limitation (< 1.25) to < 1.4 Longhorn chart (4525) - @weizhe0422 @chriscchien

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@innobead

@mantissahz

@smallteeths

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.2-rc1(Oct 3, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.2-rc1 released! 🎆

This release introduces improvements, and bug fixes as described below including data consistency, stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* or v1.3.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien

[Improvement] Using specific block size in Longhorn volume's filesystem (4594) - @derekbit @Lichiehyao

Improvement

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chriscchien

[IMPROVEMENT] Check if node schedulable condition is set before trying to read it (4581) - @weizhe0422 @Lichiehyao

[IMPROVEMENT] Review/consolidate the sectorSize in replica server, replica volume, and engine (4599) - @derekbit

[IMPROVEMENT] Reorganize longhorn-manager/k8s/patches and auto-generate preserveUnknownFields field (4600) - @yangchiu @derekbit

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

Bugs

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @Lichiehyao

[BUG] The default access mode of a restored RWX volume is RWO (3444) - @weizhe0422 @Lichiehyao

[BUG] Backend sizes do not match 5368709120 != 10737418240 in the engine initiation phase (3601) - @derekbit @chriscchien

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @Lichiehyao

[BUG] can't upgrade engine if a volume was created in Longhorn v1.0 and the volume.spec.dataLocality is "" (4412) - @derekbit @chriscchien

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable (4502) - @yangchiu @c3y1huang

[BUG] Longhorn accidentally schedule all replicas onto a worker node eventhough the setting Replica Node Level Soft Anti-Affinity is currently disabled (4546) - @mantissahz

[BUG] the values.yaml in the longhorn helm chart contains values not used. (4601) - @weizhe0422 @Lichiehyao

[BUG] Can not pull a backup created by another Longhorn system from the remote backup target (4637) - @yangchiu @mantissahz

[BUG] Updating volume-scheduling-error failure for RWX volumes and expanding volumes (4654) - @derekbit

Misc

[TEST] Add automation test case for orphaned backup cleanup (4481) - @mantissahz @chriscchien

[TASK] Update preserveUnknownFields fields in longhorn-manager CRD manifest (4505) - @derekbit @Lichiehyao

[TASK] Update longhorn components with the latest backupstore (4552) - @derekbit

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chriscchien

@derekbit

@innobead

@mantissahz

@smallteeths

@weizhe0422

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.2.5(Sep 2, 2022)
Release Note

v1.2.5 released! 🎆

This release introduces improvements and bug fixes as described below including stability, performance, availability, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Note: Please ensure your Kubernetes cluster is at least v1.18 before upgrading to Longhorn v1.2.5 from v1.1.x or v1.2.x. Only support upgrading from v1.1.x and v1.2.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

Improvement

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @chrisc1981

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @chrisc1981

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @chrisc1981

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chrisc1981

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chrisc1981

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chrisc1981

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chrisc1981

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

Bugs

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @Lichiehyao

[BUG] fatal error: concurrent map writes (3077) - @derekbit @chrisc1981

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chrisc1981

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chrisc1981

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @chrisc1981

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @chrisc1981

[BUG] Scale replica warning (4120) - @c3y1huang @chrisc1981

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chrisc1981

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chrisc1981

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chrisc1981

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @Lichiehyao

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chrisc1981

@derekbit

@ibrokethecloud

@innobead

@jenting

@kaxing

@keithalucas

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@w13915984028

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.2.5-rc2(Aug 29, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.2.5-rc2 released! 🎆

This release introduces improvements, and bug fixes as described below including stability, performance, availability, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.1.* or v1.2.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Highlights

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

Improvement

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @chrisc1981

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @chrisc1981

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @chrisc1981

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz @chrisc1981

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chrisc1981

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chrisc1981

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chrisc1981

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @yangchiu @derekbit

Bugs

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @Lichiehyao

[BUG] fatal error: concurrent map writes (3077) - @derekbit @chrisc1981

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chrisc1981

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chrisc1981

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @chrisc1981

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @chrisc1981

[BUG] Scale replica warning (4120) - @c3y1huang @chrisc1981

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chrisc1981

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chrisc1981

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chrisc1981

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @Lichiehyao

[BUG] Confusing description the label for replica delition (4430) - @yangchiu @smallteeths

[BUG] Misleading message "The volume is in expansion progress from size 20Gi to 10Gi" if the expansion is invalid (4475) - @yangchiu @smallteeths

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chrisc1981

@derekbit

@ibrokethecloud

@innobead

@jenting

@kaxing

@keithalucas

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@w13915984028

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.2.5-rc1(Aug 19, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.2.5-rc1 released! 🎆

This release introduces improvements, and bug fixes as described below including stability, performance, availability, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.1.* or v1.2.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Highlights

[IMPROVEMENT] Improve rebuilding is canceled if it takes longer than 24 hours (2765) - @PhanLe1010

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

Improvement

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @chrisc1981

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @chrisc1981

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Periodically clean up all snapshots for a volume (3836) - @shuo-wu

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @chrisc1981

[IMPROVEMENT] Support failed/obsolete orphaned backup cleanup (3898) - @mantissahz

[IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @chrisc1981

[IMPROVEMENT] Migrate PodSecurityPolicy to support K8s 1.25 (4003) - @PhanLe1010

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Reduce read and write calls in liblonghorn (tgt <-> engine) (4133) - @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @chrisc1981

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @chrisc1981

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @derekbit

Bugs

[BUG] Restore from backup sometimes failed if having high frequent recurring backup job w/ retention (3055) - @mantissahz @Lichiehyao

[BUG] fatal error: concurrent map writes (3077) - @derekbit @chrisc1981

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @chrisc1981

[Bug] Degraded volume generate failed replica make volume unschedulable (3220) - @derekbit @chrisc1981

[BUG] Backup error happen to volume with older engine image (3621) -

[BUG] When Harvester delete an uploading image, the corresponding LH temp file is not deleted (3682) - @shuo-wu

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @derekbit

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @chrisc1981

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @chrisc1981

[BUG] Can't delete volumesnapshot if backup removed first (4107) - @PhanLe1010

[BUG] Scale replica warning (4120) - @c3y1huang @chrisc1981

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @chrisc1981

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @chrisc1981

[BUG] tests related to ha_salvage_test failed & flaky (4381) - @chrisc1981

[BUG] test_all_replica_restore_failure failed in v1.3.1-rc2 (4383) - @shuo-wu @chrisc1981

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @chrisc1981

[BUG] Error/warning during uninstallation of Longhorn v1.3.1 via manifest (4405) - @PhanLe1010 @Lichiehyao

[BUG] Confusing description the label for replica delition (4430) - @smallteeths

Misc

[TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @chrisc1981

Contributors

@Lichiehyao

@PhanLe1010

@c3y1huang

@chrisc1981

@derekbit

@ibrokethecloud

@innobead

@jenting

@kaxing

@keithalucas

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@w13915984028

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.1(Aug 11, 2022)
Release Note

v1.3.1 released! 🎆

This release introduces improvements and bug fixes as described below including stability, performance, availability, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Note: Please ensure your Kubernetes cluster is at least v1.18 before upgrading to Longhorn v1.3.1 from v1.3.0 or v1.2.x. Only support upgrading from v1.3.0 and v1.2.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Improvement

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @cchien816

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @cchien816

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @cchien816

[IMPROVEMENT] Show non-JSON error from backup store (4336) - @c3y1huang

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

[BUG] test_csi_snapshot_snap_create_volume_from_snapshot failed when using HDD as Longhorn disks (4227) - @yangchiu @PhanLe1010

Bugs

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @cchien816

[BUG] A IM-proxy connection not closed in full regression 1.3 (4113) - @c3y1huang @cchien816

[BUG] Scale replica warning (4120) - @c3y1huang @cchien816

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @cchien816

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @cchien816

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @cchien816

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @yangchiu @PhanLe1010

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @cchien816

[BUG] Volume operations/rebuilding error during eviction (4294) - @yangchiu @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @cchien816

[BUG] tests related to ha_salvage_test failed (4381) - @shuo-wu @cchien816

[BUG] test_all_replica_restore_failure failed in v1.3.1-rc2 (4383) - @shuo-wu @cchien816

[BUG] After automatically force delete terminating pods of deployment on down node, data lost and I/O error (4384) - @yangchiu @derekbit @PhanLe1010

[BUG] Volume can not attach to node when engine image DaemonSet pods are not fully deployed (4386) - @PhanLe1010 @cchien816

Misc

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @yangchiu @PhanLe1010

Contributors

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@innobead

@mantissahz

@shuo-wu

@smallteeths

@w13915984028

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.1-rc2(Aug 8, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.1-rc2 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* or v1.3.0 is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Improvement

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @cchien816

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @cchien816

[IMPROVEMENT] Avoid accidentally deleting engine images that are still in use (4332) - @derekbit @cchien816

[IMPROVEMENT] Show non-JSON error from backup store (4336) - @c3y1huang

[IMPROVEMENT] show error when failed to init frontend (4362) - @c3y1huang

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

Bugs

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @cchien816

[BUG] A IM-proxy connection not closed in full regression 1.3 (4113) - @c3y1huang @cchien816

[BUG] Scale replica warning (4120) - @c3y1huang @cchien816

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @cchien816

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @cchien816

[BUG] The last healthy replica may be evicted or removed (4238) - @yangchiu @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @cchien816

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @yangchiu @PhanLe1010

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @cchien816

[BUG] Volume operations/rebuilding error during eviction (4294) - @shuo-wu

[BUG] longhorn-manager deletes same pod multi times when rebooting (4302) - @mantissahz @w13915984028

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu @cchien816

Misc

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @yangchiu @PhanLe1010

Contributors

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@innobead

@mantissahz

@shuo-wu

@smallteeths

@w13915984028

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.1-rc1(Jul 29, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.1-rc1 released! 🎆

This release introduces some improvements and bug fixes as described below including stability, performance, availability, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* or v1.3.0 is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecations and incompatibilities in this version.

Improvement

[IMPROVEMENT] Longhorn doesn't reuse failed replica on a disk with full allocated space (3921) - @PhanLe1010 @cchien816

[IMPROVEMENT] Reduce syscalls while reading and writing requests in longhorn-engine (engine <-> replica) (4122) - @yangchiu @derekbit

[IMPROVEMENT] Reduce read and write calls in liblonghorn (tgt <-> engine) (4133) - @derekbit

[IMPROVEMENT] Replace the GCC allocator in liblonghorn with a more efficient memory allocator (4136) - @yangchiu @derekbit

[IMPROVEMENT] Purging a volume before rebuilding starts (4183) - @yangchiu @shuo-wu

[IMPROVEMENT] Schedule volumes based on available disk space (4185) - @yangchiu @c3y1huang

[IMPROVEMENT] Recognize default toleration and node selector to allow Longhorn run on the RKE mixed cluster (4246) - @c3y1huang @cchien816

Performance

[BUG] Turn a node down and up, workload takes longer time to come back online in Longhorn v1.2.0 (2947) - @yangchiu @PhanLe1010

Stability

[BUG] Somehow the Rebuilding field inside volume.meta is set to true causing the volume to stuck in attaching/detaching loop (4212) - @derekbit

Bugs

[BUG] Newly created backup stays in InProgress when the volume deleted before backup finished (3122) - @mantissahz @cchien816

[BUG] a connection not closed in full regression (4113) - @c3y1huang @cchien816

[BUG] Scale replica warning (4120) - @c3y1huang @cchien816

[BUG] Wrong nodeOrDiskEvicted collected in node monitor (4143) - @yangchiu @derekbit

[BUG] Misleading log "BUG: replica is running but storage IP is empty" (4153) - @shuo-wu @cchien816

[BUG] Replica stuck in buggy state with status.currentState is error and the spec.desireState is running (4197) - @yangchiu @PhanLe1010

[BUG] After updating longhorn to version 1.3.0, only 1 node had problems and I can't even delete it (4213) - @derekbit @c3y1huang @cchien816

[BUG] The last healthy replica may be evicted or removed (4238) - @c3y1huang @shuo-wu

[BUG] Volume detaching and attaching repeatedly while creating multiple snapshots with a same id (4250) - @yangchiu @derekbit

[BUG] Backing image is not deleted and recreated correctly (4256) - @shuo-wu @cchien816

[BUG] longhorn-ui fails to start on RKE2 with cis-1.6 profile for Longhorn v1.3.0 with helm install (4266) - @yangchiu @mantissahz

[BUG] Longhorn volume stuck in deleting state (4278) - @PhanLe1010

[BUG] build longhorn-ui image error (4283) - @smallteeths

[BUG] Wrong conditions in the Chart default-setting manifest for Rancher deployed Windows Cluster feature (4289) - @derekbit @cchien816

[BUG] Volume operations/rebuilding error during eviction (4294) - @shuo-wu

[BUG] test_setting_backing_image_auto_cleanup failed because the backing image file isn't deleted on the corresponding node as expected (4308) - @shuo-wu

Misc

[IMPROVEMENT] Support bundle doesn't collect the snapshot yamls (4285) - @PhanLe1010

Contributors

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@innobead

@mantissahz

@shuo-wu

@smallteeths

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.0(Jun 15, 2022)
Release Note

v1.3.0 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, monitoring, security, and so on. Please try it and feedback. Thanks for all the contributions!

Support multi-network K8s clusters (storage network) [lep] [doc]

Support network segregation between Longhorn control plane and data plane via Multus Network Attachment Definition resource.

Support managed Kubernetes cluster (EKS, GKE, AKS) [doc]

Make Longhorn compatible with the operations of primary managed Kubernetes clusters on public cloud like upgrade, node pool replacement, etc.

Longhorn CRD Structural Schema, v1beta2 API/CRD

Introduce v1beta2 Longhorn CRDs with schema validation to provide users another interface to operate Longhorn. v1beta2 will be the default resource version.

Snapshot CRD

Provide new Snapshot CRD to operate Snapshot functions without Longhorn UI.

v1beta2 Mutating & Validating admission webhooks

Introduce mutating and validation admission webhooks along with v1beta2 API/CRD.

Conversion Webhook for v1beta1 & v1beta2

Adopt the conversation webhook for the version conversion to ensure Longhorn API/CRD compatibility between v1beta1 and v1beta2 after upgrading to 1.3.0.

Clean up orphaned unused volume replicas [lep] [doc]

Automatically identify orphaned/unused volume replicas on hosts, and also supports opt-in automatic cleanup without manual operations.

Extend CSI Snapshot support to Longhorn snapshot [lep] [doc]

Support CSI snapshot to do Longhorn snapshot and restore volume from Longhorn snapshot.

instance-manager compatibility with Cluster Autoscaler [lep] [doc] - Experimental

Opt-in adopt dynamic PDB management of each instance-manager pod to support cluster scaling via Kubernetes Cluster Autoscaler.

Snapshot purge improvement, Snapshot Prune [lep]

Support deleting the snapshot directly behind the volume head to free duplicated space usage.

Download backing images

Allow downloading backing images from Longhorn UI.

mTLS authentication for manager <-> instance-manager communication [doc]

Opt-in secure the communication via mTLS between longhorn-manager and longhorn-instance-manager pods, Longhorn control plane components.

Support s390x IBM Z - Experimental

Support s390x architecture along with the existing supported amd64 and arm64.

Installation

Please ensure your Kubernetes cluster is at least v1.18 before installing Longhorn v1.3.0.

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.18 before upgrading to Longhorn v1.3.0 from v1.2.x. Only support upgrading from 1.2.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

For existing features, there are no deprecations or incompatible changes introduced from this release.

This release has a built-in CRD conversion webhook for v1beta1 compatibility. After upgrading from 1.2.x, all v1beta1 resources will be converted to v1beta2 and v1beta1 resources are still accessible by the conversion webhook.

The communication between longhorn manager and engine processes in each instance manager has been changed. It will be proxied through instance manager to support network segregation when adopting storage network. This change will not impact any existing volume operations.

Highlights

[FEATURE] Clean up orphaned unused volume replicas (685) - @yangchiu @derekbit

[FEATURE] Longhorn CRD Structural Schema, v1beta2 API (791) - @kaxing @jenting

[FEATURE] Support managed Kubernetes cluster (EKS, GKE, AKS) (1800) - @yangchiu @c3y1huang

[FEATURE] instance-manager compatibility with Cluster Autoscaler (2203) - @kaxing @c3y1huang

[FEATURE] Support multi-network K8s clusters (storage network) (2285) - @yangchiu @c3y1huang

[FEATURE] Extend CSI Snapshot support to Longhorn snapshot (2534) - @PhanLe1010 @cchien816

[BUG] helm upgrade won't apply customize default setting (2570) - @derekbit @cchien816

[FEATURE] Introduce SUSE BCI based component images (2939) - @meldafrawi

[FEATURE] Snapshot CRD (3144) - @PhanLe1010 @cchien816

[FEATURE] Download backing images (3155) - @yangchiu @shuo-wu

[FEATURE] Mutating and Validating admission webhook (3241) - @kaxing @jenting

[FEATURE] CRD version webhook for longhorn.io/v1beta1 and longhorn.io/v1beta2 (3265) - @kaxing @derekbit @jenting

[FEATURE] Support s390x IBM Z (3425) - @Anarkis @thehejik

[FEATURE] Support operations on custom resources directly instead of Longhorn REST APIs only (3562) - @kaxing @derekbit

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

[FEATURE] mTLS authentication for manager <-> instance-manager communication (3839) - @joshimoo @cchien816

Enhancements

[FEATURE] Include a NetworkPolicy to secure communication inside of Longhorn (1805) - @kaxing @PhanLe1010

[BUG] Remove hard-coded namespace in uninstall controller (2034) - @c3y1huang

[FEATURE] Allow use of existing private registry secret (2157) - @yangchiu @c3y1huang

[FEATURE] Recurring snapshot/backup job supports imagePullSecrets (2432) - @kaxing @jenting

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

[FEATURE] podAntiAffinity for csi-attacher/csi-provisioner/csi-resizer/csi-snapshotter (2894) - @kaxing @derekbit

[FEATURE] Prometheus metrics for backups (2940) - @meldafrawi

[FEATURE] Match type of status.conditions of nodes.longhorn.io to other k8s objects (2955) - @derekbit @khushboo-rancher

[TASK] Publish Longhorn stable releases (3014) - @PhanLe1010 @smallteeths @cchien816

[FEATURE] Expect LH has a lightweight solution for volume failure like "insufficient storage" (3734) - @PhanLe1010

[FEATURE] Longhorn UI supports orphan CRs management (3826) - @yangchiu @derekbit @smallteeths

[FEATURE] Make Longhorn UI Listen Binding configurable (3829) - @yangchiu @smallteeths

Improvement

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @cchien816

[IMPROVEMENT] Upgrade path improvement (3087) - @kaxing @shuo-wu

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[IMPROVEMENT] longhorn protocol buffer namespace conflict (3343) - @kaxing @jenting

[IMPROVEMENT] Add network policy for longhorn webhook server (3513) - @kaxing @jenting

[IMPROVEMENT] The longhorn manager should wait the longhorn webhook server ready (3523) - @kaxing @jenting

[IMPROVEMENT] Add default replica auto balance to disabled for the existing volume (3524) - @derekbit @khushboo-rancher

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Implement gRPC proxy in IM to replace MGR engine binary invocation (3546) - @c3y1huang

[IMPROVEMENT] set default non-root user for all longhorn component images (3549) - @jenting

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @cchien816

[IMPROVEMENT] Bump Longhorn k8s dependency library to a newer version (3720) - @PhanLe1010

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @cchien816

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Add doc for the Orphan replica directory and the following improvement (3853) - @derekbit

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @cchien816

[IMPROVEMENT] Improve the error message from API (3956) - @derekbit @cchien816

[IMPROVEMENT] Launch a general file managing and syncing service for backing image (3970) - @shuo-wu

[IMPROVEMENT] Cleanup unnecessary intermediate proxy client logging (4008) - @c3y1huang

[IMPROVEMENT] Add gRPC conn metrics between longhorn-manager and instance-manager (4038) - @yangchiu @c3y1huang

Performance

[BUG] Backup created in prior release takes considerable time to show up on the backup page first time. (2902) - @kaxing @jenting @khushboo-rancher

[BUG] Scalability issue of volumes in Longhorn (3083) - @kaxing @PhanLe1010

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

[BUG] Upgrading from v1.2.x to v1.2.4 take a long time if there are many backup CRs in the system (3980) - @mantissahz @PhanLe1010

[TASK] Performance benchmark of Longhorn v1.2.4 and v1.3.0 on Equinix (4002) - @derekbit

[BUG] Unexpected high memory usage of the backing image manager (4055) - @shuo-wu

[BUG] gRPC proxy connection leak when failed to get backup target client (4109) - @c3y1huang @cchien816

[BUG] gRPC proxy connection leak when failed to backup (4110) - @c3y1huang @cchien816

Stability

[BUG] Unlimited volume replica rebuilding causes system unstable like latency of etcd access (2460) - @shuo-wu @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Error message is not seen on the UI when the backup to minio fails (1268) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Name of PVC overlaps on Firefox (1598) - @smallteeths @cchien816

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] Not all informer registered cacheSync function (2671) - @jenting @khushboo-rancher

[BUG] Volume operations take long time during automatic upgrading the engines in a big cluster (2697) - @kaxing @PhanLe1010

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Track terminating pods in the volume workload status (2874) - @joshimoo @khushboo-rancher

[BUG] NFS driver: creates multiple 1st volume backup and immediately list backup volume names sometimes fails (2926) - @kaxing @jenting

[BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX. (2946) - @joshimoo @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Backups don't work after upgrading 1.1.2 to 1.2.0 (2960) - @jenting @cchien816

[BUG] Error when upgrading from 1.1.2 to 1.2.0 - Operation cannot be fulfilled on volumes.longhorn.io "pvc-edf41777-589d-4806-baca-b91d0a6c0d3c": the object has been modified; please apply your changes to the latest version and try again (2961) - @kaxing @c3y1huang

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo

[BUG] RWX prior existing PV/PVC that were marked as XFS will now fail to mount, since we only supported EXT4 previously so we ignored users choice (2991) - @joshimoo @khushboo-rancher

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @cchien816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Recurring jobs showing incorrectly (2996) - @kaxing @smallteeths

[BUG] No any implication for backup being pulling from backupstore (3023) - @kaxing @smallteeths

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @cchien816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Longhorn 1.2.0 unable to handle instance-manager failure (3070) - @PhanLe1010 @cchien816

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] fatal error: concurrent map writes (3077) - @derekbit @cchien816

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] Restoring backup always point to the latest backup (3111) - @kaxing @joshimoo @PhanLe1010 @shuo-wu

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Volume API doesn't have the workloadStatus updated (3182) - @jenting

[BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @c3y1huang @cchien816

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] rwx volume fail to attach (3193) - @shuo-wu @khushboo-rancher

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] Tests related to backup failed on nightly test run (3216) - @jenting @khushboo-rancher

[BUG] Tooltip of a backup taken shows 0 size if a snapshot is taken just before it (3228) - @kaxing @jenting

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] YAML in Recurring Snapshots and Backups page is out of format (3256) - @c3y1huang

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Show snapshot in different color when backup fails (3394) - @smallteeths

[BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher

[BUG] Can't delete engineimage after upgrade from v1.2.2 to master-head (Dec 15, 2021) (3407) - @derekbit @jenting

[BUG] cache.DeletedFinalStateUnknown conversion uses pointer type (3417) - @joshimoo

[BUG] Upgrade failed through Rancher (2.5.5) UI from 1.2.0 -> 1.2.2 (3424) -

[BUG] Failed to upgrade from v1.2.3 with volumes uses backing image to master branch (3426) - @kaxing @derekbit

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Recurring backup not work on detached volume (master-head) (3470) - @c3y1huang @khushboo-rancher @cchien816

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[BUG] RecurringJob should validate retain field is <= 50 like the UI does (3493) - @kaxing @jenting

[BUG] Volume cloning feature is broken due to CRDs schema in master-head 01/06/2022 (3508) - @PhanLe1010 @khushboo-rancher

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang @khushboo-rancher

[BUG] Error while saving Node Disk setting when there is no tag (3538) - @kaxing @derekbit

[BUG] test_setting_priority_class fails on master-head with error (3548) - @derekbit

[BUG] Capitalization on the website for the blog and KB should be consistent (3573) - @derekbit

[BUG] Backup in error state can not be deleted (3620) - @derekbit @cchien816

[BUG] Volume fails to scheduled if while creating it couldn't be scheduled. (3626) - @yangchiu @PhanLe1010

[BUG] Unable to finish install or upgrade when Failed to list *v1beta2.Node: v1beta2.NodeList.Items (3631) - @kaxing @derekbit

[BUG] Statefulset pod get data input/output error after delete instance manager (3648) -

[BUG] Replica auto-balance tests fails in regression (3649) - @c3y1huang @cchien816

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Upgrade from v1.2.x to master-head fails if backupTarget exists (3677) - @yangchiu @derekbit

[BUG] Volume not become scheduling after change over provisioning form 0 to 100 (3678) - @PhanLe1010 @cchien816

[BUG] Volume fails to schedule when set Eviction Requested from true to false on node which contains replica (3679) - @kaxing @PhanLe1010

[BUG] Longhorn webhooks fail to deploy on hardened cluster (3701) - @kaxing @derekbit

[BUG] longhornio/longhorn-manager:master-head currently CrashLoopBackOff-ing (3716) -

[BUG] Data migration sample YAML moves data to the subfolder (3787) - @yangchiu @c3y1huang

[BUG] go.mod and go.sum out of date (3796) - @derekbit

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] Volume can't detach when "Concurrent Replica Rebuild Per Node Limit = 0" and set node evicted from True to False (3820) - @PhanLe1010

[BUG] Longhorn default StorageClass is not always synced with the ConfigMap (3821) - @yangchiu @shuo-wu

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] BackingImageManager version request leaked grpc connection (3838) - @kaxing @derekbit

[BUG] PVC create from volumsanpshot stuck at pending status(type=snap) (3860) - @kaxing @derekbit

[BUG] liblonghorn sends integer fields in longhorn protocol as host endianness and longhorn-engine always expects litte endian (3875) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @cchien816

[BUG] Volume created from cloned PVC will stuck at Not Ready status (3886) - @PhanLe1010 @cchien816

[BUG] UI pop error if long volume name used (when > 40 characters) (3915) - @derekbit @cchien816

[BUG] Instance manager keep restarting (3922) - @joshimoo @cchien816

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @cchien816

[BUG] Event Log on Dashboard is always empty on s390x with v1.3.0-preview1 (3958) - @mantissahz @smallteeths @cchien816

[BUG] Command longhorn-instance-manager process won't work without TLS key (3966) - @joshimoo @cchien816

[BUG][v1.3.0-rc1] faulted volume after engine upgraded (4004) - @c3y1huang @cchien816

[BUG] [v1.3.0-rc1] set createSecret :true cause error when install Longhron from Rancher Chart (4007) - @derekbit

[BUG] updateSettingDefinitions throws concurrent map read/write (4021) - @kaxing @derekbit

[BUG] Snapshot CRD feature is broken on Kubernetes v1.18 (4022) - @kaxing @PhanLe1010

[BUG] Failed to build longhorn-engine (4032) - @derekbit

[BUG] test_rwx_parallel_writing failed at missing awk (4035) - @mantissahz

[BUG] Longhorn can't handle restore command error in DR volume incremental restore (4049) - @derekbit @cchien816

[BUG] When downloading a backing image, the original file name w/ suffix should be reserved (4052) - @smallteeths @shuo-wu @cchien816

[BUG] longhorn-manager pods backoff: snapshot controller failed to get volume (4056) - @kaxing @PhanLe1010

[BUG] Error in little endian support in liblonghorn (4059) - @keithalucas

[BUG] Longhorn manager nil pointer dereference from proxy SnapshotBackup (4061) - @c3y1huang

[BUG] longhorn-manager panic if UI deployment missing and updating CA settings (4065) - @mantissahz @c3y1huang

[BUG] Downloaded backing image state incorrect after restarting/upgrading the backing image manager (4073) - @kaxing @shuo-wu

[BUG] Backing image download or upload timeout does not work (4074) - @kaxing @shuo-wu

[BUG] Backing image without checksum value created from older release will not able to be reuse: no expectedChecksum for existing file (4085) - @kaxing @shuo-wu

[BUG] Backing image manager does not release ports correctly after file syncing (4086) - @mantissahz @shuo-wu

[BUG] Delete and resync failed backing image files (4090) - @mantissahz @shuo-wu

[BUG] prometheus-operator pod in CrashLoopBackOff state following doc instruction (4094) - @c3y1huang @cchien816

[BUG] gRPC client need to use long timeout where previously binary execute without timeout (4096) - @yangchiu @c3y1huang

[BUG] kubectl drain node gets stuck forever (4103) - @mantissahz @cchien816

[BUG] longhorn-manager crashed due to "close of closed channel" (4106) - @derekbit @shuo-wu

[BUG] The last backing image file may not be cleaned up before the backing image manager shutdown (4111) - @mantissahz @shuo-wu

Misc

[BUG] After upgrade to 1.1.0 some old managers still running (2144) - @derekbit

[DOC] Change VolumeSnapshotClass API version to v1 from v1beta1 in documentation (2214) -

[REFACTORING] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[TASK] Update document to reflect on Volume Attachment Recovery Policy feature deprecation (2954) - @jenting

[TEST] Add test case in e2e integration test for fsgroup support (2967) - @khushboo-rancher

[DOC] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[UI] Notify users about newer stable versions (3032) - @smallteeths @cchien816

[TASK] Extend OS support matrix (3085) - @meldafrawi

[TASK] Remove the feature that skip taking backup/snapshot when the volume head is empty in recurring backup jobs (3128) - @derekbit @khushboo-rancher

[TASK] Change the CSI sidecar images from upstream repo to Longhorn repo (3258) - @PhanLe1010 @khushboo-rancher

[BUG] Test_settings fails in the integration test (3280) - @jenting

[TASK] Generate multiple CRD versions (3295) - @kaxing @derekbit @jenting

[TASK] The CSI resizer may not work when the Job pod status is Completed (3303) - @kaxing @shuo-wu

[DOC] Failed auto installation of iscsi on Debian 11 bullseye (3313) - @derekbit

[TASK] Handle the CRD validation error (3352) - @kaxing @derekbit

[TASK] Restore to a brand new cluster that does not contain anything (3367) - @shuo-wu @khushboo-rancher

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

[TASK] Create golang API for mounting NVMeoF targets (3602) - @keithalucas

[TASK] Improve the doc for volume actual size and space usage (3627) - @derekbit @shuo-wu

[DOC] Update RHEL instructions for open-iscsi (3630) - @keithalucas

[TASK] Need warning message on UI when deleting the last recurring job of the group (3651) - @yangchiu @smallteeths

[TASK] Create a migration path from deprecated Rancher catalog to Rancher marketplace (3714) - @PhanLe1010

[DOC] Alert LonghornNodeDown example in documentation doesn't work (3729) - @kaxing @jenting

[TASK] Add readme document for the admission webhook (3752) - @derekbit @cchien816

[TASK] Refactor the Logic for Node Monitoring (3775) - @derekbit

[TASK] Move secscan nightly tests to aws (3779) - @meldafrawi

[TASK] Improve the doc for troubleshooting NFS backuptarget (3825) - @mantissahz

[DOC] Add cluster autoscaler support (3884) - @c3y1huang

[TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @cchien816

[TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version (3891) - @yangchiu @PhanLe1010

[DOC] Update the volume size related doc after introducing the snapshot prune feature (3894) - @shuo-wu

[TASK] add finalizer to resource in mutating webhook (3918) - @derekbit

[TEST] Run e2e regression of 1.3.0 (3949) - @yangchiu

[DOC] mTLS authentication for manager <-> instance-manager communication (3953) - @joshimoo

[DOC] Improve backing documentation (4082) - @shuo-wu

Contributors

@Anarkis

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@ibrokethecloud

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@thehejik

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.0-rc3(Jun 8, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.0-rc3 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

For existing features, there are no deprecations or incompatible changes introduced in this release.

v1beta2 resource version is introduced with validating and mutating admission webhooks. v1beta2 will be the default resource version.

This release has a built-in CRD conversion webhook for v1beta1 compatibility. After upgrading from a previously supported version, all v1beta1 resources will be converted to v1beta2 but v1beta1 resources are still accessible by the conversion webhook.

The communication between the longhorn manager and each engine process in the instance manager will be proxied by the instance manager for the network segregation purpose to support the new storage network. However, this underlying change will not impact any existing volume operations.

Highlights

[FEATURE] Clean up orphaned unused volume replicas after node failure (685) - @yangchiu @derekbit

[REFACTOR] Longhorn CRD structural schema (791) - @kaxing @jenting

[FEATURE] Longhorn with managed Kubernetes cluster(EKS, GKE, AKS) (1800) - @yangchiu @c3y1huang

[FEATURE] Longhorn should support multi-network K8s clusters (storage network) (2285) - @yangchiu @c3y1huang

[FEATURE] Extend CSI Snapshot support to Longhorn snapshot (2534) - @PhanLe1010 @cchien816

[BUG] helm upgrade won't apply customize default setting (2570) - @derekbit @cchien816

[FEATURE] CRD for snapshots (3144) - @PhanLe1010 @cchien816

[FEATURE] Allowing users to download backing images (3155) - @yangchiu @shuo-wu

[FEATURE] CRD version webhook for longhorn.io/v1beta1 and longhorn.io/v1beta2 (3265) - @kaxing @derekbit @jenting

[FEATURE] Builds for IBM Z (3425) - @Anarkis

[FEATURE] Support operations on custom resources directly instead of Longhorn REST APIs only (3562) - @kaxing @derekbit

[FEATURE] mTLS authentication for manager <-> instance-manager communication (3839) - @joshimoo @cchien816

Enhancements

[FEATURE] Include a NetworkPolicy to secure communication inside of Longhorn (1805) - @kaxing @PhanLe1010

[BUG] Remove hard-coded namespace in uninstall controller (2034) - @c3y1huang

[FEATURE] Allow use of existing private registry secret (2157) - @yangchiu @c3y1huang

[FEATURE] instance-manager compatibility with Cluster Autoscaler (2203) - @kaxing @c3y1huang

[FEATURE] Recurring snapshot/backup job supports imagePullSecrets (2432) - @kaxing @jenting

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

[FEATURE] podAntiAffinity for csi-attacher/csi-provisioner/csi-resizer/csi-snapshotter (2894) - @kaxing @derekbit

[FEATURE] Introduce SUSE BCI based component images (2939) - @meldafrawi

[FEATURE] Prometheus metrics for backups (2940) - @meldafrawi

[FEATURE] Match type of status.conditions of nodes.longhorn.io to other k8s objects (2955) - @derekbit @khushboo-rancher

[TASK] Publish Longhorn stable releases (3014) - @PhanLe1010 @smallteeths @cchien816

[FEATURE] Mutating/Validating admission webhook (3241) - @kaxing @jenting

[FEATURE] Expect LH has a lightweight solution for volume failure like "insufficient storage" (3734) - @PhanLe1010

[FEATURE] Longhorn UI supports orphan CRs management (3826) - @yangchiu @derekbit @smallteeths

[FEATURE] Make Longhorn UI Listen Binding configurable (3829) - @yangchiu @smallteeths

Improvement

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @cchien816

[IMPROVEMENT] Upgrade path improvement (3087) - @kaxing @shuo-wu

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[IMPROVEMENT] longhorn protocol buffer namespace conflict (3343) - @kaxing @jenting

[IMPROVEMENT] Add network policy for longhorn webhook server (3513) - @kaxing @jenting

[IMPROVEMENT] The longhorn manager should wait the longhorn webhook server ready (3523) - @kaxing @jenting

[IMPROVEMENT] Add default replica auto balance to disabled for the existing volume (3524) - @derekbit @khushboo-rancher

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Implement gRPC proxy in IM to replace MGR engine binary invocation (3546) - @c3y1huang

[IMPROVEMENT] set default non-root user for all longhorn component images (3549) - @jenting

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @cchien816

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

[IMPROVEMENT] Bump Longhorn k8s dependency library to a newer version (3720) - @PhanLe1010

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @cchien816

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Add doc for the Orphan replica directory and the following improvement (3853) - @derekbit

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @cchien816

[IMPROVEMENT] Improve the error message from API (3956) - @derekbit @cchien816

[IMPROVEMENT] Launch a general file managing and syncing service for backing image (3970) - @shuo-wu

[IMPROVEMENT] Cleanup unnecessary intermediate proxy client logging (4008) - @c3y1huang

Performance

[BUG] Backup created in prior release takes considerable time to show up on the backup page first time. (2902) - @kaxing @jenting @khushboo-rancher

[BUG] Scalability issue of volumes in Longhorn (3083) - @kaxing @PhanLe1010

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

[BUG] Upgrading from v1.2.x to v1.2.4 take a long time if there are many backup CRs in the system (3980) - @mantissahz @PhanLe1010

[TASK] Performance benchmark of Longhorn v1.2.4 and v1.3.0 on Equinix (4002) - @derekbit

[BUG] Unexpected high memory usage of the backing image manager (4055) - @shuo-wu

Stability

[BUG] Unlimited volume replica rebuilding causes system unstable like latency of etcd access (2460) - @shuo-wu @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Error message is not seen on the UI when the backup to minio fails (1268) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Name of PVC overlaps on Firefox (1598) - @smallteeths @cchien816

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] Not all informer registered cacheSync function (2671) - @jenting @khushboo-rancher

[BUG] Volume operations take long time during automatic upgrading the engines in a big cluster (2697) - @kaxing @PhanLe1010

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Track terminating pods in the volume workload status (2874) - @joshimoo @khushboo-rancher

[BUG] NFS driver: creates multiple 1st volume backup and immediately list backup volume names sometimes fails (2926) - @kaxing @jenting

[BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX. (2946) - @joshimoo @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Backups don't work after upgrading 1.1.2 to 1.2.0 (2960) - @jenting @cchien816

[BUG] Error when upgrading from 1.1.2 to 1.2.0 - Operation cannot be fulfilled on volumes.longhorn.io "pvc-edf41777-589d-4806-baca-b91d0a6c0d3c": the object has been modified; please apply your changes to the latest version and try again (2961) - @kaxing @c3y1huang

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo

[BUG] RWX prior existing PV/PVC that were marked as XFS will now fail to mount, since we only supported EXT4 previously so we ignored users choice (2991) - @joshimoo @khushboo-rancher

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @cchien816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Recurring jobs showing incorrectly (2996) - @kaxing @smallteeths

[BUG] No any implication for backup being pulling from backupstore (3023) - @kaxing @smallteeths

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @cchien816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Longhorn 1.2.0 unable to handle instance-manager failure (3070) - @PhanLe1010 @cchien816

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] fatal error: concurrent map writes (3077) - @derekbit @cchien816

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] Restoring backup always point to the latest backup (3111) - @kaxing @joshimoo @PhanLe1010 @shuo-wu

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Volume API doesn't have the workloadStatus updated (3182) - @jenting

[BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @c3y1huang @cchien816

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] rwx volume fail to attach (3193) - @shuo-wu @khushboo-rancher

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] Tests related to backup failed on nightly test run (3216) - @jenting @khushboo-rancher

[BUG] Tooltip of a backup taken shows 0 size if a snapshot is taken just before it (3228) - @kaxing @jenting

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] YAML in Recurring Snapshots and Backups page is out of format (3256) - @c3y1huang

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Show snapshot in different color when backup fails (3394) - @smallteeths

[BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher

[BUG] Can't delete engineimage after upgrade from v1.2.2 to master-head (Dec 15, 2021) (3407) - @derekbit @jenting

[BUG] cache.DeletedFinalStateUnknown conversion uses pointer type (3417) - @joshimoo

[BUG] Upgrade failed through Rancher (2.5.5) UI from 1.2.0 -> 1.2.2 (3424) -

[BUG] Failed to upgrade from v1.2.3 with volumes uses backing image to master branch (3426) - @kaxing @derekbit

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Recurring backup not work on detached volume (master-head) (3470) - @c3y1huang @khushboo-rancher @cchien816

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[BUG] RecurringJob should validate retain field is <= 50 like the UI does (3493) - @kaxing @jenting

[BUG] Volume cloning feature is broken due to CRDs schema in master-head 01/06/2022 (3508) - @PhanLe1010 @khushboo-rancher

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang @khushboo-rancher

[BUG] Error while saving Node Disk setting when there is no tag (3538) - @kaxing @derekbit

[BUG] test_setting_priority_class fails on master-head with error (3548) - @derekbit

[BUG] Capitalization on the website for the blog and KB should be consistent (3573) - @derekbit

[BUG] Backup in error state can not be deleted (3620) - @derekbit @cchien816

[BUG] Volume fails to scheduled if while creating it couldn't be scheduled. (3626) - @yangchiu @PhanLe1010

[BUG] Unable to finish install or upgrade when Failed to list *v1beta2.Node: v1beta2.NodeList.Items (3631) - @kaxing @derekbit

[BUG] Statefulset pod get data input/output error after delete instance manager (3648) -

[BUG] Replica auto-balance tests fails in regression (3649) - @c3y1huang @cchien816

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Upgrade from v1.2.x to master-head fails if backupTarget exists (3677) - @yangchiu @derekbit

[BUG] Volume not become scheduling after change over provisioning form 0 to 100 (3678) - @PhanLe1010 @cchien816

[BUG] Volume fails to schedule when set Eviction Requested from true to false on node which contains replica (3679) - @kaxing @PhanLe1010

[BUG] Longhorn webhooks fail to deploy on hardened cluster (3701) - @kaxing @derekbit

[BUG] longhornio/longhorn-manager:master-head currently CrashLoopBackOff-ing (3716) -

[BUG] Data migration sample YAML moves data to the subfolder (3787) - @yangchiu @c3y1huang

[BUG] go.mod and go.sum out of date (3796) - @derekbit

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] Volume can't detach when "Concurrent Replica Rebuild Per Node Limit = 0" and set node evicted from True to False (3820) - @PhanLe1010

[BUG] Longhorn default StorageClass is not always synced with the ConfigMap (3821) - @yangchiu @shuo-wu

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] BackingImageManager version request leaked grpc connection (3838) - @kaxing @derekbit

[BUG] PVC create from volumsanpshot stuck at pending status(type=snap) (3860) - @kaxing @derekbit

[BUG] liblonghorn sends integer fields in longhorn protocol as host endianness and longhorn-engine always expects litte endian (3875) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @cchien816

[BUG] Volume created from cloned PVC will stuck at Not Ready status (3886) - @PhanLe1010 @cchien816

[BUG] UI pop error if long volume name used (when > 40 characters) (3915) - @derekbit @cchien816

[BUG] Instance manager keep restarting (3922) - @joshimoo @cchien816

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @cchien816

[BUG] Event Log on Dashboard is always empty on s390x with v1.3.0-preview1 (3958) - @mantissahz @smallteeths @cchien816

[BUG] Command longhorn-instance-manager process won't work without TLS key (3966) - @joshimoo @cchien816

[BUG][v1.3.0-rc1] faulted volume after engine upgraded (4004) - @c3y1huang @cchien816

[BUG] [v1.3.0-rc1] set createSecret :true cause error when install Longhron from Rancher Chart (4007) - @derekbit

[BUG] updateSettingDefinitions throws concurrent map read/write (4021) - @kaxing @derekbit

[BUG] Snapshot CRD feature is broken on Kubernetes v1.18 (4022) - @kaxing @PhanLe1010

[BUG] Failed to build longhorn-engine (4032) - @derekbit

[BUG] test_rwx_parallel_writing failed at missing awk (4035) - @mantissahz

[BUG] Longhorn can't handle restore command error in DR volume incremental restore (4049) - @derekbit @cchien816

[BUG] When downloading a backing image, the original file name w/ suffix should be reserved (4052) - @smallteeths @shuo-wu @cchien816

[BUG] longhorn-manager pods backoff: snapshot controller failed to get volume (4056) - @kaxing @PhanLe1010

[BUG] Error in little endian support in liblonghorn (4059) - @keithalucas

[BUG] Longhorn manager nil pointer dereference from proxy SnapshotBackup (4061) - @c3y1huang

[BUG] Downloaded backing image state incorrect after restarting/upgrading the backing image manager (4073) - @kaxing @shuo-wu

[BUG] Backing image download or upload timeout does not work (4074) - @kaxing @shuo-wu

[BUG] Backing image manager does not release ports correctly after file syncing (4086) - @shuo-wu

Misc

[BUG] After upgrade to 1.1.0 some old managers still running (2144) - @derekbit

[DOC] Change VolumeSnapshotClass API version to v1 from v1beta1 in documentation (2214) -

[REFACTORING] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[TASK] Update document to reflect on Volume Attachment Recovery Policy feature deprecation (2954) - @jenting

[TEST] Add test case in e2e integration test for fsgroup support (2967) - @khushboo-rancher

[DOC] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[UI] Notify users about newer stable versions (3032) - @smallteeths @cchien816

[TASK] Extend OS support matrix (3085) - @meldafrawi

[TASK] Remove the feature that skip taking backup/snapshot when the volume head is empty in recurring backup jobs (3128) - @derekbit @khushboo-rancher

[TASK] Change the CSI sidecar images from upstream repo to Longhorn repo (3258) - @PhanLe1010 @khushboo-rancher

[BUG] Test_settings fails in the integration test (3280) - @jenting

[TASK] Generate multiple CRD versions (3295) - @kaxing @derekbit @jenting

[TASK] The CSI resizer may not work when the Job pod status is Completed (3303) - @kaxing @shuo-wu

[DOC] Failed auto installation of iscsi on Debian 11 bullseye (3313) - @derekbit

[TASK] Handle the CRD validation error (3352) - @kaxing @derekbit

[TASK] Restore to a brand new cluster that does not contain anything (3367) - @shuo-wu @khushboo-rancher

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

[TASK] Create golang API for mounting NVMeoF targets (3602) - @keithalucas

[TASK] Improve the doc for volume actual size and space usage (3627) - @derekbit @shuo-wu

[DOC] Update RHEL instructions for open-iscsi (3630) - @keithalucas

[TASK] Need warning message on UI when deleting the last recurring job of the group (3651) - @yangchiu @smallteeths

[TASK] Create a migration path from deprecated Rancher catalog to Rancher marketplace (3714) - @PhanLe1010

[DOC] Alert LonghornNodeDown example in documentation doesn't work (3729) - @kaxing @jenting

[TASK] Add readme document for the admission webhook (3752) - @derekbit @cchien816

[TASK] Refactor the Logic for Node Monitoring (3775) - @derekbit

[TASK] Move secscan nightly tests to aws (3779) - @meldafrawi

[TASK] Improve the doc for troubleshooting NFS backuptarget (3825) - @mantissahz

[DOC] Add cluster autoscaler support (3884) - @c3y1huang

[TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @cchien816

[TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version (3891) - @yangchiu @PhanLe1010

[TASK] Update the volume size related doc after introducing the snapshot prune feature (3894) - @shuo-wu

[TASK] add finalizer to resource in mutating webhook (3918) - @derekbit

[DOC] mTLS authentication for manager <-> instance-manager communication (3953) - @joshimoo

[DOC] Improve backing documentation (4082) - @shuo-wu

Contributors

@Anarkis

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@ibrokethecloud

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.0-rc2(May 31, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.0-rc2 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

For existing features, there are no deprecations or incompatible changes introduced in this release.

v1beta2 resource version is introduced with validating and mutating admission webhooks. v1beta2 will be the default resource version.

This release has a built-in CRD conversion webhook for v1beta1 compatibility. After upgrading from a previously supported version, all v1beta1 resources will be converted to v1beta2 but v1beta1 resources are still accessible by the conversion webhook.

The communication between the longhorn manager and each engine process in the instance manager will be proxied by the instance manager for the network segregation purpose to support the new storage network. However, this underlying change will not impact any existing volume operations.

Highlights

[FEATURE] Clean up orphaned unused volume replicas after node failure (685) - @yangchiu @derekbit

[REFACTOR] Longhorn CRD structural schema (791) - @kaxing @jenting

[FEATURE] Longhorn with managed Kubernetes cluster(EKS, GKE, AKS) (1800) - @yangchiu @c3y1huang

[FEATURE] Longhorn should support multi-network K8s clusters (storage network) (2285) - @c3y1huang

[FEATURE] Extend CSI Snapshot support to Longhorn snapshot (2534) - @PhanLe1010 @cchien816

[BUG] helm upgrade won't apply customize default setting (2570) - @derekbit @cchien816

[POC] Rebuild logical volumes/replicas with SPDK (3044) - @keithalucas

[FEATURE] CRD for snapshots (3144) - @PhanLe1010 @cchien816

[FEATURE] Allowing users to download backing images (3155) - @shuo-wu

[FEATURE] CRD version webhook for longhorn.io/v1beta1 and longhorn.io/v1beta2 (3265) - @kaxing @derekbit @jenting

[FEATURE] Builds for IBM Z (3425) - @Anarkis

[FEATURE] Support operations on custom resources directly instead of Longhorn REST APIs only (3562) - @kaxing @derekbit

[FEATURE] mTLS authentication for manager <-> instance-manager communication (3839) - @joshimoo @cchien816

Enhancements

[FEATURE] Include a NetworkPolicy to secure communication inside of Longhorn (1805) - @kaxing @PhanLe1010

[BUG] Remove hard-coded namespace in uninstall controller (2034) - @c3y1huang

[FEATURE] Allow use of existing private registry secret (2157) - @yangchiu @c3y1huang

[FEATURE] instance-manager compatibility with Cluster Autoscaler (2203) - @kaxing @c3y1huang

[FEATURE] Recurring snapshot/backup job supports imagePullSecrets (2432) - @kaxing @jenting

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

[FEATURE] podAntiAffinity for csi-attacher/csi-provisioner/csi-resizer/csi-snapshotter (2894) - @kaxing @derekbit

[FEATURE] Introduce SUSE BCI based component images (2939) - @meldafrawi

[FEATURE] Prometheus metrics for backups (2940) - @meldafrawi

[FEATURE] Match type of status.conditions of nodes.longhorn.io to other k8s objects (2955) - @derekbit @khushboo-rancher

[TASK] Publish Longhorn stable releases (3014) - @PhanLe1010 @smallteeths @cchien816

[FEATURE] Mutating/Validating admission webhook (3241) - @kaxing @jenting

[FEATURE] Expect LH has a lightweight solution for volume failure like "insufficient storage" (3734) - @PhanLe1010

[FEATURE] Longhorn UI supports orphan CRs management (3826) - @yangchiu @derekbit @smallteeths

[FEATURE] Make Longhorn UI Listen Binding configurable (3829) - @yangchiu @smallteeths

Improvement

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Improve communication between components (GRPC) (2821) - @joshimoo

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @cchien816

[IMPROVEMENT] Upgrade path improvement (3087) - @kaxing @shuo-wu

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[IMPROVEMENT] longhorn protocol buffer namespace conflict (3343) - @kaxing @jenting

[IMPROVEMENT] Add network policy for longhorn webhook server (3513) - @kaxing @jenting

[IMPROVEMENT] The longhorn manager should wait the longhorn webhook server ready (3523) - @kaxing @jenting

[IMPROVEMENT] Add default replica auto balance to disabled for the existing volume (3524) - @derekbit @khushboo-rancher

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Implement gRPC proxy in IM to replace MGR engine binary invocation (3546) - @c3y1huang

[IMPROVEMENT] set default non-root user for all longhorn component images (3549) - @jenting

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @cchien816

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

[IMPROVEMENT] Bump Longhorn k8s dependency library to a newer version (3720) - @PhanLe1010

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @cchien816

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Add doc for the Orphan replica directory and the following improvement (3853) - @derekbit

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @cchien816

[IMPROVEMENT] Improve the error message from API (3956) - @derekbit @cchien816

[IMPROVEMENT] Refactor to use dedicated goroutine for proxy client handling (3967) - @c3y1huang

[IMPROVEMENT] Launch a general file managing and syncing service (3970) - @shuo-wu

[IMPROVEMENT] Send credential to process server via request (3977) - @c3y1huang

[IMPROVEMENT] Cleanup unnecessary intermediate proxy client logging (4008) - @c3y1huang

Performance

[BUG] Backup created in prior release takes considerable time to show up on the backup page first time. (2902) - @kaxing @jenting @khushboo-rancher

[BUG] Scalability issue of volumes in Longhorn (3083) - @kaxing @PhanLe1010

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

[BUG] Upgrading from v1.2.x to v1.2.4 take a long time if there are many backup CRs in the system (3980) - @mantissahz @PhanLe1010

[TASK] Performance benchmark of Longhorn v1.2.4 and v1.3.0 on Equinix (4002) - @derekbit

Stability

[BUG] Unlimited volume replica rebuilding causes system unstable like latency of etcd access (2460) - @shuo-wu @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Error message is not seen on the UI when the backup to minio fails (1268) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Name of PVC overlaps on Firefox (1598) - @smallteeths @cchien816

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] Not all informer registered cacheSync function (2671) - @jenting @khushboo-rancher

[BUG] Volume operations take long time during automatic upgrading the engines in a big cluster (2697) - @kaxing @PhanLe1010

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Track terminating pods in the volume workload status (2874) - @joshimoo @khushboo-rancher

[BUG] NFS driver: creates multiple 1st volume backup and immediately list backup volume names sometimes fails (2926) - @kaxing @jenting

[BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX. (2946) - @joshimoo @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Backups don't work after upgrading 1.1.2 to 1.2.0 (2960) - @jenting @cchien816

[BUG] Error when upgrading from 1.1.2 to 1.2.0 - Operation cannot be fulfilled on volumes.longhorn.io "pvc-edf41777-589d-4806-baca-b91d0a6c0d3c": the object has been modified; please apply your changes to the latest version and try again (2961) - @kaxing @c3y1huang

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo

[BUG] RWX prior existing PV/PVC that were marked as XFS will now fail to mount, since we only supported EXT4 previously so we ignored users choice (2991) - @joshimoo @khushboo-rancher

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @cchien816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Recurring jobs showing incorrectly (2996) - @kaxing @smallteeths

[BUG] No any implication for backup being pulling from backupstore (3023) - @kaxing @smallteeths

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @cchien816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Longhorn 1.2.0 unable to handle instance-manager failure (3070) - @PhanLe1010 @cchien816

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] fatal error: concurrent map writes (3077) - @derekbit @cchien816

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] Restoring backup always point to the latest backup (3111) - @kaxing @joshimoo @PhanLe1010 @shuo-wu

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Volume API doesn't have the workloadStatus updated (3182) - @jenting

[BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @c3y1huang @cchien816

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] rwx volume fail to attach (3193) - @shuo-wu @khushboo-rancher

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] Tests related to backup failed on nightly test run (3216) - @jenting @khushboo-rancher

[BUG] Tooltip of a backup taken shows 0 size if a snapshot is taken just before it (3228) - @kaxing @jenting

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] YAML in Recurring Snapshots and Backups page is out of format (3256) - @c3y1huang

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Show snapshot in different color when backup fails (3394) - @smallteeths

[BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher

[BUG] Can't delete engineimage after upgrade from v1.2.2 to master-head (Dec 15, 2021) (3407) - @derekbit @jenting

[BUG] cache.DeletedFinalStateUnknown conversion uses pointer type (3417) - @joshimoo

[BUG] Upgrade failed through Rancher (2.5.5) UI from 1.2.0 -> 1.2.2 (3424) -

[BUG] Failed to upgrade from v1.2.3 with volumes uses backing image to master branch (3426) - @kaxing @derekbit

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Recurring backup not work on detached volume (master-head) (3470) - @c3y1huang @khushboo-rancher @cchien816

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[BUG] RecurringJob should validate retain field is <= 50 like the UI does (3493) - @kaxing @jenting

[BUG] Volume cloning feature is broken due to CRDs schema in master-head 01/06/2022 (3508) - @PhanLe1010 @khushboo-rancher

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang @khushboo-rancher

[BUG] Error while saving Node Disk setting when there is no tag (3538) - @kaxing @derekbit

[BUG] test_setting_priority_class fails on master-head with error (3548) - @derekbit

[BUG] Capitalization on the website for the blog and KB should be consistent (3573) - @derekbit

[BUG] Backup in error state can not be deleted (3620) - @derekbit @cchien816

[BUG] Volume fails to scheduled if while creating it couldn't be scheduled. (3626) - @yangchiu @PhanLe1010

[BUG] Unable to finish install or upgrade when Failed to list *v1beta2.Node: v1beta2.NodeList.Items (3631) - @kaxing @derekbit

[BUG] Statefulset pod get data input/output error after delete instance manager (3648) -

[BUG] Replica auto-balance tests fails in regression (3649) - @c3y1huang @cchien816

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Upgrade from v1.2.x to master-head fails if backupTarget exists (3677) - @yangchiu @derekbit

[BUG] Volume not become scheduling after change over provisioning form 0 to 100 (3678) - @PhanLe1010 @cchien816

[BUG] Volume fails to schedule when set Eviction Requested from true to false on node which contains replica (3679) - @kaxing @PhanLe1010

[BUG] Longhorn webhooks fail to deploy on hardened cluster (3701) - @kaxing @derekbit

[BUG] longhornio/longhorn-manager:master-head currently CrashLoopBackOff-ing (3716) -

[BUG] Data migration sample YAML moves data to the subfolder (3787) - @yangchiu @c3y1huang

[BUG] go.mod and go.sum out of date (3796) - @derekbit

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] Volume can't detach when "Concurrent Replica Rebuild Per Node Limit = 0" and set node evicted from True to False (3820) - @PhanLe1010

[BUG] Longhorn default StorageClass is not always synced with the ConfigMap (3821) - @yangchiu @shuo-wu

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @shuo-wu

[BUG] BackingImageManager version request leaked grpc connection (3838) - @kaxing @derekbit

[BUG] PVC create from volumsanpshot stuck at pending status(type=snap) (3860) - @kaxing @derekbit

[BUG] liblonghorn sends integer fields in longhorn protocol as host endianness and longhorn-engine always expects litte endian (3875) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @cchien816

[BUG] Volume created from cloned PVC will stuck at Not Ready status (3886) - @PhanLe1010 @cchien816

[BUG] UI pop error if long volume name used (when > 40 characters) (3915) - @derekbit @cchien816

[BUG] Instance manager keep restarting (3922) - @joshimoo @cchien816

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @cchien816

[BUG] Event Log on Dashboard is always empty on s390x with v1.3.0-preview1 (3958) - @mantissahz @smallteeths @cchien816

[BUG] Command longhorn-instance-manager process won't work without TLS key (3966) - @joshimoo @cchien816

[BUG][v1.3.0-rc1] faulted volume after engine upgraded (4004) - @c3y1huang @cchien816

[BUG] [v1.3.0-rc1] set createSecret :true cause error when install Longhron from Rancher Chart (4007) - @derekbit

[BUG] updateSettingDefinitions throws concurrent map read/write (4021) - @kaxing @derekbit

[BUG] Snapshot CRD feature is broken on Kubernetes v1.18 (4022) - @kaxing @PhanLe1010

[BUG] Failed to build longhorn-engine (4032) - @derekbit

[BUG] test_rwx_parallel_writing failed at missing awk (4035) - @mantissahz

[BUG] Longhorn can't handle restore command error in DR volume incremental restore (4049) - @derekbit

Misc

[BUG] After upgrade to 1.1.0 some old managers still running (2144) - @derekbit

[DOC] Change VolumeSnapshotClass API version to v1 from v1beta1 in documentation (2214) -

[REFACTORING] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[TASK] Deprecate volume spec recurringJobs and storageClass recurringJobs field (2865) - @derekbit

[TASK] Update document to reflect on Volume Attachment Recovery Policy feature deprecation (2954) - @jenting

[TEST] Add test case in e2e integration test for fsgroup support (2967) - @khushboo-rancher

[DOC] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[UI] Notify users about newer stable versions (3032) - @smallteeths @cchien816

[TASK] Extend OS support matrix (3085) - @meldafrawi

[TASK] Remove the feature that skip taking backup/snapshot when the volume head is empty in recurring backup jobs (3128) - @derekbit @khushboo-rancher

[TASK] Change the CSI sidecar images from upstream repo to Longhorn repo (3258) - @PhanLe1010 @khushboo-rancher

[BUG] Test_settings fails in the integration test (3280) - @jenting

[TASK] Generate multiple CRD versions (3295) - @kaxing @derekbit @jenting

[TASK] The CSI resizer may not work when the Job pod status is Completed (3303) - @kaxing @shuo-wu

[DOC] Failed auto installation of iscsi on Debian 11 bullseye (3313) - @derekbit

[TASK] Handle the CRD validation error (3352) - @kaxing @derekbit

[TASK] Restore to a brand new cluster that does not contain anything (3367) - @shuo-wu @khushboo-rancher

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

[TASK] Create golang API for mounting NVMeoF targets (3602) - @keithalucas

[TASK] Improve the doc for volume actual size and space usage (3627) - @derekbit @shuo-wu

[DOC] Update RHEL instructions for open-iscsi (3630) - @keithalucas

[TASK] Need warning message on UI when deleting the last recurring job of the group (3651) - @yangchiu @smallteeths

[TASK] Create a migration path from deprecated Rancher catalog to Rancher marketplace (3714) - @PhanLe1010

[DOC] Alert LonghornNodeDown example in documentation doesn't work (3729) - @kaxing @jenting

[TASK] Add readme document for the admission webhook (3752) - @derekbit @cchien816

[TASK] Refactor the Logic for Node Monitoring (3775) - @derekbit

[TASK] Move secscan nightly tests to aws (3779) - @meldafrawi

[TASK] Improve the doc for troubleshooting NFS backuptarget (3825) - @mantissahz

[DOC] Add cluster autoscaler support (3884) - @c3y1huang

[TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @cchien816

[TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version (3891) - @yangchiu @PhanLe1010

[TASK] Update the volume size related doc after introducing the snapshot prune feature (3894) - @shuo-wu

[TASK] add finalizer to resource in mutating webhook (3918) - @derekbit

[DOC] mTLS authentication for manager <-> instance-manager communication (3953) - @joshimoo

Contributors

@Anarkis

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@ibrokethecloud

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.0-rc1(May 17, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.0-rc1 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

For existing features, there are no deprecations or incompatible changes introduced in this release.

v1beta2 resource version is introduced with validating and mutating admission webhooks. v1beta2 will be the default resource version.

This release has a built-in CRD conversion webhook for v1beta1 compatibility. After upgrading from a previously supported version, all v1beta1 resources will be converted to v1beta2 but v1beta1 resources are still accessible by the conversion webhook.

Highlights

[FEATURE] Clean up orphaned unused volume replicas after node failure (685) - @yangchiu @derekbit

[REFACTOR] Longhorn CRD structural schema (791) - @kaxing @jenting

[FEATURE] Longhorn with managed Kubernetes cluster(EKS, GKE, AKS) (1800) - @yangchiu @c3y1huang

[FEATURE] Longhorn should support multi-network K8s clusters (storage network) (2285) - @c3y1huang

[FEATURE] Extend CSI Snapshot support to Longhorn snapshot (2534) - @PhanLe1010 @cchien816

[BUG] helm upgrade won't apply customize default setting (2570) - @derekbit @cchien816

[POC] Rebuild logical volumes/replicas with SPDK (3044) - @keithalucas

[FEATURE] CRD for snapshots (3144) - @PhanLe1010 @cchien816

[FEATURE] Allowing users to download backing images (3155) - @shuo-wu

[FEATURE] CRD version webhook for longhorn.io/v1beta1 and longhorn.io/v1beta2 (3265) - @kaxing @derekbit @jenting

[FEATURE] Builds for IBM Z (3425) - @Anarkis

[FEATURE] Support operations on custom resources directly instead of Longhorn REST APIs only (3562) - @kaxing @derekbit

[FEATURE] mTLS authentication for manager <-> instance-manager communication (3839) - @joshimoo @cchien816

Enhancements

[FEATURE] Include a NetworkPolicy to secure communication inside of Longhorn (1805) - @kaxing @PhanLe1010

[BUG] Remove hard-coded namespace in uninstall controller (2034) - @c3y1huang

[FEATURE] Allow use of existing private registry secret (2157) - @yangchiu @c3y1huang

[FEATURE] instance-manager compatibility with Cluster Autoscaler (2203) - @kaxing @c3y1huang

[FEATURE] Recurring snapshot/backup job supports imagePullSecrets (2432) - @kaxing @jenting

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

[FEATURE] podAntiAffinity for csi-attacher/csi-provisioner/csi-resizer/csi-snapshotter (2894) - @kaxing @derekbit

[FEATURE] Introduce SUSE BCI based component images (2939) - @meldafrawi

[FEATURE] Prometheus metrics for backups (2940) - @meldafrawi

[FEATURE] Match type of status.conditions of nodes.longhorn.io to other k8s objects (2955) - @derekbit @khushboo-rancher

[TASK] Publish Longhorn stable releases (3014) - @PhanLe1010 @smallteeths @cchien816

[FEATURE] Mutating/Validating admission webhook (3241) - @kaxing @jenting

[FEATURE] Expect LH has a lightweight solution for volume failure like "insufficient storage" (3734) - @PhanLe1010

[FEATURE] Longhorn UI supports orphan CRs management (3826) - @yangchiu @derekbit @smallteeths

[FEATURE] Make Longhorn UI Listen Binding configurable (3829) - @yangchiu @smallteeths

Improvement

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Improve communication between components (GRPC) (2821) - @joshimoo

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @cchien816

[IMPROVEMENT] Upgrade path improvement (3087) - @kaxing @shuo-wu

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[IMPROVEMENT] longhorn protocol buffer namespace conflict (3343) - @kaxing @jenting

[IMPROVEMENT] Add network policy for longhorn webhook server (3513) - @kaxing @jenting

[IMPROVEMENT] The longhorn manager should wait the longhorn webhook server ready (3523) - @kaxing @jenting

[IMPROVEMENT] Add default replica auto balance to disabled for the existing volume (3524) - @derekbit @khushboo-rancher

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Implement gRPC proxy in IM to replace MGR engine binary invocation (3546) - @c3y1huang

[IMPROVEMENT] set default non-root user for all longhorn component images (3549) - @jenting

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @cchien816

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

[IMPROVEMENT] Bump Longhorn k8s dependency library to a newer version (3720) - @PhanLe1010

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @cchien816

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Add doc for the Orphan replica directory and the following improvement (3853) - @derekbit

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @cchien816

[IMPROVEMENT] Improve the error message from API (3956) - @derekbit @cchien816

[IMPROVEMENT] Refactor to use dedicated goroutine for proxy client handling (3967) - @c3y1huang

[IMPROVEMENT] Remove proxy server for instance manager replica pod (3968) - @c3y1huang

[IMPROVEMENT] Launch a general file managing and syncing service (3970) - @shuo-wu

[IMPROVEMENT] Implement TLS to proxy client (3975) - @c3y1huang

[IMPROVEMENT] Send credential to process server via request (3977) - @c3y1huang

[IMPROVEMENT] Consolidate command line and proxy implementations (3981) - @c3y1huang

Performance

[BUG] Backup created in prior release takes considerable time to show up on the backup page first time. (2902) - @kaxing @jenting @khushboo-rancher

[BUG] Scalability issue of volumes in Longhorn (3083) - @kaxing @PhanLe1010

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

Stability

[BUG] Unlimited volume replica rebuilding causes system unstable like latency of etcd access (2460) - @shuo-wu @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Error message is not seen on the UI when the backup to minio fails (1268) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Name of PVC overlaps on Firefox (1598) - @smallteeths @cchien816

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] Not all informer registered cacheSync function (2671) - @jenting @khushboo-rancher

[BUG] Volume operations take long time during automatic upgrading the engines in a big cluster (2697) - @kaxing @PhanLe1010

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Track terminating pods in the volume workload status (2874) - @joshimoo @khushboo-rancher

[BUG] NFS driver: creates multiple 1st volume backup and immediately list backup volume names sometimes fails (2926) - @kaxing @jenting

[BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX. (2946) - @joshimoo @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Backups don't work after upgrading 1.1.2 to 1.2.0 (2960) - @jenting @cchien816

[BUG] Error when upgrading from 1.1.2 to 1.2.0 - Operation cannot be fulfilled on volumes.longhorn.io "pvc-edf41777-589d-4806-baca-b91d0a6c0d3c": the object has been modified; please apply your changes to the latest version and try again (2961) - @kaxing @c3y1huang

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo

[BUG] RWX prior existing PV/PVC that were marked as XFS will now fail to mount, since we only supported EXT4 previously so we ignored users choice (2991) - @joshimoo @khushboo-rancher

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @cchien816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Recurring jobs showing incorrectly (2996) - @kaxing @smallteeths

[BUG] No any implication for backup being pulling from backupstore (3023) - @kaxing @smallteeths

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @cchien816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Longhorn 1.2.0 unable to handle instance-manager failure (3070) - @PhanLe1010 @cchien816

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] fatal error: concurrent map writes (3077) - @derekbit @cchien816

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] Restoring backup always point to the latest backup (3111) - @kaxing @joshimoo @PhanLe1010 @shuo-wu

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Volume API doesn't have the workloadStatus updated (3182) - @jenting

[BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @c3y1huang @cchien816

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] rwx volume fail to attach (3193) - @shuo-wu @khushboo-rancher

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] Tests related to backup failed on nightly test run (3216) - @jenting @khushboo-rancher

[BUG] Tooltip of a backup taken shows 0 size if a snapshot is taken just before it (3228) - @kaxing @jenting

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] YAML in Recurring Snapshots and Backups page is out of format (3256) - @c3y1huang

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Show snapshot in different color when backup fails (3394) - @smallteeths

[BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher

[BUG] Can't delete engineimage after upgrade from v1.2.2 to master-head (Dec 15, 2021) (3407) - @derekbit @jenting

[BUG] cache.DeletedFinalStateUnknown conversion uses pointer type (3417) - @joshimoo

[BUG] Upgrade failed through Rancher (2.5.5) UI from 1.2.0 -> 1.2.2 (3424) -

[BUG] Failed to upgrade from v1.2.3 with volumes uses backing image to master branch (3426) - @kaxing @derekbit

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Recurring backup not work on detached volume (master-head) (3470) - @c3y1huang @khushboo-rancher @cchien816

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[BUG] RecurringJob should validate retain field is <= 50 like the UI does (3493) - @kaxing @jenting

[BUG] Volume cloning feature is broken due to CRDs schema in master-head 01/06/2022 (3508) - @PhanLe1010 @khushboo-rancher

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang @khushboo-rancher

[BUG] Error while saving Node Disk setting when there is no tag (3538) - @kaxing @derekbit

[BUG] test_setting_priority_class fails on master-head with error (3548) - @derekbit

[BUG] Capitalization on the website for the blog and KB should be consistent (3573) - @derekbit

[BUG] Backup in error state can not be deleted (3620) - @derekbit @cchien816

[BUG] Volume fails to scheduled if while creating it couldn't be scheduled. (3626) - @yangchiu @PhanLe1010

[BUG] Unable to finish install or upgrade when Failed to list *v1beta2.Node: v1beta2.NodeList.Items (3631) - @kaxing @derekbit

[BUG] Statefulset pod get data input/output error after delete instance manager (3648) -

[BUG] Replica auto-balance tests fails in regression (3649) - @c3y1huang @cchien816

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Upgrade from v1.2.x to master-head fails if backupTarget exists (3677) - @yangchiu @derekbit

[BUG] Volume not become scheduling after change over provisioning form 0 to 100 (3678) - @PhanLe1010 @cchien816

[BUG] Volume fails to schedule when set Eviction Requested from true to false on node which contains replica (3679) - @kaxing @PhanLe1010

[BUG] When Harvester delete an uploading image, the corresponding LH temp file is not deleted (3682) - @shuo-wu

[BUG] Longhorn webhooks fail to deploy on hardened cluster (3701) - @kaxing @derekbit

[BUG] longhornio/longhorn-manager:master-head currently CrashLoopBackOff-ing (3716) -

[BUG] Data migration sample YAML moves data to the subfolder (3787) - @yangchiu @c3y1huang

[BUG] go.mod and go.sum out of date (3796) - @derekbit

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] Volume can't detach when "Concurrent Replica Rebuild Per Node Limit = 0" and set node evicted from True to False (3820) - @PhanLe1010

[BUG] Longhorn default StorageClass is not always synced with the ConfigMap (3821) - @yangchiu @shuo-wu

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @shuo-wu

[BUG] BackingImageManager version request leaked grpc connection (3838) - @kaxing @derekbit

[BUG] PVC create from volumsanpshot stuck at pending status(type=snap) (3860) - @kaxing @derekbit

[BUG] liblonghorn sends integer fields in longhorn protocol as host endianness and longhorn-engine always expects litte endian (3875) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @cchien816

[BUG] Volume created from cloned PVC will stuck at Not Ready status (3886) - @PhanLe1010 @cchien816

[BUG] UI pop error if long volume name used (when > 40 characters) (3915) - @derekbit @cchien816

[BUG] Instance manager keep restarting (3922) - @joshimoo @cchien816

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths @cchien816

[BUG] Event Log on Dashboard is always empty on s390x with v1.3.0-preview1 (3958) - @mantissahz @smallteeths @cchien816

[BUG] Command longhorn-instance-manager process won't work without TLS key (3966) - @joshimoo @cchien816

Misc

[BUG] After upgrade to 1.1.0 some old managers still running (2144) - @derekbit

[DOC] Change VolumeSnapshotClass API version to v1 from v1beta1 in documentation (2214) -

[REFACTORING] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[TASK] Deprecate volume spec recurringJobs and storageClass recurringJobs field (2865) - @c3y1huang

[TASK] Update document to reflect on Volume Attachment Recovery Policy feature deprecation (2954) - @jenting

[TEST] Add test case in e2e integration test for fsgroup support (2967) - @khushboo-rancher

[DOC] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[UI] Notify users about newer stable versions (3032) - @smallteeths @cchien816

[TASK] Extend OS support matrix (3085) - @meldafrawi

[TASK] Remove the feature that skip taking backup/snapshot when the volume head is empty in recurring backup jobs (3128) - @derekbit @khushboo-rancher

[TASK] Change the CSI sidecar images from upstream repo to Longhorn repo (3258) - @PhanLe1010 @khushboo-rancher

[BUG] Test_settings fails in the integration test (3280) - @jenting

[TASK] Generate multiple CRD versions (3295) - @kaxing @derekbit @jenting

[TASK] The CSI resizer may not work when the Job pod status is Completed (3303) - @kaxing @shuo-wu

[DOC] Failed auto installation of iscsi on Debian 11 bullseye (3313) - @derekbit

[TASK] Handle the CRD validation error (3352) - @kaxing @derekbit

[TASK] Restore to a brand new cluster that does not contain anything (3367) - @shuo-wu @khushboo-rancher

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

[TASK] Create golang API for mounting NVMeoF targets (3602) - @keithalucas

[TASK] Improve the doc for volume actual size and space usage (3627) - @derekbit @shuo-wu

[DOC] Update RHEL instructions for open-iscsi (3630) - @keithalucas

[TASK] Need warning message on UI when deleting the last recurring job of the group (3651) - @yangchiu @smallteeths

[TASK] Create a migration path from deprecated Rancher catalog to Rancher marketplace (3714) - @shuo-wu

[DOC] Alert LonghornNodeDown example in documentation doesn't work (3729) - @kaxing @jenting

[TASK] Add readme document for the admission webhook (3752) - @derekbit @cchien816

[TASK] Refactor the Logic for Node Monitoring (3775) - @derekbit

[TASK] Move secscan nightly tests to aws (3779) - @meldafrawi

[TASK] Improve the doc for troubleshooting NFS backuptarget (3825) - @mantissahz

[DOC] Add cluster autoscaler support (3884) - @c3y1huang

[TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @cchien816

[TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version (3891) - @joshimoo

[TASK] Update the volume size related doc after introducing the snapshot prune feature (3894) - @shuo-wu

[TASK] add finalizer to resource in mutating webhook (3918) - @derekbit

[DOC] mTLS authentication for manager <-> instance-manager communication (3953) - @joshimoo

Contributors

@Anarkis

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@ibrokethecloud

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@mantissahz

@meldafrawi

@shuo-wu

@smallteeths

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.3.0-preview1(May 6, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.0-preview1 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

For existing features, there are no deprecations or incompatible changes introduced in this release.

v1beta2 resource version is introduced with validating and mutating admission webhooks. v1beta2 will be the default resource version.

This release has a built-in CRD conversion webhook for v1beta1 compatibility. After upgrading from a previously supported version, all v1beta1 resources will be converted to v1beta2 but v1beta1 resources are still accessible by the conversion webhook.

Highlights

[FEATURE] Clean up orphaned unused volume replicas after node failure (685) - @yangchiu @derekbit

[REFACTOR] Longhorn CRD structural schema (791) - @kaxing @jenting

[FEATURE] Longhorn with managed Kubernetes cluster(EKS, GKE, AKS) (1800) - @kaxing @c3y1huang

[FEATURE] Longhorn should support multi-network K8s clusters (storage network) (2285) - @c3y1huang

[FEATURE] Extend CSI Snapshot support to Longhorn snapshot (2534) - @PhanLe1010 @cchien816

[POC] Rebuild logical volumes/replicas with SPDK (3044) - @keithalucas

[FEATURE] CRD for snapshots (3144) - @PhanLe1010

[FEATURE] Allowing users to download backing images (3155) - @shuo-wu

[FEATURE] CRD version webhook for longhorn.io/v1beta1 and longhorn.io/v1beta2 (3265) - @kaxing @derekbit @jenting

[FEATURE] Builds for IBM Z (3425) - @Anarkis

[FEATURE] Support operations on custom resources directly instead of Longhorn REST APIs only (3562) - @kaxing @derekbit

[FEATURE] mTLS authentication for manager <-> instance-manager communication (3839) - @joshimoo

Enhancements

[FEATURE] Include a NetworkPolicy to secure communication inside of Longhorn (1805) - @kaxing @PhanLe1010

[BUG] Remove hard-coded namespace in uninstall controller (2034) - @c3y1huang

[FEATURE] Allow use of existing private registry secret (2157) - @yangchiu @c3y1huang

[FEATURE] instance-manager compatibility with Cluster Autoscaler (2203) - @kaxing @c3y1huang

[FEATURE] Recurring snapshot/backup job supports imagePullSecrets (2432) - @kaxing @jenting

[BUG] helm upgrade won't apply customize default setting (2570) - @kaxing @shuo-wu

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

[FEATURE] podAntiAffinity for csi-attacher/csi-provisioner/csi-resizer/csi-snapshotter (2894) - @kaxing @derekbit

[FEATURE] Introduce SUSE BCI based component images (2939) - @meldafrawi

[FEATURE] Prometheus metrics for backups (2940) - @meldafrawi

[FEATURE] Match type of status.conditions of nodes.longhorn.io to other k8s objects (2955) - @derekbit @khushboo-rancher

[TASK] Publish Longhorn stable releases (3014) - @PhanLe1010 @smallteeths @cchien816

[FEATURE] Mutating/Validating admission webhook (3241) - @kaxing @jenting

[FEATURE] Expect LH has a lightweight solution for volume failure like "insufficient storage" (3734) - @PhanLe1010

[FEATURE] Longhorn UI supports orphan CRs management (3826) - @yangchiu @derekbit @smallteeths

[FEATURE] Make Longhorn UI Listen Binding configurable (3829) - @yangchiu @smallteeths

Improvement

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

[IMPROVEMENT] Pre-condition check required for the snapshot purge (2777) - @derekbit

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Improve communication between components (GRPC) (2821) - @joshimoo

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @cchien816

[IMPROVEMENT] Upgrade path improvement (3087) - @kaxing @shuo-wu

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[IMPROVEMENT] longhorn protocol buffer namespace conflict (3343) - @kaxing @jenting

[IMPROVEMENT] Add network policy for longhorn webhook server (3513) - @kaxing @jenting

[IMPROVEMENT] The longhorn manager should wait the longhorn webhook server ready (3523) - @kaxing @jenting

[IMPROVEMENT] Add default replica auto balance to disabled for the existing volume (3524) - @derekbit @khushboo-rancher

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit

[IMPROVEMENT] Implement gRPC proxy in IM to replace MGR engine binary invocation (3546) - @c3y1huang

[IMPROVEMENT] set default non-root user for all longhorn component images (3549) - @jenting

[IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @cchien816

[IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu

[IMPROVEMENT] Bump Longhorn k8s dependency library to a newer version (3720) - @PhanLe1010

[IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @cchien816

[IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths

[IMPROVEMENT] Add doc for the Orphan replica directory and the following improvement (3853) - @derekbit

[IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @cchien816

Performance

[BUG] Backup created in prior release takes considerable time to show up on the backup page first time. (2902) - @kaxing @jenting @khushboo-rancher

[BUG] Scalability issue of volumes in Longhorn (3083) - @kaxing @PhanLe1010

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

[BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

Stability

[BUG] Unlimited volume replica rebuilding causes system unstable like latency of etcd access (2460) - @shuo-wu @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Error message is not seen on the UI when the backup to minio fails (1268) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Name of PVC overlaps on Firefox (1598) - @smallteeths @cchien816

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] Not all informer registered cacheSync function (2671) - @jenting @khushboo-rancher

[BUG] Volume operations take long time during automatic upgrading the engines in a big cluster (2697) - @kaxing @PhanLe1010

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Track terminating pods in the volume workload status (2874) - @joshimoo @khushboo-rancher

[BUG] NFS driver: creates multiple 1st volume backup and immediately list backup volume names sometimes fails (2926) - @kaxing @jenting

[BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX. (2946) - @joshimoo @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Backups don't work after upgrading 1.1.2 to 1.2.0 (2960) - @jenting @cchien816

[BUG] Error when upgrading from 1.1.2 to 1.2.0 - Operation cannot be fulfilled on volumes.longhorn.io "pvc-edf41777-589d-4806-baca-b91d0a6c0d3c": the object has been modified; please apply your changes to the latest version and try again (2961) - @kaxing @c3y1huang

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo

[BUG] RWX prior existing PV/PVC that were marked as XFS will now fail to mount, since we only supported EXT4 previously so we ignored users choice (2991) - @joshimoo @khushboo-rancher

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @cchien816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Recurring jobs showing incorrectly (2996) - @kaxing @smallteeths

[BUG] No any implication for backup being pulling from backupstore (3023) - @kaxing @smallteeths

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @cchien816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Longhorn 1.2.0 unable to handle instance-manager failure (3070) - @PhanLe1010 @cchien816

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] fatal error: concurrent map writes (3077) - @derekbit @cchien816

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] Restoring backup always point to the latest backup (3111) - @kaxing @joshimoo @PhanLe1010 @shuo-wu

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Volume API doesn't have the workloadStatus updated (3182) - @jenting

[BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @c3y1huang @cchien816

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] rwx volume fail to attach (3193) - @shuo-wu @khushboo-rancher

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] Tests related to backup failed on nightly test run (3216) - @jenting @khushboo-rancher

[BUG] Tooltip of a backup taken shows 0 size if a snapshot is taken just before it (3228) - @kaxing @jenting

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] YAML in Recurring Snapshots and Backups page is out of format (3256) - @c3y1huang

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Show snapshot in different color when backup fails (3394) - @smallteeths

[BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher

[BUG] Can't delete engineimage after upgrade from v1.2.2 to master-head (Dec 15, 2021) (3407) - @derekbit @jenting

[BUG] cache.DeletedFinalStateUnknown conversion uses pointer type (3417) - @joshimoo

[BUG] Upgrade failed through Rancher (2.5.5) UI from 1.2.0 -> 1.2.2 (3424) -

[BUG] Failed to upgrade from v1.2.3 with volumes uses backing image to master branch (3426) - @kaxing @derekbit

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Recurring backup not work on detached volume (master-head) (3470) - @c3y1huang @khushboo-rancher @cchien816

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[BUG] RecurringJob should validate retain field is <= 50 like the UI does (3493) - @kaxing @jenting

Volume cloning feature is broken due to CRDs schema in master-head 01/06/2022 (3508) - @PhanLe1010 @khushboo-rancher

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang @khushboo-rancher

[BUG] Error while saving Node Disk setting when there is no tag (3538) - @kaxing @derekbit

[BUG] test_setting_priority_class fails on master-head with error (3548) - @derekbit

[BUG] Capitalization on the website for the blog and KB should be consistent (3573) - @derekbit

[BUG] Backup in error state can not be deleted (3620) - @derekbit @cchien816

[BUG] Volume fails to scheduled if while creating it couldn't be scheduled. (3626) - @PhanLe1010

[BUG] Unable to finish install or upgrade when Failed to list *v1beta2.Node: v1beta2.NodeList.Items (3631) - @kaxing @derekbit

[BUG] Statefulset pod get data input/output error after delete instance manager (3648) -

[BUG] Replica auto-balance tests fails in regression (3649) - @c3y1huang @cchien816

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Upgrade from v1.2.x to master-head fails if backupTarget exists (3677) - @yangchiu @derekbit

[BUG] Volume not become scheduling after change over provisioning form 0 to 100 (3678) - @PhanLe1010 @cchien816

[BUG] Volume fails to schedule when set Eviction Requested from true to false on node which contains replica (3679) - @kaxing @PhanLe1010

[BUG] When Harvester delete an uploading image, the corresponding LH temp file is not deleted (3682) - @shuo-wu

[BUG] Longhorn webhooks fail to deploy on hardened cluster (3701) - @kaxing @derekbit

[BUG] longhornio/longhorn-manager:master-head currently CrashLoopBackOff-ing (3716) -

[BUG] Data migration sample YAML moves data to the subfolder (3787) - @yangchiu @c3y1huang

[BUG] go.mod and go.sum out of date (3796) - @derekbit

[BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths

[BUG] Volume can't detach when "Concurrent Replica Rebuild Per Node Limit = 0" and set node evicted from True to False (3820) - @PhanLe1010

[BUG] Longhorn default StorageClass is not always synced with the ConfigMap (3821) - @yangchiu @shuo-wu

[BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas

[BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @shuo-wu

[BUG] BackingImageManager version request leaked grpc connection (3838) - @kaxing @derekbit

[BUG] PVC create from volumsanpshot stuck at pending status(type=snap) (3860) - @kaxing @derekbit

[BUG] liblonghorn sends integer fields in longhorn protocol as host endianness and longhorn-engine always expects litte endian (3875) - @keithalucas

[BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @cchien816

[BUG] Volume created from cloned PVC will stuck at Not Ready status (3886) -

[BUG] UI pop error if long volume name used (when > 40 characters) (3915) - @derekbit @cchien816

[BUG] Instance manager keep restarting (3922) - @joshimoo @cchien816

[BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths

Misc

[BUG] After upgrade to 1.1.0 some old managers still running (2144) - @derekbit

[DOC] Change VolumeSnapshotClass API version to v1 from v1beta1 in documentation (2214) -

[REFACTOR] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[TASK] Deprecate volume spec recurringJobs and storageClass recurringJobs field (2865) - @c3y1huang

[TASK] Update document to reflect on Volume Attachment Recovery Policy feature deprecation (2954) - @jenting

[TEST] Add test case in e2e integration test for fsgroup support (2967) - @khushboo-rancher

[DOC] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[UI] Notify users about newer stable versions (3032) - @smallteeths @cchien816

[TASK] Extend OS support matrix (3085) - @meldafrawi

[TASK] Remove the feature that skip taking backup/snapshot when the volume head is empty in recurring backup jobs (3128) - @derekbit @khushboo-rancher

[TASK] Change the CSI sidecar images from upstream repo to Longhorn repo (3258) - @PhanLe1010 @khushboo-rancher

[BUG] Test_settings fails in the integration test (3280) - @jenting

[TASK] Generate multiple CRD versions (3295) - @kaxing @derekbit @jenting

[TASK] The CSI resizer may not work when the Job pod status is Completed (3303) - @kaxing @shuo-wu

Failed auto installation of iscsi on Debian 11 bullseye (3313) - @derekbit

[TASK] Handle the CRD validation error (3352) - @kaxing @derekbit

[TASK] Restore to a brand new cluster that does not contain anything (3367) - @shuo-wu @khushboo-rancher

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

[TASK] Create golang API for mounting NVMeoF targets (3602) - @keithalucas

[TASK] Improve the doc for volume actual size and space usage (3627) - @derekbit @shuo-wu

[DOC] Update RHEL instructions for open-iscsi (3630) - @keithalucas

[TASK] Need warning message on UI when deleting the last recurring job of the group (3651) - @yangchiu @smallteeths

[TASK] Create a migration path from deprecated Rancher catalog to Rancher marketplace (3714) - @shuo-wu

[DOC] Alert LonghornNodeDown example in documentation doesn't work (3729) - @kaxing @jenting

[TASK] Add readme document for the admission webhook (3752) - @derekbit @cchien816

[TASK] Collect and analyze reasons for unstable k8s cluster when running longhorn-tests on Jenkins (3770) - @yangchiu

[TASK] Refactor the Logic for Node Monitoring (3775) - @derekbit

[TASK] Move secscan nightly tests to aws (3779) - @meldafrawi

[TASK] Improve the doc for troubleshooting NFS backuptarget (3825) - @derekbit

[DOC] Add cluster autoscaler support (3884) - @c3y1huang

[TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @cchien816

[TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version (3891) - @joshimoo

[TASK] Update the volume size related doc after introducing the snapshot prune feature (3894) - @shuo-wu

[TASK] add finalizer to resource in mutating webhook (3918) - @derekbit

[TASK] Release 1.3.0 (3938) - @innobead

Contributors

@Anarkis

@PhanLe1010

@c3y1huang

@cchien816

@derekbit

@ibrokethecloud

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

@yangchiu

Source code(tar.gz)
Source code(zip)
v1.2.4(Mar 17, 2022)
Release Note

v1.2.4 released! 🎆

This release introduces bug fixes as described below including stability, scalability, performance, security hardening, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecated or incompatible changes are introduced in this release.

Performance

[BUG] Scalability issue in Longhorn (3083) - @kaxing @PhanLe1010

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

Stability

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

Bugs

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Page navigation icon has moved up in v1.2.x - UI (3676) - @smallteeths

Misc

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[BUG] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[IMPROVEMENT] Upgrade path improvement (3087) - @meldafrawi @shuo-wu

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

Contributors

@PhanLe1010

@c3y1huang

@innobead

@jenting

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.2.4-rc1(Mar 4, 2022)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.2.4-rc1 released! 🎆

This release introduces bug fixes as described below including stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.1./v1.2. is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecated or incompatible changes are introduced in this release.

Performance

[BUG] Scalability issue in Longhorn (3083) - @kaxing @PhanLe1010

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

Performance

[BUG] Scalability issue in Longhorn (3083) - @kaxing @PhanLe1010

[IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths

[BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

Stability

[IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu

Bugs

[BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010

[BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths

[BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher

[BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting

[BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher

[BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher

[BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu

[Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher

[BUG] Recurring Job delete not working correctly (3517) - @c3y1huang

[BUG] Backup error happen to volume with older engine image (3621) - @jenting

[BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher

[BUG] Page navigation icon has moved up in v1.2.x - UI (3676) - @smallteeths

Misc

[IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting

[BUG] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting

[IMPROVEMENT] Upgrade path improvement (3087) - @meldafrawi @shuo-wu

[IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths

[DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting

Contributors

@PhanLe1010

@c3y1huang

@innobead

@jenting

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.2.3(Dec 17, 2021)
Release Note

Please read the install/upgrade notes before installing/upgrading to this Longhorn version.

v1.2.3 released! 🎆

This release introduces bug fixes as described below including security, stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Security Fixes for Vulnerabilities

CVE-2021-36779: Host operations allowed in privileged Longhorn managed pods

CVE-2021-36780: Unauthorized data access from replicas through vulnerable instance manager pods

For more details, see each issue and security advisories.

CVE-2021-36779

CVE-2021-36780

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecated or incompatible changes are introduced in this release.

Enhancements

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

Performance

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @chanow816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher

[BUG] Show snapshot in different color when backup fails (3394) - @smallteeths

[BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher

Misc

[REFACTOR] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @chanow816

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

Contributors

@PhanLe1010

@chanow816

@derekbit

@innobead

@jenting

@joshimoo

@kaxing

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.1.3(Dec 17, 2021)
Release Note

Please read the install/upgrade notes before installing/upgrading to this Longhorn version.

v1.1.3 released! 🎆

This release introduces bug fixes as described below including security, stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Security Fixes for Vulnerabilities

CVE-2021-36779: Host operations allowed in privileged Longhorn managed pods

CVE-2021-36780: Unauthorized data access from replicas through vulnerable instance manager pods

For more details, see each issue and security advisories.

CVE-2021-36779

CVE-2021-36780

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Follow the upgrade instructions here.

Deprecation & Incompatibilities

No deprecated or incompatible changes are introduced in this release.

Highlights

[IMPROVEMENT] Enhance Longhorn data plane on low performance environment (spinning disk, 1GBps network, low CPU etc) (2206) - @keithalucas @khushboo-rancher

Performance

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Investigate engine integration test failure - restore_with_frontend: can only restore backup from replica in mode RW, got ERR (1628) - @keithalucas

[BUG] High CPU utilization for Longhorn manager sometimes due to exhaustion of all the available sockets because of socket leaking log stream (2778) - @meldafrawi @PhanLe1010

[BUG] longhorn-engine image no longer contains longhorn-instance manager (2796) - @innobead @khushboo-rancher

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Longhorn-csi-plugin pods restart because the Longhorn client 10 secs timeout (2816) - @jenting

[BUG] High CPU utilization for Longhorn manager and replica instance manager sometimes due to overwhelming of number of backupstatus in engine (2818) - @joshimoo @khushboo-rancher

[BUG] fix instance manager grpc connection leak (VersionGet, ProcessLog, ProcessWatch) (2824) - @joshimoo

[BUG] Datastore::GetEngine returns raw cached object which is then modified (2827) - @joshimoo @khushboo-rancher

[BUG] Node deletion leads volume to get stuck in attaching state (2848) - @joshimoo @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding fails and eventually pass after few attempts (2849) - @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding gets triggered if network bandwidth is restricted below 80mbit (2882) - @keithalucas

[BUG] Instance managers keep terminating and getting created while uninstallation is in progress (2919) - @meldafrawi @shuo-wu @khushboo-rancher

[BUG] DR volume does not continue to restore after node reboot (2920) - @c3y1huang @shuo-wu

[BUG] Power off node during backup volume restore does not continue restore (2929) - @kaxing @c3y1huang @shuo-wu @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @chanow816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Fail to live upgrade to v1.2.x or v1.1.3 (3052) - @shuo-wu @khushboo-rancher

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] [v1.1.3] Longhorn upgrade from v1.1.2 to v1.1.3-rc1 failed (3297) - @PhanLe1010 @khushboo-rancher

[BUG] V1 Chart install V1.1.3-rc2 fail (3356) - @PhanLe1010 @chanow816

[BUG] Using a backing image created from a YAML file will lead to NPE (3379) - @kaxing @shuo-wu

Misc

[UI] Notify users about newer stable versions (3032) - @smallteeths @chanow816

[IMPROVEMENT] Longhorn v1.1.3 and maximum supported Kubernetes version (3319) - @kaxing @jenting

Contributors

@PhanLe1010

@c3y1huang

@chanow816

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.2.3-rc2(Dec 10, 2021)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.2.3-rc2 released! 🎆

This release introduces bug fixes as described below including stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.1./v1.2. is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecated or incompatible changes are introduced in this release.

Enhancements

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

Performance

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo @khushboo-rancher

[BUG] Volume UI displays only the last backup when using the recurring job (2997) - @kaxing @jenting

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @chanow816

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010

Misc

[REFACTOR] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @kaxing @jenting

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

Contributors

@PhanLe1010

@chanow816

@derekbit

@innobead

@jenting

@joshimoo

@kaxing

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.1.3-rc3(Dec 10, 2021)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.1.3-rc3 released! 🎆

This release introduces bug fixes as described below including stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.0./v1.1. is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

No deprecated or incompatible changes are introduced in this release.

Highlights

[IMPROVEMENT] Enhance Longhorn data plane on low performance environment (spinning disk, 1GBps network, low CPU etc) (2206) - @keithalucas @khushboo-rancher

Performance

[BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Investigate engine integration test failure - restore_with_frontend: can only restore backup from replica in mode RW, got ERR (1628) - @keithalucas

[BUG] High CPU utilization for Longhorn manager sometimes due to exhaustion of all the available sockets because of socket leaking log stream (2778) - @meldafrawi @PhanLe1010

[BUG] longhorn-engine image no longer contains longhorn-instance manager (2796) - @innobead @khushboo-rancher

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Longhorn-csi-plugin pods restart because the Longhorn client 10 secs timeout (2816) - @jenting

[BUG] High CPU utilization for Longhorn manager and replica instance manager sometimes due to overwhelming of number of backupstatus in engine (2818) - @joshimoo @khushboo-rancher

[BUG] fix instance manager grpc connection leak (VersionGet, ProcessLog, ProcessWatch) (2824) - @joshimoo

[BUG] Datastore::GetEngine returns raw cached object which is then modified (2827) - @joshimoo @khushboo-rancher

[BUG] Node deletion leads volume to get stuck in attaching state (2848) - @joshimoo @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding fails and eventually pass after few attempts (2849) - @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding gets triggered if network bandwidth is restricted below 80mbit (2882) - @keithalucas

[BUG] Instance managers keep terminating and getting created while uninstallation is in progress (2919) - @meldafrawi @shuo-wu @khushboo-rancher

[BUG] DR volume does not continue to restore after node reboot (2920) - @c3y1huang @shuo-wu

[BUG] Power off node during backup volume restore does not continue restore (2929) - @kaxing @c3y1huang @shuo-wu @khushboo-rancher

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @chanow816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Fail to live upgrade to v1.2.x or v1.1.3 (3052) - @shuo-wu @khushboo-rancher

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher

[BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010

[BUG] [v1.1.3] Longhorn upgrade from v1.1.2 to v1.1.3-rc1 failed (3297) - @PhanLe1010 @khushboo-rancher

[BUG] V1 Chart install V1.1.3-rc2 fail (3356) - @PhanLe1010 @chanow816

[BUG] Using a backing image created from a YAML file will lead to NPE (3379) - @kaxing @shuo-wu

Misc

[UI] Notify users about newer stable versions (3032) - @smallteeths @chanow816

[IMPROVEMENT] Longhorn v1.1.3 and maximum supported Kubernetes version (3319) - @kaxing @jenti

Contributors

@PhanLe1010

@c3y1huang

@chanow816

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.2.3-rc1(Nov 25, 2021)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.2.3-rc1 released! 🎆

This release introduces bug fixes as described below including stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.1./v1.2. is not supported until the GA release. When available, follow the upgrade instructions here. here.

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit

[BUG] Volume UI displays only the last backup (2997) - @jenting

[BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting

[BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @PhanLe1010

[BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths

[BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit

[BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher

[BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @shuo-wu

[BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher

[BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @PhanLe1010

[BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths

[BUG] Volume crash during migrating (3206) - @shuo-wu

[BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting

[Backport] [BUG] Support bundle missed newly added Longhorn CRDs (3145) - @jenting @khushboo-rancher

[BACKPORT][BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (3218) - @kaxing @derekbit

[Backuport][BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3244) - @kaxing

[BACKPORT][BUG] Validate S3 input to make sure they don't contain trailing newline or space (3266) - @kaxing @jenting

[BACKPORT][BUG] Instance process is still running when the corresponding engine/replica CR is gone (3287) - @shuo-wu

Misc

[IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher

[REFACTOR] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher

[IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting

[IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths

[IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher

[IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths

[Backport] [BUG] Backing Image upload failed when access through Rancher UI (3138) - @kaxing @smallteeths

[Backport] [BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3154) - @kaxing @derekbit

[BACKPORT] [BUG] Restored volume wrongly restarts restoration during migration (3223) - @meldafrawi @shuo-wu

[Backport] [FEATURE] move filesystem expansion into the csi driver (3227) - @joshimoo @khushboo-rancher

[Backport] Support Filesystem expansion for encrypted volumes (3240) - @joshimoo @khushboo-rancher

Contributors

@derekbit

@innobead

@jenting

@joshimoo

@kaxing

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.1.3-rc2(Nov 25, 2021)
DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.1.3-rc2 released! 🎆

This release introduces bug fixes as described below including stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.0./v1.1. is not supported until the GA release. When available, follow the upgrade instructions here. here.

Highlights

[IMPROVEMENT] Enhance Longhorn data plane on low performance environment (spinning disk, 1GBps network, low CPU etc) (2206) - @keithalucas @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Investigate engine integration test failure - restore_with_frontend: can only restore backup from replica in mode RW, got ERR (1628) - @keithalucas

[BUG] High CPU utilization for Longhorn manager sometimes due to exhaustion of all the available sockets because of socket leaking log stream (2778) - @meldafrawi @PhanLe1010

[BUG] longhorn-engine image no longer contains longhorn-instance manager (2796) - @innobead @khushboo-rancher

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Longhorn-csi-plugin pods restart because the Longhorn client 10 secs timeout (2816) - @jenting

[BUG] High CPU utilization for Longhorn manager and replica instance manager sometimes due to overwhelming of number of backupstatus in engine (2818) - @joshimoo @khushboo-rancher

[BUG] fix instance manager grpc connection leak (VersionGet, ProcessLog, ProcessWatch) (2824) - @joshimoo

[BUG] Datastore::GetEngine returns raw cached object which is then modified (2827) - @joshimoo @khushboo-rancher

[BUG] Node deletion leads volume to get stuck in attaching state (2848) - @joshimoo @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding fails and eventually pass after few attempts (2849) - @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding gets triggered if network bandwidth is restricted below 80mbit (2882) - @keithalucas

[BUG] Instance managers keep terminating and getting created while uninstallation is in progress (2919) - @meldafrawi @shuo-wu @khushboo-rancher

[BUG] DR volume does not continue to restore after node reboot (2920) - @c3y1huang @shuo-wu

[BUG] Power off node during backup volume restore does not continue restore (2929) - @kaxing @c3y1huang @shuo-wu @khushboo-rancher

[BUG] Workload with RWX volume cannot recover when Kubelet restarts (2933) - @joshimoo

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @chanow816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Fail to live upgrade to v1.2.x or v1.1.3 (3052) - @shuo-wu @khushboo-rancher

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @PhanLe1010

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting

[Backport][BUG] Disk eviction not doing anything (3009) - @shuo-wu @khushboo-rancher

[Backport][BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (3010) - @PhanLe1010 @khushboo-rancher

[BACKPORT][BUG] Validate S3 input to make sure they don't contain trailing newline or space (3267) - @kaxing @jenting

[BACKPORT][BUG] Instance process is still running when the corresponding engine/replica CR is gone (3288) - @shuo-wu @khushboo-rancher

[BUG] [v1.1.3] Longhorn upgrade from v1.1.2 to v1.1.3-rc1 failed (3297) - @PhanLe1010 @khushboo-rancher

Misc

[UI] Notify users about newer stable versions (3032) - @smallteeths @chanow816

[Backport] consider backporting RWX fixes to v1.1.3 (3003) - @joshimoo

[Backport][BUG] Over-provisioning dosen't work properly with values smaller than 200% (3011) - @PhanLe1010 @khushboo-rancher

[Backport] [BUG] fsfreeze race condition (3127) - @joshimoo @khushboo-rancher

[IMPROVEMENT] Longhorn v1.1.3 and maximum supported Kubernetes version (3319) - @jenting

Contributors

@PhanLe1010

@c3y1huang

@chanow816

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.1.3-rc1(Nov 18, 2021)
Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.1.3-rc1 released! 🎆

This release introduces bug fixes as described below including stability, performance, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.0./v1.1. is not supported until the GA release. When available, follow the upgrade instructions here. here.

Highlights

[IMPROVEMENT] Enhance Longhorn data plane on low performance environment (spinning disk, 1GBps network, low CPU etc) (2206) - @keithalucas @khushboo-rancher

Bugs

[BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting

[BUG] Investigate engine integration test failure - restore_with_frontend: can only restore backup from replica in mode RW, got ERR (1628) - @keithalucas

[BUG] High CPU utilization for Longhorn manager sometimes due to exhaustion of all the available sockets because of socket leaking log stream (2778) - @meldafrawi @PhanLe1010

[BUG] longhorn-engine image no longer contains longhorn-instance manager (2796) - @innobead @khushboo-rancher

[BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher

[BUG] Longhorn-csi-plugin pods restart because the Longhorn client 10 secs timeout (2816) - @jenting

[BUG] High CPU utilization for Longhorn manager and replica instance manager sometimes due to overwhelming of number of backupstatus in engine (2818) - @joshimoo @khushboo-rancher

[BUG] fix instance manager grpc connection leak (VersionGet, ProcessLog, ProcessWatch) (2824) - @joshimoo

[BUG] Datastore::GetEngine returns raw cached object which is then modified (2827) - @joshimoo @khushboo-rancher

[BUG] Node deletion leads volume to get stuck in attaching state (2848) - @joshimoo @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding fails and eventually pass after few attempts (2849) - @PhanLe1010 @khushboo-rancher

[BUG] Replica rebuilding gets triggered if network bandwidth is restricted below 80mbit (2882) - @keithalucas

[BUG] Instance managers keep terminating and getting created while uninstallation is in progress (2919) - @meldafrawi @shuo-wu @khushboo-rancher

[BUG] DR volume does not continue to restore after node reboot (2920) - @c3y1huang @shuo-wu

[BUG] Power off node during backup volume restore does not continue restore (2929) - @kaxing @c3y1huang @shuo-wu @khushboo-rancher

[BUG] Workload with RWX volume cannot recover when Kubelet restarts (2933) - @joshimoo

[BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010

[BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010

[BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @chanow816

[BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu

[BUG] Fail to live upgrade to v1.2.x or v1.1.3 (3052) - @shuo-wu @khushboo-rancher

[BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @PhanLe1010

[BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher

[BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher

[BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu

[Backport][BUG] Disk eviction not doing anything (3009) - @shuo-wu @khushboo-rancher

[Backport][BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (3010) - @PhanLe1010 @khushboo-rancher

[BACKPORT-v1.1.x][BUG] Validate S3 input to make sure they don't contain trailing newline or space (3267) - @jenting

Misc

[UI] Notify users about newer stable versions (3032) - @smallteeths @chanow816

[Backport] consider backporting RWX fixes to v1.1.3 (3003) - @joshimoo

[Backport][BUG] Over-provisioning dosen't work properly with values smaller than 200% (3011) - @PhanLe1010 @khushboo-rancher

[Backport] [BUG] fsfreeze race condition (3127) - @joshimoo @khushboo-rancher

Contributors

@PhanLe1010

@c3y1huang

@chanow816

@innobead

@jenting

@joshimoo

@kaxing

@keithalucas

@khushboo-rancher

@meldafrawi

@shuo-wu

@smallteeths

Source code(tar.gz)
Source code(zip)
v1.2.2(Oct 8, 2021)
Release Note

Please read the install/upgrade notes before installing/upgrading to this Longhorn version.

v1.2.2 released! 🎆

This release introduces important bug fixes as described below. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

1. Please ensure your Kubernetes cluster is at least v1.18 before upgrading to Longhorn v1.2.2, because the supported Kubernetes version has been updated (>= v1.18) in v1.2. 2. After the upgrade, the recurring job settings of volumes will be migrated to new recurring job resources and the RecurringJobs field in the volume spec will be deprecated. [doc]

Follow the upgrade instructions here.

Misc

[Backport] Restoring backup always point to the latest backup (3114) - @kaxing @PhanLe1010

[Backport] [BUG] fsfreeze race condition (3126) - @joshimoo @khushboo-rancher

Contributors

@PhanLe1010

@innobead

@joshimoo

@kaxing

@khushboo-rancher

Source code(tar.gz)
Source code(zip)