Release Notes

Core Component Updates -

• Simplified the error logging with a stack trace and added user-friendly fail steps by @ispeakc0de in https://github.com/litmuschaos/litmus-go/pull/615

• Depreciated the Pumba and Powerfulseal libs by @ispeakc0de in https://github.com/litmuschaos/litmus-go/pull/613

• Enhanced the disk-fill experiment to use pod PID instead of container path by @avaakash in https://github.com/litmuschaos/litmus-go/pull/605

ChaosCenter(3.0.0-beta2) Updates -

• Adding multiple workloads and pod filtering by @ispeakc0de in https://github.com/litmuschaos/litmus/pull/3836

• Updated upgrade agent for 3.0.0-beta2 and minor fix with version checks by @amityt in https://github.com/litmuschaos/litmus/pull/3860

• Fixed analytics graph and empty retry option in probes by @amityt in https://github.com/litmuschaos/litmus/pull/3861

Litmusctl (0.16.0) -

• Made litmsuctl create scenario command independent of install-chaos-experiments step by @iamnicoj in https://github.com/litmuschaos/litmusctl/pull/114

• Enhanced Litmusctl to check for existence of workflowName before scheduling & add the same if not present already by @iamnicoj in https://github.com/litmuschaos/litmusctl/pull/117

Shoutouts!

Thanks to our existing and new contributors for this release @iamnicoj @amityt @Saranya-jena @ispeakc0de @avaakash

Full Changelog: https://github.com/litmuschaos/litmus/compare/3.0-beta1...3.0.0-beta2

Release Notes (3.0-beta1)

• Improved scalability in the execution of network & stress experiments via helper pod optimization (one helper per node).

• Introduction of new experiment category for chaos on springboot applications.

• Enhanced K8s probe which allows filter by resource names (in addition to label and field selectors).

• Fixes regression caused in experiment helper base images which prevents execution of privileged tasks.

• Simplification of chaos candidate selection process - via removal of annotation checks.

• Fixes a bug (race condition) in installation of execution plane components (workflow controller/configmap).

Release Notes (2.14.0)

Core Component Updates -

• Adds the containerd CRI support to the DNS experiments https://github.com/litmuschaos/litmus-go/pull/577

• Adds support for execution of http-chaos experiments in service-mesh environments https://github.com/litmuschaos/litmus-go/pull/578

• Enhances the network experiments to support port specification in source and destination FQDN/IPs for faults injection https://github.com/litmuschaos/litmus-go/pull/570

• Adds an option to provide the custom labels to the chaos-runner pod. It can be defined at spec.components.runner.labels inside the chaosengine https://github.com/litmuschaos/chaos-operator/pull/427

• Enhances the cmdprobe-source pods to inherit the experiment pod spec attributes. It allows the source pod to inherit the ENVs, volumes, and volumeMounts https://github.com/litmuschaos/chaos-operator/pull/421

• Enhances the probe status schema representation inside the chaos-result to add the probe verdict, description, and mode https://github.com/litmuschaos/chaos-operator/pull/419 https://github.com/litmuschaos/litmus-go/pull/566

• Upgraded the operator-sdk version from 0.15.2 to 1.19.0 to remove the indirect unlicensed dependencies https://github.com/litmuschaos/chaos-operator/pull/420/

• Enhance the HTTP probe to use the global probe timeout as response timeout. https://github.com/litmuschaos/chaos-operator/pull/422

• Adds the probe retry as an optional parameter inside the probe to support 0 retries(single iteration) and rename the defaultAppHealthCheck to defaultHealthCheck https://github.com/litmuschaos/chaos-operator/pull/424

• Refactored the chaos-operator for better logging and event creation https://github.com/litmuschaos/chaos-operator/pull/428

• Fixes chaosresult update conflict issue in abort scenarios https://github.com/litmuschaos/litmus-go/pull/567

• Fixes the probeSuccessPercentage evaluation inconsistency for the probe failures https://github.com/litmuschaos/litmus-go/pull/568

• Fixes the stress experiments to clean all the stress-ng child processes during abort https://github.com/litmuschaos/litmus-go/pull/569

• Adds the steady state checks as tunables based on appHealthChecks parameter inside the chaosengine and removed the AUT status checks from the node and infra experiments https://github.com/litmuschaos/litmus-go/pull/576

ChaosCenter Updates -

• Enhances MyHub to clone a single branch with one depth https://github.com/litmuschaos/litmus/pull/3776 https://github.com/litmuschaos/litmus/pull/3792

• Updates the httpProbe UI screens to remove responseTimeout parameters from the httpProbes/inputs and uses the global timeout https://github.com/litmuschaos/litmus/pull/3796

• Adds the probe retry parameter to an optional field https://github.com/litmuschaos/litmus/pull/3797

Shoutouts!

Thanks to our existing and new contributors for this release @asashiskumarnaik @chinmaym07 @SaptarshiSarkar12 @amityt @avaakash @imrajdas @ispeakc0de @Jonsy13 @neelanjan00 @uditgaurav

Installation

Litmus-2.14.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.14.0/mkdocs/docs/2.14.0/litmus-2.14.0.yaml

Or

Litmus-2.14.0 (Stable) namespace scope manifest.

#Create a namespace eg: litmus
kubectl create ns litmus
#Install CRDs, if SELF_AGENT env is set to TRUE
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.14.0/litmus-portal-crds-2.14.0.yml
#Install ChaosCenter
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.14.0/litmus-namespaced-2.14.0.yaml -n litmus

Upgrading from 2.13.0 to 2.14.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.14.0/mkdocs/docs/2.14.0/upgrade-agent.yaml

Full Changelog: https://github.com/litmuschaos/litmus/compare/2.13.0...2.14.0

Release Notes (3.0-beta0)

• Introduction of HTTP Chaos experiment suite

• Support for network & stress experiments on new versions of Kubernetes (1.21+) & OpenShift (>4.x)

• Support for network chaos experiments (Latency, Packet Loss, HTTP, DNS) on service-mesh enabled environments

• Randomization (across range) support for fault inputs

• Redundancy (HA) for the Chaos Operator

• Chaos Workflow trigger, list and delete support using CLI (litmusctl)

• Improved support for containerd & CRIO runtime (DNS, HTTP)

• More powerful experiment bootstrapping with Litmus SDK

• Newer faults centered on Kubernetes Nodes (targeting storage volumes)

• Enabling more powerful hypothesis validation (by widening the scope of command probes)

• Refactoring of Chaos Server APIs to make them more user-friendly

• Improvements to the automated e2e test suite

• Creation of GoogleCodeLabs based Litmus usage tutorials

• Improvement of security posture (optimization of execution privileges, simplifying docs)

Release Notes (2.13.0)

Core Component Updates -

• Enhance network experiments to derive IPs of the pods for Kubernetes service if the target pod has a service mesh sidecar. This will enable us to run all network chaos experiments with service mesh enabled effectively. https://github.com/litmuschaos/litmus-go/pull/558

• Adds Chaos SDK Templates for non-Kubernetes experiments (that is aws, gcp, vmware, azure) this will help us to speed up the experiment development process with a proper template for non-k8s services. To know more refer to the developer docs. https://github.com/litmuschaos/litmus-go/pull/560

• Fixes the stress-chaos experiments to run chaos (helper pod) with minimum capabilities. This will allow running the stress experiments in a restricted environment (like Openshift) with lesser capabilities as mentioned in SCC docs. https://github.com/litmuschaos/litmus-go/pull/557

• Enhance the HTTP status code experiment to have the ability to modify the response body. Also, it adds support to provide the content type and encoding for the body in modify body and status code experiment. https://github.com/litmuschaos/litmus-go/pull/556

• Adds the ability to provide custom Service Account value for helper pod using CHAOS_SERVICE_ACCOUNT Env. It is optional, if not provided the helper pod will run with the same service account as the experiment pod. https://github.com/litmuschaos/chaos-runner/pull/178

• Enhance chaos-operator to enable the leader-elect this ensures with multiple replicas a leader is elected and is the only one actively reconciling the set.https://github.com/litmuschaos/chaos-operator/pull/417

• Refactor chaos operator code to convert the History field in ChaosResult spec as a Go pointer. https://github.com/litmuschaos/chaos-operator/pull/416

• Enhance the chaos (helper pod) status check when waiting for completion with proper error handling. https://github.com/litmuschaos/litmus-go/pull/552

• Adds document content for a better understanding of new HTTP chaos experiments and tunables. https://github.com/litmuschaos/litmus/pull/3755

ChaosCenter Updates -

• Adds enhancement in chaos-scenarios replacing instance-id label with workflow-run-id, which is generated at runtime resulting in unique scenario/run creation on any scenario CRUD operations https://github.com/litmuschaos/litmus/pull/3758

• Upgraded chaos operator go-pkg to 2.12.0 in gql-server & subscriber introducing support for source imagePullSecrets in chaosEngine & along with updates in core components based on 2.12.0. https://github.com/litmuschaos/litmus/pull/3759

• Updated CRDs for ChaosEngines with source attributes updates in 2.12.0 CRD manifest https://github.com/litmuschaos/litmus/pull/3742

• Adds support for providing service-type & makes clusterIP as default service-type for all services in helm-chart https://github.com/litmuschaos/litmus-helm/pull/257

LitmusCTL Updates -

• Adds changes for Error handling in litmusctl apply manifest logic for better debugging & Usability. https://github.com/litmuschaos/litmusctl/pull/97

• Adds .exe extension to binaries on Windows https://github.com/litmuschaos/litmusctl/pull/96

• Upgrades gopkgs for argo-workflows to v3.3.1 and chaos-operator to 2.12.0 versions reducing vulnerabilities. https://github.com/litmuschaos/litmusctl/pull/98

Shoutouts!

Thanks to our existing and new contributors for this release- @chandra-dixit-hcl @alebcay @Jasstkn @amityt @Saranya-jena @SarthakJain26 @Adarshkumar14 @Jonsy13 @ispeakc0de @avaakash @uditgaurav

Installation

Litmus-2.13.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.13.0/mkdocs/docs/2.13.0/litmus-2.13.0.yaml

Or

Litmus-2.13.0 (Stable) namespace scope manifest.

#Create a namespace eg: litmus
kubectl create ns litmus
#Install CRDs, if SELF_AGENT env is set to TRUE
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.13.0/litmus-portal-crds-2.13.0.yml
#Install ChaosCenter
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.13.0/litmus-namespaced-2.13.0.yaml -n litmus

Upgrading from 2.12.0 to 2.13.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.13.0/mkdocs/docs/2.13.0/upgrade-agent.yaml

New Contributors

• @chandra-dixit-hcl made their first contribution in https://github.com/litmuschaos/litmus/pull/3752

Full Changelog: https://github.com/litmuschaos/litmus/compare/2.12.0...2.13.0

Release Notes (2.12.0)

Core Component Updates:

• Adds toxicity support in HTTP Chaos experiments. Toxicity can be used to define the percentage of HTTP requests that should be affected for example 50% toxicity will mean only half of the HTTP requests will be affected. https://github.com/litmuschaos/litmus-go/pull/544

• Introduce randomized status code selection when multiple status code is provided as a fault in input for HTTP status code chaos experiment. https://github.com/litmuschaos/litmus-go/pull/545

• Adds support for experiments with helper pod in chaoslib in litmus SDK with this, the user can create a chaos template with helper pod model using the same litmus SDK. https://github.com/litmuschaos/litmus-go/pull/546

• Adds support for imagePullSecret in source cmd Probe this will enable the user run cmd probe pod with private image. https://github.com/litmuschaos/chaos-operator/pull/415

• Fixes litmus-e2e pipeline and unit test pipeline by specifying version for creating k3s cluster. https://github.com/litmuschaos/litmus-e2e/pull/391

• Fixes chaos execution on openshift-4.x by providing finalizer permission to chaos-operator service account, with the new operator upgrade in litmus 2.11.0 brings down this requirement. https://github.com/litmuschaos/chaos-operator/pull/413

• Fixes chaos-operator vulnerabilities detected by snyc and trivy in the build pipeline. https://github.com/litmuschaos/chaos-operator/pull/414

ChaosCenter Updates -

• Adds an advanced tuning feature for experiments individually with more supported configurations #3715

• Adds a new feature for connecting a remote ChaosHub to ChaosCenter #3722

• Adds a new API getServerVersion for fetching GQL server version #3721

• Makes clusterIP to be used by default for generating endpoints for internal delegates/self-agent #3709

• Adds an additional permission required for execution plane to be deployed in openshift clusters #3708

• Upgrades gopkgs for Chaos-Operator to v2.11.0 & Argo to v3.3.1 reducing some of the vulnerabilities in Graphql-server & Subscriber. #3718

• Removes the support for dashboards & datasources configuration from UI. #3716

• Fixes a bug of workflow getting failed when using dex/ providing email as username #3705

• Fixes a bug of frontend crashing when providing invalid cron-syntax #3714

• Fixes a bug when editing schedule for first time due to state not getting cleaned up #3727

• Adds fix for workflows comparison plot not coming #3729

• Adds fix for logs not coming for install experiments step #3729

• Adds fix for UI not getting updated for deactivate user and created_at column showing invalid date in Users Table in admin-account #3733

Litmusctl Updates -

• With litmusctl 0.12.0 release, we have updated some commands with the new terminologies :sparkles::

  • litmusctl connect agent is now litmusctl connect chaos-delegate
  • litmusctl disconnect agent is now litmusctl disconnect chaos-delegate
  • litmusctl get agents is now litmusctl get chaos-delegates
  • litmusctl create workflow is now litmusctl create chaos-scenario
  • litmusctl delete workflow is now litmusctl delete chaos-scenario
  • litmusctl describe workflow is now litmusctl describe chaos-scenario
  • litmusctl get workflows is now litmusctl get chaos-scenarios
  • litmusctl get workflowruns is now litmusctl get chaos-scenario-runs
  • litmusctl upgrade agent is now litmusctl upgrade chaos-delegate

• Improves error handling logic for upgrade chaos-delegate, describe chaos-scenario and disconnect agent commands. https://github.com/litmuschaos/litmusctl/pull/81

• Adds --kubeconfig flag in upgrade chaos-delegate command. https://github.com/litmuschaos/litmusctl/pull/89

• Adds fix for passing namespace in the SelfSubjectAccessReviews function. https://github.com/litmuschaos/litmusctl/pull/82

• litmusctl will now prompt if installed version of litmusctl is not compatible with installed litmus control plane. https://github.com/litmuschaos/litmusctl/pull/84

• Users can check the compatibility matrix in the readme or by using litmusctl version command. https://github.com/litmuschaos/litmusctl/pull/84

Users can also find the commands in readme: https://github.com/litmuschaos/litmusctl#readme

NOTE: - Along with above terminologies updates, we will also be updating directory structure of ChaosHub for better readability and scalability of experiments and scenarios. With upcoming release, the charts directory will be renamed to experiments directory and workflows will be renamed to scenarios directory. Same changes have been already done in 2.11.0 branch/version of ChaosHub. So, Users are requested to upgrade their ChaosHub directory structure with provided changes before upgrading to upcoming release - 2.13.0 version of ChaosCenter.

Shoutouts!

Thanks to our existing and new contributors for this release- @dnielsen @deividMatos @amityt @Saranya-jena @SarthakJain26 @Adarshkumar14 @imrajdas @Jonsy13

Full Changelog: https://github.com/litmuschaos/litmus/compare/2.11.0...2.12.0

Installation

Litmus-2.12.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.12.0/mkdocs/docs/2.12.0/litmus-2.12.0.yaml

Or

Litmus-2.12.0 (Stable) namespace scope manifest.

#Create a namespace eg: litmus
kubectl create ns litmus
#Install CRDs, if SELF_AGENT env is set to TRUE
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.12.0/litmus-portal-crds-2.12.0.yml
#Install ChaosCenter
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.12.0/litmus-namespaced-2.12.0.yaml -n litmus

Upgrading from 2.11.0 to 2.12.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.12.0/mkdocs/docs/2.12.0/upgrade-agent.yaml

Release Notes (2.11.0)

Core Component Updates -

• Introduce different new HTTP chaos experiments for Kubernetes targets, this will allow users to introduce the following faults:

  • pod-http-status-code (modifies the status code of the response)
  • pod-http-modify-body (modifies the body of the response)
  • pod-http-modify-header (modify headers of incoming requests or the response from the service targetted)
  • pod-http-reset-peer (stops outgoing http requests by resetting the TCP connection)

  To know more checkout the experiment-docs.

• Upgrade the operator SDK version to 1.14.0 for chaos operator and code refator for other litmus components to achieve compatibility.

• Adds support for non-default vpc for the AWS ELB az-down experiment. As earlier the experiment only targets the ELB(s) that are associated with default vpc, so this will also allow the user to target the ELB(s) that are not associated with default vpc.

• Enhance cmd probe spec to support different configuration for probe pod and containers like imagePullPolicy, cmd, args and so on. This will allow the user to run the cmd probe in more controlled way when used in source mode.

• Fixes the error handling for application status check with litmus annotation for pod-level experiments.

• Adds litmusctl docker image which will allow the users to install the agents (delegates) from a pod/container.

ChaosCenter Updates -

• Updates terminologies for different entities -

  • Agents --> Chaos Delegates
  • Workflows --> Chaos Scenarios
  • Charts --> Chaos Experiments

• Reduces the permissions in namespaced mode for execution plane components

• Fixes an issue in SyncHub API when there is an error in communication between graphql-server and MongoDB and all hubs were getting deleted while trying to reclone current chaoshub.

NOTE: - Along with above terminologies updates, we will also be updating directory structure of ChaosHub for better readability and scalability of experiments and scenarios. With upcoming release, the charts will be renamed to experiments directory and workflows will be renamed to scenarios. Same changes have been already done in 2.11.0 version of ChaosHub. So, Users are requested to upgrade their ChaosHub directory structure with provided changes before upgrading to upcoming release - 2.12.0 version of ChaosCenter.

Installation

Litmus-2.11.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.11.0/mkdocs/docs/2.11.0/litmus-2.11.0.yaml

Or

Litmus-2.11.0 (Stable) namespace scope manifest.

#Create a namespace eg: litmus
kubectl create ns litmus
#Install CRDs, if SELF_AGENT env is set to TRUE
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.11.0/litmus-portal-crds-2.11.0.yml
#Install ChaosCenter
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.11.0/litmus-namespaced-2.11.0.yaml -n litmus

Upgrading from 2.10.0 to 2.11.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.11.0/mkdocs/docs/2.11.0/upgrade-agent.yaml

Release Notes (2.10.0)

Core Component Updates -

• Adds HTTP chaos experiment for Kubernetes applications using toxiproxy. This will allow you to introduce latency in the target application service and check the application availability.

• Introduces the first Beta version for m-agent (machine-agent) which enable us to run chaos on non-k8s target. It also includes a new cpu-stress experiment that can run CPU chaos on the target VM(s).

• Adds the missing –stress-image parameter for the pod-io-stress experiment that enables us to add a custom stress image for the experiment when using Pumba lib.

• Enhanced the recovery of node cordon experiment when app status check fails during the chaos.

• Fixes the chaos result verdict update for GCP disk loss by label experiment during the different stages of chaos.

• Fixes node level e2e check for every build on a pull request for the litmus-go repository.

• Adds docs for the new HTTP chaos experiment, updates the GCP experiment docs, and introduces some more examples in docs for pod network latency experiment with jitter.

• Adds chaos charts for AWS AZ down experiment in the hub this will help to get the manifest for the experiment workflow preparation.

• Fixes the GCP and m-agent e2e pipeline to run the automated tests seamlessly.

ChaosCenter Updates -

• Refactored graphql-server for extracting queries, mutations, and subscriptions to the respective schema files

• Added support for Envoy proxy when using frontend nginx.

• Added UI enhancement for allowing scrolling to the invitations tab after clicking on the Invitations button.

• Fixed issues with httpProbe and promProbe in tune workflow section due to the addition of httpProbe/inputs: {} when adding multiple probes.

• Added check for invalid schedule type when trying to proceed in workflow construction wizard without selecting schedule type.

• Fixed issue in GitOps when updating the git repository configuration.

• Added CHAOS_CENTER_UI_ENDPOINT env for specifying a one-time UI endpoint for the control plane, so that all external agents can be provided with the same (Available for cluster and namespace scope).

• Added support for automatically adding imagePullSecrets in for Engine, Runner & Experiments pods from configured image registry.

Litmusctl Updates:

• Added commands like litmusctl get workflows , litmusctl create workflow , litmsuctl describe workflow , litmusctl delete workflow and litmusctl get workflow-runs for workflow CRUD operations.

• Renamed litmusctl create agent command to litmusctl connect agent

• Added new command disconnect agent for disconnecting agents from the Control plane.

• Enhanced logging for better debugging.

Note: - For using newly added commands, users will have to download the v0.11.0 version of litmusctl. - litmusctl v0.11.0 only supports litmus v2.10.0 or higher versions

New Contributors

• @QAInsights made their first contribution in https://github.com/litmuschaos/litmus/pull/3612

Full Changelog: https://github.com/litmuschaos/litmus/compare/2.9.0...2.10.0

Release Notes (2.9.0)

Core Component Updates -

• Adds GCP Instance Stop and Disk Fill Experiments by the label that supports multiple target selection using common labels.

• Refactor the GCP disk loss experiment to remove the device name input and auto-derive it within the experiment.

• Enhance EC2 Terminate experiment checks when the managed nodegroup env is enabled to validate the node recovery through AWS APIs instead of Kubernetes APIs.

• Introduces GCP IAM integration through workload identity for keyless authentication in GKE clusters.

• Fixes Litmus SDK and introduces the default app health check feature in the SDK template.

• Fixes error handling for an invalid appinfo input. The experiments are now supposed to fail in the pre chaos validation and won’t go to the chaos injection stage.

• Fixes chaos injection issues on OpenShift due to the helper pod failure caused due to the same IP it was assigned as the experiment pod, it occurs due to the same annotation shared by helper and experiment pods. Now, this can be tuned via an env called SET_HELPER_DATA if we don’t want to use the experiment data inside helper pods we can enable it.

• Adds chaos charts for process kill experiment which is in beta version and runs with the help of m-agent.

• Fixes the default value of the chaos tunable in the stress chaos experiment to support the default execution of the experiment.

• Cleanup the stale workflows from chaos charts and retain the active workflows only.

• Adds document fixes for stress chaos experiment and introduces docs for GCP IAM integration.

• Adds m-agent e2e pipeline that includes tests for process kill experiment.

ChaosCenter Updates -

• Weight and experiments mapping logic has been updated for handling cases where one workflow contains more than one instance of the same experiment. Now chaosengine name will be used for mapping experiment to the corresponding weight.

• Refactored GQL server APIs for better documentation & following a standard schema.

• Updated API docs according to new changes and added missing APIs.

• Added support for using a custom image registry inside the experiment when using a custom image registry.

• Updated the namespaced scope manifest to reduce the complexity of installation instructions.

• Fixed permission issue for event-tracker in namespaced mode.

• Fixed image version in revert-chaos step for using versioned tag instead of latest.

• Fixed error logging in subscriber for handling error when there are fewer permissions available due to old installation.

• Fixed error logging in Control-plane upgrade agent when new and current versions are the same by providing a warning instead of the error.

NOTE: As all the APIs on graphql-server have been updated for following a standard schema, Litmusctl has also been updated for supporting the same. So, ChaosCenter v2.9.0 will be compatible with Litmusctl v0.10.0. Please make sure to upgrade the litmusctl before using with ChaosCenter v2.9.0.

Installation

Litmus-2.9.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.9.0/mkdocs/docs/2.9.0/litmus-2.9.0.yaml

Or

Litmus-2.9.0 (Stable) namespace scope manifest.

#Create a namespace eg: litmus
kubectl create ns litmus
#Install CRDs, if SELF_AGENT env is set to TRUE
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.9.0/litmus-portal-crds-2.9.0.yml
#Install ChaosCenter
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.9.0/litmus-namespaced-2.9.0.yaml -n litmus

Upgrading from 2.8.0 to 2.9.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.9.0/mkdocs/docs/2.9.0/upgrade-agent.yaml

Release Notes (2.8.0)

Core Component Updates

• Fixes the incorrect target container name passed to the helper pod. This occurs when multiple different applications containing different container names were selected as chaos candidates. https://github.com/litmuschaos/litmus-go/pull/496

• Fixes the statusCheckTimeout feature for container kill experiment to enable the helper pod to use custom timeouts. The helper pod of the container kill experiment was only using 180sec as a timeout. https://github.com/litmuschaos/litmus-go/pull/498

• Fixes the default path of the SSM docs (fault template) used in the SSM chaos experiments to support the chaos execution. The SSM docs contain the chaos injection logic, one can also provide their own docs to run this experiment using configmap or secrets. https://github.com/litmuschaos/litmus-go/pull/508

• Enhance experiment logs to handle warnings like “Neither --kubeconfig nor --master was specified for InClusterConfig” - such warnings were appearing in experiment logs for every run which are now handled properly. https://github.com/litmuschaos/litmus-go/pull/507

• Fixes litmus-python vulnerability issues by bumping the python version from 3.8 to 3.8.13. https://github.com/litmuschaos/litmus-python/pull/41

• Enable litmus snyk image to support python libraries which also fixes snyk security scan check on litmus-python.

• Adds docs examples to show the selection of target pods from a specific node using a NODE_LABEL env.

Chaos Center Updates

• Migrated FluxCD to ArgoCD and added terraform scripts to provision EKS cluster to deploy LitmusChaos applications (v1-Docs, Docs, Hub)

• Added ability to configure self-agent component's node selector and tolerations

• Added support to subscriber and event tracker to store and retrieve CLUSTER_ID and ACCESS_KEY in Kubernetes secrets

• Added email validation and RBAC check get_user_with_project API

• Added workflow type filter, pre-defined workflow redirect link, and updated analytics routes

• Fixed workflow failure issue while adding experiments with the same name

• Synced the ChaosCenter APIs document with the latest schema changes

• Upgraded Workflow-controller (v3.2.9), Argo-executor(v3.2.9) and LitmusCore(2.7.0)

Installation

Litmus-2.8.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.8.0/mkdocs/docs/2.8.0/litmus-2.8.0.yaml

Or

Litmus-2.8.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.8.0/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.8.0/mkdocs/docs/2.8.0/litmus-namespaced-2.8.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Upgrading from 2.7.0 to 2.8.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.8.0/mkdocs/docs/2.8.0/upgrade-agent.yaml

Release Updates

Core Component Updates

• Fixes the higher image vulnerabilities for different litmus components by updating the binaries used in it.

• Fixes the code scan vulnerabilities (CVEs) by updating the go packages for different litmus components. Also, adds sync actions to get the code vulnerabilities on every code commit.

• Enhance pod-level experiments by supporting randomized chaos tunables which enable us to provide the range of value in chaos parameter.

• Adds support for cgroup version 2 in stress chaos experiments(pod cpu/mem/io).This will be backward compatible with the version 1 as well.

• Adds node label filter for selecting the target pods in pod-level experiments. This helps us to select the pods scheduled on specific nodes as chaos candidates. This feature requires having a node-level permission for filtering pods on a specific node.

• Adds some more experiment example docs at https://litmuschaos.github.io/litmus/

Chaos Center Updates

• Added RBAC checks in all the APIs of the chaos center to make it more secure

• Optimized MyHub APIs to fetch the data more efficiently

• Added /readiness route for readiness probes

• Following changes in the deployment manifest and helm charts

  • Added resource {limit, request}, automountServiceAccountToken, readOnlyRootFilesystem and securityContexts for all the components
  • Separated auth-server container from litmus-server deployment and made it as deployment
  • Moved some credentials from configmap to secrets
  • Added a headless service to mongo statefulsets

Installation

Litmus-2.7.0 (Stable) cluster scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.7.0/mkdocs/docs/2.7.0/litmus-2.7.0.yaml

Or

Litmus-2.7.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.7.0/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.7.0/mkdocs/docs/2.7.0/litmus-namespaced-2.7.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Upgrading from 2.6.0 to 2.7.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.7.0/mkdocs/docs/2.7.0/upgrade-agent.yaml

Release Updates

• Enhance CPU stress experiments (both node-cpu-hog and pod-cpu-hog) to consume the resources in percentage of available CPU cores.

• Fixes the permission issue faced by container user of experiment image to access its directory in an Openshift cluster.

• Adds AWS AZ (Availability Zone) experiment in litmus python which disables availability zones for load-balancer. https://github.com/litmuschaos/litmus-python/pull/36

• Fixes the verdict metrics in chaos exporter now it won't export/override the metrics if chaosengine is in completed state and experiment's final verdict is already exported.

• Adds more docs examples for the new experiment tunables in https://litmuschaos.github.io/litmus/

• Enhances the interface to make workflows more auditable by adding a new field “Executed by” in the runs/schedules table that shows the username who scheduled the corresponding workflow.

• Bug fixed to ignore deleted workflows during the schedule of a new workflow.

• Lower down the vulnerabilities of the docker images.

• Added a new set of API tests for MyHub & User related operations.

Installation

Litmus-2.6.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.6.0/mkdocs/docs/2.6.0/litmus-2.6.0.yaml

Or

Litmus-2.6.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.6.0/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.6.0/mkdocs/docs/2.6.0/litmus-namespaced-2.6.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Upgrading from 2.5.0 to 2.6.0

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/mkdocs/docs/2.6.0/upgrade-agent.yaml

Shoutouts!

Thanks to our existing and new contributors for this release- @victorlcm @Aman-Codes @neelanjan00 @amityt @gdsoumya @Jonsy13 @rajdas98 @uditgaurav

Core Component Updates

• Introduce a pre-alpha version of litmus m-agent (Machine Agent) which is a platform-generic daemon agent that can remotely inject faults into an OS scoped resource, as part of the LitmusChaos Experiments.

• Adds an pre-alpha version of process kill experiment that makes use of m-agent to inject process kill chaos in the target OS.

• Enhance pod network latency experiment to support jitter functionality in the network delay simulated by the experiment.

• Introduces new helm charts (kube-azure and kube-gcp) for azure and gcp chaos experiments.

• Fixes to support the custom experiment for pod level stress-chaos experiment (cpu, memory, IO). This will enable us to use any custom names for the experiment.

• Adds new e2e-pipeline for testing the azure chaos experiments from hub. It supports both nightly and manual builds.

• Updates failstep messages for Litmus Go SDK and refactor azure chaos experiment to improve the code quality.

Chaos Center Updates

• Adds support for skipping SSL verification & Self-signed certificates for agent connection.

• Updates the API documentation.

• Fixes the issue with Minikube & GRPC by making localhost as grpc endpoint.

• Fixes the Role-based permission issues in some specific APIs.

• Enhances the logs for better debugging & logging.

• Adds E2E Tests for observability dashboards configuration & deletion.

Notes

• For using Self-signed certificates & skip-agent-ssl flag, users will have to use litmusctl v0.9.0

Installation

Litmus-2.5.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.5.0/mkdocs/docs/2.5.0/litmus-2.5.0.yaml

Or

Litmus-2.5.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.5.0/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.5.0/mkdocs/docs/2.5.0/litmus-namespaced-2.5.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Core Component Updates

• Fixes the permission issue faced by container user (of different litmus components) to access its directory in an Openshift cluster as the openshift cluster runs the container with an arbitrary uid in the root group.

• Enhance the stress chaos experiments to gracefully terminate the stress process in case it is not completed and remove on its own.

• Enhance the node drain experiment to skip the uncordon step post chaos if the target node is not available. It occurs if we enable the autoscaling for the node pool on the cloud provider.

• Updates the experiment contributing docs with the usage of litmus SDK and also adds the chaos-chart details in the repo readme.

• Adds an overview section in litmus-python with the detailed steps for preparing and using the experiment and workflows in python.

• Adds new e2e-pipeline for testing the vm-ware experiments from hub. It supports both nightly and manual builds.

• Enhance the e2e-dashboard to show the logs of the tests running in the e2e pipeline.

• Enhance GitHub Chaos Actions to support AWS EKS clusters, with this now we can integrate any external EKS cluster with chaos action to run the experiments.

Chaos Center Updates

• Upgrading argo workflow and event tracker API versions to support chaos center on Kubernetes 1.22.0+

• Addition of a new event tracker policy operator (known as Change) to track values of any field in the Kubernetes resource and removed the dependency of event tracker from last applied configuration annnotation

• Addition of new permissions to docker images of ChaosCenter server to OpenShift friendly

• Migration of APIs from graphql-server to auth-server and introduction of gRPC for the internal communication

• Addition of "--skipSSL" flag in litmusctl to disable SSL for the communication between control-plane and execution plan

• IPV6 support for control plane

• Addition of new E2E tests for observability.

Notes

• The CRDs for ChaosCenter will be versioned now . So, CRDs for ChaosCenter v2.x.x will be available in 2.x.x branch.
• The Project DB migration has been done in this release along with some enhancements (like addition of new flag --skipSSL), so any version of litmusctl below v0.7.0 won't be compatible with ChaosCenter 2.4.0 or above.

Installation

Litmus-2.4.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.4.0/mkdocs/docs/2.4.0/litmus-2.4.0.yaml

Or

Litmus-2.4.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.4.0/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.4.0/mkdocs/docs/2.4.0/litmus-namespaced-2.4.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Shoutouts!

Thanks to our existing and new contributors for this release- @Aman-Codes @DhananjayPurohit @tuananh @andrewhu-hcl @Vr00mm @nicjohnson145 @DarthBenro008 @mehabhalodiya @bbarin @amityt @gdsoumya @Saranya-jena @SarthakJain26 @ispeakc0de @Jonsy13 @rajdas98 @uditgaurav

New Features & Enhancements

• Adds support for ipv6 inside network experiments and other litmus components which enable us to run litmus on ipv6 (ipFamily) clusters

• During the initial few seconds of the pod run, sometimes pods failed to request the kubeapi server. Adds the retries inside experiment & runner pods while querying kubeapi server.

• Upgrade the Pumba older version(0.8.0) to the latest version. It fixes the vulnerability of the Pumba binaries.

• Adds the networking policies permissions(which are required for the pod-network-partition experiment) inside the litmus-admin rbac.

• Removes an extra job-name label from the helper pods and refactor the experiment code to improve the logs

• Fixes stress chaos experiment with Pumba lib to take custom stress image

• Reorder the RBAC permissions and add comments description for all the experiments

• Updates the Kyverno pod security policies. Also, enhance the FAQ and PSP in the experiment-docs

• Adds test-cases, readme, and rbac for pod-delete experiment in litmus-python. Modified the litmus-python SDK documentation for the latest changes

• Introduces chaosresult tag annotation to the dashboard for Postgres to add.

• Enhance e2e-dashboard to include card view for different pipelines, support dark-mode, and add all-workflows.

ChaosCenter -

• Fixed Subscriber version compatibility check for the smooth upgrade process.

• Upgraded Litmus-ui version for components optimization.

• Updated the token name to litmus-cc-token.

• Added advanced tune section for workflows for better visibility & allowing users to have more control over workflows.

• Optimized upload functionality for allowing upload of basic argo workflow ( Users will be able to run any generic workflow )

• Fixed some bugs in the frontend & graphql-server.

• Enhanced E2E pipeline by adding resource validation on k8s cluster after scheduling the workflows.

Note:

• With this release of ChaosCenter, Litmus core components are upgraded to v2.2.0 & default chaoshub is updated to 2.2.x

Installation

Litmus-2.3.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.3.0/mkdocs/docs/2.3.0/litmus-2.3.0.yaml

Or

Litmus-2.3.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.3.0/mkdocs/docs/2.3.0/litmus-namespaced-2.3.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

New Features & Enhancements

• Adds more secure litmus hardened alpine base images to all litmus components.

• Fixes the image vulnerability, add security scanner and health checks for all the litmus components

• Adds the ability to provide terminationGracePeriodSeconds inside the helper pods. This also fixes the execution of node-level service kill experiments for a longer chaos duration.

• Refactor the experiments with enhanced logging, error handling, and go-report.

• Adds support to execute the vm-poweroff experiment in serial or parallel sequence

• Adds the okteto configuration to test the experiments generated by litmus python-sdk

• Introduces E2E tests for the litmus-python repository via GitHub actions

• Adds kyverno pod security policies for the litmus pods to enable the execution of all supported (out-of-the-box) experiments listed in the chaoshub.

• Adds the support of the hostNetwork inside the cmdProbe in workflows.

• Introduced a first-cut visualization of the e2e dashboard to monitor different pipelines and stability of an experiment

• Add a new pipeline to test GCP based chaos experiments that is vm-instance-stop and vm-disk-loss tests

ChaosCenter

• Adds frontend Bug fixes in Probes wizard, Analytics & Teaming

• Adds Nginx configuration in a separate configMap for more configuration

• Adds kelseyhightower/envconfig pkg for validating the environment variables

• Adds RBAC for the newly added mutations

• Adds Agent Upgrade feature, which will help in upgrading agents when the control plane is upgraded.

• Moves DB_ADMIN, DB_PASSWORD, and JWT_SECRET environment variables to k8s secret

• Adds option for adding tolerations to the agent plane components using litmusctl

• Enhances the ChaosCenter e2e pipeline for testing with LoadBalancer & Ingress based access setups.

Note:

• With this release of ChaosCenter, Litmus core components are upgraded to v2.1.1, which is having cmd probeSchema changes. In case users have their workflows scheduled with an older schema, they will have to update it according to the new schema.

• For using agent upgrade feature & tolerations addition to agents plane components, Users have to use v0.5.0 of litmusctl

Installation

Litmus-2.2.0 (Stable) Cluster Scope manifest

kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/2.2.0/mkdocs/docs/2.2.0/litmus-2.2.0.yaml

Or

Litmus-2.2.0 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="<namespace>"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/2.2.0/mkdocs/docs/2.2.0/litmus-namespaced-2.2.0.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst '${LITMUS_PORTAL_NAMESPACE}' < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

New Features & Enhancements

• Introduces azure-disk-loss experiment, which detaches the virtual disk from an Azure instance

• Introduces pod-network-partition experiment, which blocks Ingress and Egress traffic of the target application. It provides the option to specify the pod-selector, namespace-selector, and ports.

• Introduces the ability to provide default application status checks as a tunable for the pod-specific experiments from the ChaosEngine. It is helpful for the scenarios where we don’t want to validate the application status as a mandatory check during pre & post chaos.

• Enhance the GCP VM Instance Stop experiment for VMs that are a part of an Autoscaling Group, where the VMs are now awaited to fully stop before proceeding with the experiment.

• Enhancements for the cmdProbe to run the probe pod with the host network.

• Enhance probe pod functionality where the experiment serviceAccount is now passed to the probe pod. It is helpful for the scenarios where the probe pod needs specific RBAC permissions for the execution.

• litmuschaos_awaited_experiment metric contains an injection_time label. Adds injection_time label as a tunable via the INJECTION_TIME_FILTER ENV. Which can be used for the annotations inside the interleaving dashboards

• Enhance the chaos-scheduler to specify the minutes and hours separately. This provides the ability to schedule the chaos at the nth minutes per x hours. Which will be helpful for the staggered schedules to avoid collisions.

• Adds the ability to search experiments based on certain keywords in Chaos Hub. Added all the related keywords for individual experiments.

• Improves the error handling in the chaos-operator for chaosengine status nil check

• Removes the init container which modifies the permissions of the docker socket path, and instead provides the docker socket path inside the DOCKER_HOST ENV for the Pumba Lib.

• litmus metrics are now exported only for the active ChaosEngines. It stops overriding the value of existing metrics as soon as ChaosEngine comes to the completed state.

• Enhance the e2e pipeline by adding debug steps and using shared runners to run tests in parallel, also allow the ability to run a test with a custom service account to validate a specific set of permissions required for an experiment.

Major Bug Fixes

• Fixes chaos-result name generation where the experiment-name and instance-id are passed inside the helper pods, which are used to generate the chaos-result name.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v2.1.0.yaml

Verify your installation

• Verify if the chaos operator is running kubectl get pods -n litmus

• Verify if chaos CRDs are installed kubectl get crds | grep chaos

For more details refer to the documentation at Docs

Introduction

Version 2.0.0 brings newer capabilities to the LitmusChaos platform, enabling a more efficient practice of chaos engineering. The major version upgrade is being carried out to reflect significant improvements and new features in the platform - many of which were introduced & curated across several preceding 2.0 beta releases with community feedback (thanks to all the early adopters & beta testers for your continued support. Some of these changes, especially. newer experiments and observability improvements have been made available in 1.x too).

Litmus 1.x brought a cloud-native approach to chaos engineering to the definition and execution of chaos intent, along with a ready set of experiments maintained in the ChaosHub. Along the way, newer requirements were incorporated into the project, most notably around a centralized management approach for managing chaos across environments (K8s clusters and cloud instances) and the ability to define workflows to stitch together multiple experiments as part of a complex scenario.

The 2.0 GA release brings these features into the mainstream, having been validated for their usefulness & architecture. Subsequent improvements to these will be carried out in 2.x releases. Some salient features are described briefly in below sections:

Chaos Center

• A chaos control plane or portal which provides centralized management of chaos operations on multiple clusters across datacenters/cloud. The control plane carries out experiments through agents installed on the registered clusters.
• Comprises documented APIs that can be used to invoke chaos programmatically
• Provides visualization capabilities and analytics around chaos execution.
• Supports a project-teams-users structure to enable collaboration within teams for chaos operations.

Litmus Workflows

• Introduces chaos workflows - to (a) automate dependency setup (b) aid creation of complex chaos scenarios with multiple faults (c) support definition of load/validation jobs along with chaos injection
• Provides flexibility in creating/running workflows in different ways - via templates, from an integrated hub, and custom uploads.

Multi-Tenancy

• Supports setup (control plane & agents) and execution of chaos experiments in both: cluster-scoped and namespace-scoped modes to help operations in shared clusters with a self-service model

Observability & Steady State Hypothesis Validation

• Provides an increased set of Prometheus metrics with additional filters - which can be used for instrumenting application dashboards to observe chaos impact
• Provides diverse set of probes to automate validation of steady-state hypothesis - thereby improving the efficiency of running automated chaos experiments

GitOps for Chaos

• Integrates with Git-based SCM to provide a single-source-of-truth for chaos artifacts (workflows), such that changes are synchronized bi-directionally b/w the git source and the chaos center - thereby pulling the latest artifact for execution.
• Provides an event-tracker microservice to automatically launch “subscribed” chaos workflows upon app upgrades effected by GitOps tools like ArgoCD, Flux

Non-Kubernetes Chaos

Adds experiments to inject chaos on infrastructure (cloud) resources such as VMs/instances and disks (AWS, GCP, Azure, VMWare) - irrespective of whether they host a Kubernetes cluster or not.

Release Cadence & Versioning

The release cadence & naming conventions continue to adhere to the principles followed thus far in the Litmus project: the monthly minor version releases (2.x.0) will happen on the 15th, with patch releases/hotfixes going into 2.x.x, on a need/demand basis. The 1.x version will be stopped at 1.13.x (1.13.8 at this point) and further patches will be made only upon request/community need.

Backward Compatibility

Having said that, Litmus 2.x completely lends itself to the 1.x mode of execution the users are familiar with, i.e., you could still continue to deploy the latest chaos-operator deployment in admin/namespace mode, pull ChaosExperiment templates/CRs from the ChaosHub & trigger chaos by applying the ChaosEngine CR. The latest chaos-exporter & chaos-scheduler will continue to be operable as they are. However, the introduction of the Chaos-Center (also commonly referenced as Litmus Portal by the beta test community) simplifies the above process greatly while giving you additional nuts & bolts.

Migration from 1.x to 2.x

To make use of the Chaos Center and other capabilities of Litmus 2.0, please remove any existing ChaosEngines, uninstall the chaos operator deployment & follow the Litmus 2.x installation instructions.

If you would like to consume just the backend infrastructure components (chaos operator, crds et al), please follow the regular procedure in applying the latest operator manifest or start using the operator helm chart to allow for subsequent helm upgrades.

If you are a beta user on 2.0.0-beta9, follow the upgrade procedure to start using the Litmus 2.0 GA build.

Documentation

The documentation has undergone considerable changes - in terms of content and structure and it continues to undergo improvements as of the 2.0 release. We expect that a few more iterations are needed to sort out the Information Architecture.

The installation details for the 2.0 platform along with detailed introductions to concepts, architecture as well as a user guide are now available at https://docs.litmuschaos.io/

The latest chaos experiment details along with chaos custom resource schema specifications (tunables, examples, etc.,) and detailed FAQs & troubleshooting info can be found in https://litmuschaos.github.io/litmus/

For those continuing to use 1.x releases, please note that the docs are now moved to: https://v1-docs.litmuschaos.io/

Misc (monthly changelog between 15/07/2021 to 15/08/2021)

Notes on changes to control plane (chaos center) since 2.0.0-beta9

• Added new API routes to check the status of the authentication server and to update the user details
• Added an API to terminate chaos workflow
• Added namespace scope support for event tracker
• Bugs fixes/enhancement in the frontend
• Typo in the nodeSelector schema key
• Adheres to correct schema in the steady-state validation wizard for Litmus Probes
• Fixes the inability to login/authenticate after upgrade of chaos-center

Notes on changes to backend execution infrastructure (chaos operator, experiments) since 1.13.8

• Supports VM belonging to scale-sets (VMSS) as target resources in the Azure instance stop experiment
• Fixes the limitation/inability to perform abort operations in the namespaced mode of operation in a chaos operator.
• Fixes an issue (edge case in scaled scenarios) within the abort functionality for the “exec” based chaos experiments (pod-cpu-hog-exec & pod-memory-hog-exec) wherein chaos injection continues to occur even post issual of abort.
• Adds fix to fail faster when helper pods do not run successfully in an experiment (fail immediately upon identifying helper failure instead of waiting for the customary statusCheckTimeout of 180s, as the helper pods are usually brought up with restartNever policy)
• Fixes the inability of certain experiments (pod-cpu-hog-exec, pod-memory-hog-exec and pod-dns-error, pod-dns-spoof) to select targets serially for cases where 0 > PODS_AFFECTED_PERC <= 100.
• Adds a condition to error out/call out the engine schema when neither the .spec.appinfo.applabel nor TARGET_PODS env are specified.
• Adds missing ability to perform auxiliary application health check in the node-memory-hog experiment and missing support for specifying multiple target nodes via a comma-separated list in node-cpu-hog, node-memory-hog & node-io-stress experiment.
• Fixes a regression in recent 1.13.x experiments wherein the .spec.appinfo.appkind is mandated (in order to derive parent controller name for pods - as this is used to patch the chaosresult status with target info) even when .spec.annotationCheck set to false. With this fix, you will be able to see older behavior wherein appkind can be left empty for cases where annotationCheck is set to false in the ChaosEngine CR.

Major Updates

• Refactored the authentication server to reduce the network latency and added support for enabling/disabling users
• Added a new pre-defined chaos workflow to check the resiliency of the Bank of Anthos application
• Enhanced the analytics module with API and user interface optimization
• Added a usage module in the litmusportal for admin users to check the agent and workflow usage across all accounts
• Replaced all the privileged permissions with minimal permissions in the cluster and namespace manifest
• Built a mirror docker image of frontend application with the based path of “/litmuschaos” that can be used in ingress paths
• Optimized the UI rendering by scoping out the header and sidebar scaffold to a global level and enhanced the user experience with the support of code-splitting at the component level.
• Added support for multi-architecture(ARM64 and AMD64) docker images for Argo workflow executor and Argo workflow controller
• Enhanced the E2E pipeline with parallel build to fasten the testing using comment driven approach

Minor Updates

• Added node Selector as a configurable environment variable for agent manifests
• Added support to tune chaos workflow environment variables from the user interface
• Upgraded the go version to 1.16, litmus UI to 1.4.0, and MyHub go pkg to a stable version of go-git

NOTE: With the revamp of authentication module and addition of a new feature Deactivate/Activate User, we have made some changes in Auth Schema and module. This has been done to reduce the network latency and optimize the user experience. With these new changes, who have been running Portal previously won't be able to login after upgrading. We are working on the streamlined Upgrade/Migration process, for now we suggest you to re-setup portal for upgrading to Beta9.

Installation

Applying k8s manifest

Litmus-2.0.0-Beta9 (Stable) cluster scope manifest

kubectl apply -f https://litmuschaos.github.io/litmus/2.0.0-Beta/litmus-2.0.0-Beta.yaml

Or

Litmus-2.0.0-Beta9 (Stable) namespace scope manifest.

export LITMUS_PORTAL_NAMESPACE="litmus"
kubectl create ns ${LITMUS_PORTAL_NAMESPACE}
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml
curl https://raw.githubusercontent.com/litmuschaos/litmus/master/docs/2.0.0-Beta/litmus-namespaced-2.0.0-Beta.yaml --output litmus-portal-namespaced-k8s-template.yml
envsubst < litmus-portal-namespaced-k8s-template.yml > ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml
kubectl apply -f ${LITMUS_PORTAL_NAMESPACE}-ns-scoped-litmus-portal-manifest.yml -n ${LITMUS_PORTAL_NAMESPACE}

Major Updates

• Added chaos scheduler dependency in the control plane server to support cron scheduling.
• Added support for ARM64 architecture for Litmus control plane and agent plane components
• A warning will be displayed for long-running workflow after 20 mins and added support for deleting and syncing long-running chaos workflows.
• Added support to use predefined workflow from the connected MyHubs with the project.
• Optimised the workflow’s graphql queries with support of pagination, sorting, and filtering.
• Introduced chaos engine as a standalone workflow which makes it independent of Argo Workflow.
• Added podGC strategy along with revert chaos to remove workflow related artifacts after the completion of workflow Note: Logs won’t be accessible if the revert chaos step is enabled.
• Added new graph components (stack-bar graph, line-area graph, and radial charts) in the litmus-ui NPM package
• Restructured the directory of the subscriber and bug fixes of kubeobject in the namespace mode.
• Redesigned analytics dashboards with the new UI components and enhanced some of the existing features like workflow comparison, query manipulation, and dashboard operation. Also, Optimised the graphql APIs to fasten the real-time response.

Minor Updates

• Removed Argo server deployment and its dependencies from the agent plane list
• Enhanced the user interface of workflow editor in the litmus portal.

Major Updates

• Added MongoDB go-interface and refactored the database operations and structure to accommodate the test cases easily.
• Support for adding custom container image registry to chaos workflow manifest.
• Enhanced the performance of the analytics APIs with memory caching and added APIs to fetching labels and values for a Prometheus series.
• Added support for mutating the sequence of the workflow steps by drag and drop which reflect the live changes in the DAG.
• Enhanced the workflow graph to show other node phases such as Omitted, Skipped, and Error for a good user experience.
• Enhanced the verify and commit page to allow users to have a final review and edit their workflow details before scheduling the same.
• Bug fixed for some user management operations and refactored teaming APIs to increase the performance.
• Enhanced the litmusportal user interface to fastens the onboarding process.

Minor Updates

• Adding support for liveness check of the dependent applications in the agent plane before going active.
• AirGapped support for the pre-defined workflows by moving the fetching logic to the backend.
• Added instance-id label in the chaos workflow manifest to avoid multiple scheduling in the multi-Argo server cluster.
• Added validations for workflow name, GitHub URL, and different probe inputs.

Major Updates

• Fixes the inability to successfully register the agents/targets when litmus portal server is brought up with loadbalancer/nodeport service type
• Makes MyHub source configurable by branch so that latest stable versions of experiments are pulled for custom & predefined workflows
• Updates the chaos operator dependencies on the subscriber to make use of the latest api changes for chaos resources
• Updates the chaos operator, runner & exporter image tunables/ENVs in the subscriber so that the latest stable versions are installed on the targets
• Updates Okteto dev setup instructions to reflect latest image versions and changes in specification (env) as well as instructions
• Updates the chaosengine CRD validation schema for annotation injection in the manifests maintained & installed by the subscriber

Minor Updates

• Improves the icons for revert chaos and workflow scheduling
• Optimizes the teaming code to remove redundant conditions
• Improved styling & background adopted from litmus-ui

Litmus 2.0.0-Beta3

Major Updates

• Support for policy-based control of event tracker where users can define their own policy using JMESPath query and based on that event-tracker will react to the application changes.
• Enhanced UI for workflow Scheduling, gives users the ability to tune annotations, target application details like application namespace, labels, and kind, and probe data using User Interface.
• New UI for workflow visualization for showing information about workflow and nodes in a better way.
• We made the onboarding process for users and easier to use through the new UI.
• Enhanced the homepage to show information like Recent workflow runs, Agent details, and Project details.
• Shifting project switching from using Redux-based technique to URL-based technique to avoid caching problems.
• Migrated CircleCI to GitHub workflow and enhanced the continuous integration of the project.
• Enhanced the analytics module in terms of UI and computation
• Enhanced the browse workflows table to show resilience score and the total number of experiments passed for the listed workflows.* Support role-based access control in the backend for handling authorization for all requests.
• Support for storing scheduled workflow templates and adding some new podtato-head predefined workflow templates

Minor Updates

• Increment in the Better Code Hub(BCH) score
• Optimized the frontend by shifting the resiliency score calculation to the backend.
• Restructured the directory structure for settings in the frontend to modularise the code.
• Support for a reinstall of litmus agents by moving the litmus-portal-config configmap independent of the subscriber.
• Support for Ingress and Load balancer network type for connecting external agents with Litmus Portal. Based on the server service type, it will generate the endpoint for the external agent.

Major Updates

• Support for in-built analytics, where users can connect their data sources and generate dashboard panels.
• Support for Git as a single source of truth for workflow artifacts. This enables users to have their workflows synced between the portal and Git source.
• Introduces the event-tracker microservice to trigger chaos workflows automatically upon change to application images. This feature works in tandem with GitOps frameworks that rollout changes to applications upon manual changes in the Git source or upon image push to registries.
• Support for re-running of existing chaos workflow from the litmus portal.
• Adding a command-line tool called litmusctl to manage litmus portal services. The key role of litmusctl is to connect the external cluster with the litmus server and install the external agents.
• Redesigning the teaming user interface and adding some significant features such as leave project, decline invitation.
• Recreating litmus docs for litmus 2.0.x. For more information, visit https://litmusdocs-beta.netlify.app/
• Integration of Litmus-UI with litmus portal components
• Major directory restructuring of litmus portal’s server for database handlers

Minor updates

• Changing MongoDB kind from deployment to statefulsets
• Adding chaos-exporter as default external cluster agents for litmusportal
• Refactoring authentication server to accommodate new teaming integration
• Removing some unnecessary inputs from the welcome modal and predefined chaos workflow

New Features & Enhancements

• Introduces upgraded pod-cpu-hog & pod-memory-hog experiments that inject stress-ng based chaos stressors into target containers pid namespace (non-exec model).

• Supports multi-arch images for chaos-scheduler controller

• Supports CIDR apart from destination IPs/hostnames in the network chaos experiments

• Refactors the litmus-python repository structure to match the litmus-go & litmus-ansible repos. Introduces a sample python-based pod-delete experiment with the same flow/constructs as its go-equivalent to help establish a common flow for future additions. Also adds a BYOC folder/category to hold non-litmus native experiment patterns.

• Refactors the litmus-ansible repo to remove the stale experiments (which have been migrated and improved in litmus-go). Retains (improves) samples to help establish a common flow for future additions

• Adds GCP chaos experiments (GCP VM stop, GPD detach) in technical-preview mode

Major Bug Fixes

• Fixes erroneous logs in the chaos-operator seen while attempting to remove finalizer on chaosengine

• Fixes a condition where the chaos revert information is present in both annotations as well as the status of chaosresult CR (the inject/revert status is typically maintained/updated as an annotation on the chaosresult before it is updated into the status and cleared/removed from annotations)

• Removes hardcoded experiment job entrypoint, instead of picking from the ChaosExperiment CR’s .spec.definition.command

• Fixes a scheduler bug that interprets a minChaosInterval mentioned in hours (ex: 1h) in minutes

• Improves the scheduler reconcile to stop flooding/logging every “reconcile” seconds irrespective of the minChaosInterval

• Enables the scheduler to start off with the chaos injection immediately upon application of the ChaosSchedule CR without waiting for the first installment of minChaosInterval period - in repeat mode with only the minChaosInterval specified

• Handles edge/boundary conditions where chaos StartTimeis behind CreationTimeStampof ChaosSchedule OR next iteration of chaos as per minChaosInterval is beyond the EndTime

• Adds a check to ignore chaos pods (operator, runner, experiment/helper/probe pods) and blacklist them from being chaos candidates (esp. needed when appinfo.applabel is configured with exclusion patterns such as: !keys OR <key> notin <value>)

• Removes hostIPC, hostNetworkpermissions for pod stress chaos experiments

• Fixes an incorrect env key for TOTAL_CHAOS_DURATION in pod-dns experiments

• Fixes a regression introduced in 1.13.6 wherein the experiment expected the parent workloads (deployment, statefulset et al) to carry labels specified in appinfo.applabel, apart from just the pods even when .spec.annotationCheck was set to false in the ChaosEngine. Prior to this, the parent workloads needed to have the label only when .spec.annotationCheck was set to true. This has been re-corrected as per earlier expectations.

Limitations

• Chaos abort (via .spec.engineState set to stop OR via chaosengine deletion) operation is known to have an issue with the namespace scoped chaos-operator in 1.13.8, i.e., an operator running with WATCH_NAMESPACE env set to a specific value and using role permissions. In such cases, the finalizer on the ChaosEngine needs to be removed manually and the resource deleted to ensure the operator functions properly.

  This is not needed/necessary for cluster scoped operators (which is the default mode of usage)(where WATCH_NAMESPACE env is set to empty string to cover all ns & leverages clusterrole permissions.)

  The fix for correcting the behavior of namespace scoped operators will be added in the next patch.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.8.yaml

Verify your installation

• Verify if the chaos operator is running kubectl get pods -n litmus

• Verify if chaos CRDs are installed kubectl get crds | grep chaos

For more details refer to the documentation at Docs