Skip to main content

Safely upgrade ETCD v3.5 -> v3.6

Upgrade ETCD from v3.5 to v3.6 in vCluster​

In vCluster, etcd can serve as the underlying storage backend, similar to vanilla Kubernetes. Its version can be configured either explicitly via a controlPlane.backingStore.etcd.deploy.statefulSet.image.tag section in the vcluster.yaml or implicitly derived from the vCluster version. Before upgrading, we recommend taking an etcd snapshot for safety; the procedure is detailed in the backup documentation.

etcd has identified and fixed an additional scenario that may cause upgrade failures when moving from etcd v3.5 to v3.6, specifically during rolling replacement upgrades where a new learner may receive an incorrect snapshot from an older member (≤ v3.5.19). This issue, detailed in issue #20793, was addressed in etcd v3.5.24 via PR #20797.
The recommended solution is to first upgrade to etcd v3.5.24 (or higher) before proceeding to v3.6. For users on v3.5.20-v3.5.23 from a fresh install, a direct upgrade to v3.6 is safe.
To safely upgrade etcd in vCluster, continue reading this page for detailed steps.

For more details, refer to the etcd upgrade guide.

Version matrix for embedded ETCD and vCluster​

vCluster VersionETCD Version
v0.26.4v3.5.21
v0.26.5v3.5.25
v0.27.2v3.5.21
v0.27.3v3.5.25
v0.28.1v3.5.21
v0.28.2v3.5.25
v0.29.*+v3.6.*

Important: Before upgrading, read the ETCD blog post about upgrading from 3.5 to 3.6.

First upgrade to v0.26.5, it uses ETCD v3.5.25. Then, your ETCD data will be saved in a format compatible with upgrade to v3.6

Config reference​

backingStore object ​

BackingStore defines which backing store to use for virtual cluster. If not defined will use embedded database as a default backing store.

etcd object ​

Etcd defines that etcd should be used as the backend for the virtual cluster

embedded object ​

Embedded defines to use embedded etcd as a storage backend for the virtual cluster

enabled boolean false ​

Enabled defines if the embedded etcd should be used.

migrateFromDeployedEtcd boolean false ​

MigrateFromDeployedEtcd signals that vCluster should migrate from the deployed external etcd to embedded etcd.

snapshotCount integer ​

SnapshotCount defines the number of snapshots to keep for the embedded etcd. Defaults to 10000 if less than 1.

extraArgs string[] [] ​

ExtraArgs are additional arguments to pass to the embedded etcd.

deploy object ​

Deploy defines to use an external etcd that is deployed by the helm chart

enabled boolean false ​

Enabled defines that an external etcd should be deployed.

statefulSet object ​

StatefulSet holds options for the external etcd statefulSet.

enabled boolean true ​

Enabled defines if the statefulSet should be deployed

EnableServiceLinks for the StatefulSet pod

image object ​

Image is the image to use for the external etcd statefulSet

registry string registry.k8s.io ​

Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub.

repository string etcd ​

Repository is the repository of the container image, e.g. my-repo/my-image

tag string 3.6.8-0 ​

Tag is the tag of the container image, and is the default version.

imagePullPolicy string ​

ImagePullPolicy is the pull policy for the external etcd image

env object[] [] ​

Env are extra environment variables

extraArgs string[] [] ​

ExtraArgs are appended to the etcd command.

resources object ​

Resources the etcd can consume

limits object ​

Limits are resource limits for the container

requests object map[cpu:20m memory:150Mi] ​

Requests are minimal resources that will be consumed by the container

pods object ​

Pods defines extra metadata for the etcd pods.

annotations object {} ​

Annotations are extra annotations for this resource.

labels object {} ​

Labels are extra labels for this resource.

highAvailability object ​

HighAvailability are high availability options

replicas integer 1 ​

Replicas are the amount of pods to use.

scheduling object ​

Scheduling options for the etcd pods.

nodeSelector object {} ​

NodeSelector is the node selector to apply to the pod.

affinity object {} ​

Affinity is the affinity to apply to the pod.

tolerations object[] [] ​

Tolerations are the tolerations to apply to the pod.

priorityClassName string ​

PriorityClassName is the priority class name for the the pod.

podManagementPolicy string Parallel ​

PodManagementPolicy is the statefulSet pod management policy.

topologySpreadConstraints object[] [] ​

TopologySpreadConstraints are the topology spread constraints for the pod.

security object ​

Security options for the etcd pods.

podSecurityContext object {} ​

PodSecurityContext specifies security context options on the pod level.

containerSecurityContext object {} ​

ContainerSecurityContext specifies security context options on the container level.

persistence object ​

Persistence options for the etcd pods.

volumeClaim object ​

VolumeClaim can be used to configure the persistent volume claim.

enabled boolean true ​

Enabled enables deploying a persistent volume claim.

accessModes string[] [ReadWriteOnce] ​

AccessModes are the persistent volume claim access modes.

retentionPolicy string Retain ​

RetentionPolicy is the persistent volume claim retention policy.

size string 5Gi ​

Size is the persistent volume claim storage size.

storageClass string ​

StorageClass is the persistent volume claim storage class.

volumeClaimTemplates object[] [] ​

VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet

addVolumes object[] [] ​

AddVolumes defines extra volumes for the pod

addVolumeMounts object[] ​

AddVolumeMounts defines extra volume mounts for the container

name string ​

This must match the Name of a Volume.

readOnly boolean ​

Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.

mountPath string ​

Path within the container at which the volume should be mounted. Must not contain ':'.

subPath string ​

Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).

mountPropagation string ​

mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.

subPathExpr string ​

Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive.

annotations object {} ​

Annotations are extra annotations for this resource.

labels object {} ​

Labels are extra labels for this resource.

service object ​

Service holds options for the external etcd service.

enabled boolean true ​

Enabled defines if the etcd service should be deployed

annotations object {} ​

Annotations are extra annotations for the external etcd service

headlessService object ​

HeadlessService holds options for the external etcd headless service.

annotations object {} ​

Annotations are extra annotations for the external etcd headless service

external object ​

External defines to use a self-hosted external etcd that is not deployed by the helm chart

enabled boolean false ​

Enabled defines if the external etcd should be used.

endpoint string ​

Endpoint holds the endpoint of the external etcd server, e.g. my-example-service:2379

tls object ​

TLS defines the tls configuration for the external etcd server

caFile string ​

CaFile is the path to the ca file

certFile string ​

CertFile is the path to the cert file

keyFile string ​

KeyFile is the path to the key file

database object ​

Database defines that a database backend should be used as the backend for the virtual cluster. This uses a project called kine under the hood which is a shim for bridging Kubernetes and relational databases.

embedded object ​

Embedded defines that an embedded database (sqlite) should be used as the backend for the virtual cluster

enabled boolean false ​

Enabled defines if the database should be used.

dataSource string ​

DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:

  • mysql: mysql://username:password@tcp(hostname:3306)/vcluster
  • postgres: postgres://username:password@hostname:5432/vcluster
identityProvider string ​

IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:

  • aws: RDS IAM Authentication
keyFile string ​

KeyFile is the key file to use for the database. This is optional.

certFile string ​

CertFile is the cert file to use for the database. This is optional.

caFile string ​

CaFile is the ca file to use for the database. This is optional.

extraArgs string[] [] ​

ExtraArgs are additional arguments to pass to Kine.

external object ​

External defines that an external database should be used as the backend for the virtual cluster

enabled boolean false ​

Enabled defines if the database should be used.

dataSource string ​

DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:

  • mysql: mysql://username:password@tcp(hostname:3306)/vcluster
  • postgres: postgres://username:password@hostname:5432/vcluster
identityProvider string ​

IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:

  • aws: RDS IAM Authentication
keyFile string ​

KeyFile is the key file to use for the database. This is optional.

certFile string ​

CertFile is the cert file to use for the database. This is optional.

caFile string ​

CaFile is the ca file to use for the database. This is optional.

extraArgs string[] [] ​

ExtraArgs are additional arguments to pass to Kine.

connector string ​

Connector specifies a secret located in a connected vCluster Platform that contains database server connection information to be used by Platform to create a database and database user for the vCluster. and non-privileged user. A kine endpoint should be created using the database and user on Platform registration. This is optional.