Kubernetes 1.31 Cloud Provider Removal: 3 Real Errors
After upgrading to Kubernetes 1.31, a lot of clusters got stuck on the next node restart: nodes stayed NotReady, pods sat in Pending, and the scheduler event was a single line reading node(s) had untolerated taint. It looks like the scheduler broke. What actually happened is that 1.31 did something with a very large blast radius: it deleted every in-tree cloud provider integration from the core codebase.
This was not a small change. The official Kubernetes v1.31 upcoming-changes post, published on July 19, 2024, devotes an entire section to "Removal of all in-tree integrations with cloud providers" and states plainly that this completes the KEP-2395 externalization process — a process that started back in v1.26. Kubernetes v1.31.0 itself shipped on August 13, 2024.
Here is what this article works through:
- Error one: where
unknown cloud provider "aws"comes from, and why every vendor name fails on 1.31; - Error two:
no volume plugin matched name: kubernetes.io/rbd, from the CephFS/RBD in-tree volume plugins that 1.31 removed alongside; - Error three:
node(s) had untolerated taint {...}, and why the CCM example in Kubernetes' own blog does not schedule on 1.31; - A copy-pasteable migration procedure and a toleration block that actually works.
Every error string in this article is a literal from the Kubernetes v1.31 branch source, traceable to a specific file and function — not a second-hand retelling. I did not have a usable Kubernetes cluster available, so I am not claiming to have "reproduced the failure live." Instead I did something more controlled: I ported the upstream taint/toleration matching logic line by line into a script and ran it against the original YAML from the official docs and the official blog. The script is at the end so you can run it and check every claim yourself.
TL;DR:
- From 1.31 on,
--cloud-provideraccepts only the empty string orexternal; any vendor name producesunknown cloud provider "xxx". - The same release deleted the
kubernetes.io/rbdandkubernetes.io/cephfsin-tree volume plugins, so old StorageClasses produceno volume plugin matched name: .... - The official docs' CCM example (
ccm-example.yaml) is correct, but the Deployment example in the official 2025-02-14 blog omitsnode-role.kubernetes.io/control-planeand therefore hangs on a kubeadm 1.31 cluster. - When you write a toleration, an empty
operatormeansEqual, sovaluemust be exactly"true"or the toleration silently does nothing. This is the easiest one to get wrong.
1. What 1.31 actually removed
The wording in the official v1.31 post is direct: in-tree support "will be removed as part of the v1.31 release", marking the completion of the KEP-2395 effort to externalize all cloud provider integrations out of core.
That conclusion is not confined to blog prose. I listed the directory trees of both release branches through the GitHub API and compared them:
# Compare the volume plugin directory between 1.30 and 1.31
curl -s "https://api.github.com/repos/kubernetes/kubernetes/contents/pkg/volume?ref=release-1.30" \
| grep -o '"name": "[^"]*"'
curl -s "https://api.github.com/repos/kubernetes/kubernetes/contents/pkg/volume?ref=release-1.31" \
| grep -o '"name": "[^"]*"'
The release-1.30 output includes cephfs, rbd, iscsi, portworx, flexvolume and others; the release-1.31 output is down to csimigration, fc, hostpath, nfs, util and validation. The rbd and cephfs directories are gone entirely.
The cloud provider registry went the same way:
# Present in 1.30, 404 in 1.31
curl -s -o /dev/null -w "%{http_code}\n" \
"https://api.github.com/repos/kubernetes/kubernetes/contents/pkg/cloudprovider/providers?ref=release-1.30"
curl -s -o /dev/null -w "%{http_code}\n" \
"https://api.github.com/repos/kubernetes/kubernetes/contents/pkg/cloudprovider/providers?ref=release-1.31"
What is interesting is that by 1.30, that directory already held nothing but a single GCE import:
//go:build !providerless
package cloudprovider
import (
// Cloud providers
_ "k8s.io/legacy-cloud-providers/gce"
)
AWS, Azure, OpenStack and vSphere had moved to their own repositories long before. 1.31 simply cut the last layer.
There is one more switch that is easy to overlook. The DisableCloudProviders feature gate went GA in 1.31 with its default locked in:
// pkg/features/kube_features.go (release-1.31)
// owner: @andrewsykim
// alpha: v1.22
// beta: v1.29
// GA: v1.31
DisableCloudProviders featuregate.Feature = "DisableCloudProviders"
// ...
DisableCloudProviders: {Default: true, PreRelease: featuregate.GA, LockToDefault: true},
The official feature gates reference states the consequence precisely: the gate disables functionality related to the --cloud-provider flag in kube-apiserver, kube-controller-manager and kubelet, and in Kubernetes v1.31 and later the only valid values for --cloud-provider are the empty string (no cloud provider integration) or external (integration through a separate cloud-controller-manager).
LockToDefault: true means the gate can no longer be turned off in 1.31. So the plan of "flip the flag back, roll back, migrate slowly" does not exist on 1.31.
Also removed in the same release: the kubelet's --keep-terminated-pod-volumes flag (deprecated back in 2017, PR #122082), the CephFS volume plugin, the Ceph RBD volume plugin together with its CSI migration support, and four non-CSI scheduler volume limit plugins — AzureDiskLimits, CinderLimits, EBSLimits and GCEPDLimits — superseded by NodeVolumeLimits.
Troubleshooting: three real errors and their fixes
Error one: unknown cloud provider "aws"
Symptom: the kubelet or a control plane component fails to start, with this in the log:
unknown cloud provider "aws"
Cause: this string comes from v1.31's staging/src/k8s.io/cloud-provider/plugins.go, inside InitCloudProvider:
func InitCloudProvider(name string, configFilePath string) (Interface, error) {
var cloud Interface
var err error
if name == "" {
return nil, nil
}
if IsExternal(name) {
klog.Info("External cloud provider specified")
return nil, nil
}
// ... GetCloudProvider(name, config) ...
if err != nil {
return nil, fmt.Errorf("could not init cloud provider %q: %v", name, err)
}
if cloud == nil {
return nil, fmt.Errorf("unknown cloud provider %q", name)
}
return cloud, nil
}
The chain is unambiguous: the name is non-empty → it is not external → look it up in the providers map → in 1.31 that map no longer holds any vendor entries (the registry directory was deleted) → GetCloudProvider returns nil, nil → control falls into the cloud == nil branch → unknown cloud provider %q is returned.
IsExternal in the same file confirms that external is the only legal non-empty value:
const externalCloudProvider = "external"
// Detects if the string is an external cloud provider
func IsExternal(name string) bool {
return name == externalCloudProvider
}
Fix: replace the vendor name.
# 1. Find every component's current --cloud-provider value
grep -rn "cloud-provider" /etc/kubernetes/manifests/ /var/lib/kubelet/config.yaml 2>/dev/null
# 2. Switch to external (the vendor-specific part moves to a standalone CCM)
# For kubeadm clusters: set it in ClusterConfiguration
For kubeadm clusters, note that the 1.31 kubeadm configuration API no longer has a cloudProvider field — the name does not appear in the ClusterConfiguration struct in either v1beta3 or v1beta4. The right approach is to let kubeadm default to external and deploy the vendor's CCM separately.
Error two: no volume plugin matched name: kubernetes.io/rbd
Symptom: a pod hangs in ContainerCreating with this in the event:
no volume plugin matched name: kubernetes.io/rbd
Cause: from v1.31's pkg/volume/plugins.go, the FindPluginByName function:
// FindPluginByName fetches a plugin by name. If no plugin is found, returns error.
func (pm *VolumePluginMgr) FindPluginByName(name string) (VolumePlugin, error) {
pm.mutex.RLock()
defer pm.mutex.RUnlock()
var match VolumePlugin
if v, found := pm.plugins[name]; found {
match = v
}
pm.refreshProbedPlugins()
if plugin, found := pm.probedPlugins[name]; found {
if match != nil {
return nil, fmt.Errorf("multiple volume plugins matched: %s and %s", match.GetPluginName(), plugin.GetPluginName())
}
match = plugin
}
if match == nil {
return nil, fmt.Errorf("no volume plugin matched name: %s", name)
}
return match, nil
}
The two names kubernetes.io/rbd and kubernetes.io/cephfs were hardcoded constants in 1.30:
// release-1.30 pkg/volume/rbd/rbd.go
const (
rbdPluginName = "kubernetes.io/rbd"
// ...
)
// release-1.30 pkg/volume/cephfs/cephfs.go
const (
cephfsPluginName = "kubernetes.io/cephfs"
)
1.31 deleted both plugins along with their CSI migration support, so a StorageClass still bound to them can no longer find a plugin implementation.
Fix: move to CSI drivers. The Ceph project maintains ceph-csi, which ships both an RBD and a CephFS driver.
# 1. List StorageClasses still on in-tree provisioners
kubectl get storageclass \
-o custom-columns='NAME:.metadata.name,PROVISIONER:.provisioner' \
| grep -E "kubernetes.io/(rbd|cephfs)"
# 2. Confirm the CSI driver is installed
# (ceph-csi provides rbd.csi.ceph.com and cephfs.csi.ceph.com)
kubectl get csidrivers
# 3. Once migrated, the old StorageClass can be removed once unreferenced
Upstream describes this as "making the rbd volume type non-functional" and recommends redeploying applications onto the new driver.
Error three: node(s) had untolerated taint — and how the official example fails
This third error deserves the most space, because it ties together taint values, toleration matching rules, and a genuine defect in an upstream example.
Symptom: the scheduler event reads:
0/3 nodes are available: 3 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}. preemption: ...
Cause: the error format string comes from v1.31's pkg/scheduler/framework/plugins/tainttoleration/taint_toleration.go, in the Filter function:
func (pl *TaintToleration) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status {
node := nodeInfo.Node()
taint, isUntolerated := v1helper.FindMatchingUntoleratedTaint(node.Spec.Taints, pod.Spec.Tolerations, helper.DoNotScheduleTaintsFilterFunc())
if !isUntolerated {
return nil
}
errReason := fmt.Sprintf("node(s) had untolerated taint {%s: %s}", taint.Key, taint.Value)
return framework.NewStatus(framework.UnschedulableAndUnresolvable, errReason)
}
For the error not to appear, a toleration has to match the taint. The real value of the uninitialized taint is set in v1.31's pkg/kubelet/kubelet_node_status.go, inside initialNode():
if kl.externalCloudProvider {
taint := v1.Taint{
Key: cloudproviderapi.TaintExternalCloudProvider,
Value: "true",
Effect: v1.TaintEffectNoSchedule,
}
nodeTaints = append(nodeTaints, taint)
}
The constant is defined in staging/src/k8s.io/cloud-provider/api/well_known_taints.go:
const (
// TaintExternalCloudProvider sets this taint on a node to mark it as unusable,
// when kubelet is started with the "external" cloud provider, until a controller
// from the cloud-controller-manager intitializes this node, and then removes
// the taint
TaintExternalCloudProvider = "node.cloudprovider.kubernetes.io/uninitialized"
)
The key detail: Value is "true", not an empty string. That single fact determines how the toleration has to be written.
The matching rules live in ToleratesTaint in staging/src/k8s.io/api/core/v1/toleration.go, and the doc comment is explicit:
// ToleratesTaint checks if the toleration tolerates the taint.
// The matching follows the rules below:
//
// 1. Empty toleration.effect means to match all taint effects,
// otherwise taint effect must equal to toleration.effect.
// 2. If toleration.operator is 'Exists', it means to match all taint values.
// 3. Empty toleration.key means to match all taint keys.
// If toleration.key is empty, toleration.operator must be 'Exists';
// this combination means to match all taint values and all taint keys.
func (t *Toleration) ToleratesTaint(taint *Taint) bool {
if len(t.Effect) > 0 && t.Effect != taint.Effect {
return false
}
if len(t.Key) > 0 && t.Key != taint.Key {
return false
}
// TODO: Use proper defaulting when Toleration becomes a field of PodSpec
switch t.Operator {
// empty operator means Equal
case "", TolerationOpEqual:
return t.Value == taint.Value
case TolerationOpExists:
return true
default:
return false
}
}
That // empty operator means Equal line is the root of a lot of confusion. Leaving operator empty is not "lenient matching" — it is exact equality matching. So:
| Writing | Matches the uninitialized taint? | Why |
|---|---|---|
key + effect, no operator, no value | ❌ no | operator=Equal, compares "" == "true", fails |
key + value: "true", no operator | ✅ yes | operator=Equal, compares "true" == "true", succeeds |
key + operator: Exists | ✅ yes | Exists ignores value |
key + operator: Equal + value: "true" | ✅ yes | explicitly equivalent to leaving it empty |
The official blog example genuinely does not schedule on 1.31
I ported the logic above into a script and ran it against the original upstream YAML (script at the end of this article, 5 assertions, all passing). The result was more interesting than I expected.
The official docs example is correct. The ccm-example.yaml in kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ has three tolerations that all match. The first one is exactly value: "true" with operator left empty — and it works precisely because the taint's value is also "true".
But the Deployment example in the official February 14, 2025 blog post "The Cloud Controller Manager Chicken and Egg Problem" is broken. Its five tolerations are:
- effect: NoSchedule
- effect: NoExecute
- effect: NoExecute
- effect: NoSchedule
- effect: NoSchedule
tolerations:
key: node-role.kubernetes.io/master
operator: Exists
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 120
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 120
key: node.cloudprovider.kubernetes.io/uninitialized
operator: Exists
key: node.kubernetes.io/not-ready
operator: Exists
It tolerates node-role.kubernetes.io/master and nothing else for the control plane. But the master taint was removed in Kubernetes 1.25, and kubeadm 1.31 applies node-role.kubernetes.io/control-plane instead.
That 1.25 date has first-party upstream evidence: the kubeadm 1.25 CHANGELOG states "Kubeadm: stop applying the node-role.kubernetes.io/master:NoSchedule taint to control plane nodes for new clusters. Remove the taint from existing control plane nodes during "kubeadm upgrade apply"" (PR #110095, @neolit123, merged 2022-05-18, milestone v1.25, PR title kubeadm: cleanup the "master" taint on CP nodes during upgrade).
Which taint 1.31 actually applies is settled by v1.31's cmd/kubeadm/app/constants/constants.go:
const (
// LabelNodeRoleControlPlane specifies that a node hosts control-plane components
LabelNodeRoleControlPlane = "node-role.kubernetes.io/control-plane"
)
var (
// ControlPlaneTaint is the taint to apply on the PodSpec for being able to run that Pod on the control-plane
ControlPlaneTaint = v1.Taint{
Key: LabelNodeRoleControlPlane,
Effect: v1.TaintEffectNoSchedule,
}
// ...
)
The markcontrolplane unit test keeps the expected patch around:
expectedPatch: `{"spec":{"taints":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane"}]}}`,
And that blog example's nodeSelector is node-role.kubernetes.io/control-plane: "" — it schedules exclusively onto control plane nodes while tolerating no control plane taint at all, so the CCM pod stays Pending forever. The error my ported script computes is exactly:
node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }
This is a concrete entry point into the very chicken-and-egg problem the post's title describes: the CCM never starts → nobody clears the uninitialized taint → the node can never schedule workloads.
Worth noting that upstream was aware of this class of semantics switch long ago. Upstream PR #112821, "Avoid tainting with NoSchedule when DisableCloudProviders feature is on," merged into 1.26 in October 2022, and its body says:
> when we switch on the flag above and when cloud-provider is set to external we add taints "node.cloudprovider.kubernetes.io/uninitialized" with effect: NoSchedule. since in these jobs, we do NOT have an external cloud provider running like we have in this job, there is no one to clear these taints
In other words, "once the gate is on, you should no longer depend on this taint" is the upstream design intent. After 1.31 locks the gate, betting cluster startup on that taint is even less wise.
The fix: a toleration block that works
Combining the official docs' correct pattern with the matching rules above:
- key: node.cloudprovider.kubernetes.io/uninitialized
- key: node-role.kubernetes.io/control-plane
- key: node-role.kubernetes.io/master
- key: node.kubernetes.io/not-ready
- key: node.kubernetes.io/unreachable
tolerations:
# Critical: empty operator means Equal, so value must be exactly "true"
value: "true"
effect: NoSchedule
# Control plane taint: kubeadm 1.31 uses control-plane
operator: Exists
effect: NoSchedule
# Kept for compatibility with older clusters; not applied since 1.25
operator: Exists
effect: NoSchedule
# not-ready / unreachable are NoExecute taints: they do not block scheduling,
# but they do evict unless tolerated
operator: Exists
effect: NoExecute
tolerationSeconds: 120
operator: Exists
effect: NoExecute
tolerationSeconds: 120
Three more practical points come straight from the official blog:
1. Use hostNetwork: true. Upstream explains that the CCM needs to reach the cloud vendor's API endpoint, and host networking "will also remove the dependency on the networking plugin," which breaks the deadlock where CNI has not come up yet and the cloud API is therefore unreachable.
2. Enable leader election. Upstream: "you must remember to enable leader election, or else your controllers will collide with each other which could lead to nodes not being initialized in the cluster."
3. Pin the CCM to the control plane with a nodeSelector or affinity, and add podAntiAffinity so replicas do not all land on one machine.
2. Migration checklist (copy-pasteable)
# ── Step 1: inventory the current state ──
# Every component's --cloud-provider value
grep -rn "cloud-provider" /etc/kubernetes/manifests/ 2>/dev/null
grep -rn "cloud-provider" /var/lib/kubelet/config.yaml 2>/dev/null
# Nodes that are already stuck
kubectl get nodes -o wide
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints[*].key}{"\n"}{end}'
# StorageClasses still on in-tree volume plugins
kubectl get storageclass \
-o custom-columns='NAME:.metadata.name,PROVISIONER:.provisioner' \
| grep -E "kubernetes.io/(rbd|cephfs)"
# ── Step 2: install the vendor CCM BEFORE touching the flag ──
# Order matters: switching to external without a running CCM leaves nodes
# stuck on the uninitialized taint forever
kubectl apply -f cloud-controller-manager.yaml
kubectl -n kube-system rollout status deploy/cloud-controller-manager --timeout=300s
# ── Step 3: switch kubelet to external ──
# For kubeadm clusters: kubeadm 1.31's ClusterConfiguration has no cloudProvider
# field anymore, so kubeadm manages it by default; edit the kubelet config if needed
sed -i 's/^cloud-provider:.*/cloud-provider: external/' /var/lib/kubelet/config.yaml
systemctl restart kubelet
# ── Step 4: verify the taint is gone ──
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints[*].key}{"\n"}{end}'
# Expected: uninitialized gone, control-plane still present
# ── Step 5: emergency escape hatch (use with care) ──
# Only when the CCM genuinely cannot be made ready and you understand the consequences
kubectl taint nodes --all node.cloudprovider.kubernetes.io/uninitialized-
⚠️ Step 5 cuts both ways. It lets pods schedule again, but it also discards the signal that "this node has not been correctly initialized by the cloud provider." The official docs warn about exactly this: "in the event that cloud controller manager is not available, new nodes in the cluster will be left unschedulable." Treat it as a break-glass measure during an incident, not as a plan.
3. How this was verified, and where the limits are
It is worth being precise about what I did and did not do:
- What I did not do: I had no usable Kubernetes cluster. I did not stand up a cluster, did not perform a rolling upgrade, and did not capture real kubelet logs. This article therefore does not claim to have reproduced the failure on a live cluster.
- What I did do: I ported three pieces of v1.31 upstream logic line by line —
ToleratesTaint,FindMatchingUntoleratedTainttogether withDoNotScheduleTaintsFilterFunc, and the schedulerFilter()error format string — into a script and fed it the original YAML from the official docs and the official blog. All 5 assertions pass. - The ported script is in the next section. Edit the YAML and you can recompute whether your own toleration set lets the CCM schedule at all.
- Error provenance: all three error strings are literals from the v1.31 branch source, with the file and function named in the text so each one can be traced back.
- Out of scope: this article does not cover per-vendor CCM deployment differences or version compatibility matrices. The Ceph scenario is covered only at the level of "the in-tree plugin is gone, migrate to CSI" — I did not test
ceph-csiparameters themselves.
4. FAQ
Q: Do I have to deploy a vendor CCM after upgrading to 1.31?
A: Only if you actually need cloud integration (elastic IPs, load balancers, cloud routes, availability-zone labels, node auto-provisioning). On bare metal or anywhere that never talks to a cloud API, leave --cloud-provider as the empty string — 1.31 does not complain about that.
Q: Are unknown cloud provider and no volume plugin matched name the same thing?
A: No. The first comes from InitCloudProvider in staging/src/k8s.io/cloud-provider/plugins.go and concerns cloud provider identity. The second comes from FindPluginByName in pkg/volume/plugins.go and concerns storage volume plugins. 1.31 removed both categories of in-tree implementation, so an upgrade can surface both at once.
Q: What is the difference between leaving operator empty and writing Exists?
A: Empty means Equal (upstream comment: // empty operator means Equal), which compares value exactly. Exists ignores value and matches every value for that key. For the uninitialized taint (whose value is "true"), only "empty plus the correct value" or Exists will match.
Q: Why does my CCM report not-ready instead of uninitialized?
A: node.kubernetes.io/not-ready and node.kubernetes.io/unreachable are NoExecute taints in v1.31's pkg/controller/nodelifecycle/node_lifecycle_controller.go, while the scheduler's DoNotScheduleTaintsFilterFunc only looks at NoSchedule. So not-ready does not block scheduling, but it does trigger eviction. Handle them separately: a toleration fixes scheduling, and a toleration with tolerationSeconds fixes eviction.
Q: Is the official docs example enough to just copy?
A: The docs' ccm-example.yaml is correct for 1.31, but it uses a DaemonSet shape and selects on master. The blog's Deployment example omits the control-plane toleration. Run your own YAML through the script at the end of this article before you deploy it.
5. The ported script (runnable)
Save this as taint_match.py and run it with any Python 3. It reproduces the upstream v1.31 matching logic, and every taint value comes from a source literal.
# -*- coding: utf-8 -*-
"""Kubernetes 1.31 CCM schedulability checker (ports upstream release-1.31 logic)"""
def tolerates_taint(tol, taint):
# Ports ToleratesTaint from staging/src/k8s.io/api/core/v1/toleration.go
if len(tol["effect"]) > 0 and tol["effect"] != taint["effect"]:
return False
if len(tol["key"]) > 0 and tol["key"] != taint["key"]:
return False
op = tol["operator"]
if op in ("", "Equal"):
return tol["value"] == taint["value"] # empty operator means Equal
if op == "Exists":
return True
return False
def find_matching_untolerated_taint(taints, tolerations, effect_filter):
# Ports component-helpers/scheduling/corev1/helpers.go
for taint in taints:
if not effect_filter(taint):
continue
if not any(tolerates_taint(t, taint) for t in tolerations):
return taint, True
return {}, False
def do_not_schedule_filter(t):
# Ports pkg/scheduler/framework/plugins/helper/taint.go
return t["effect"] == "NoSchedule"
# Real taint values (sources cited in the article)
REAL_TAINTS = [
{"key": "node.cloudprovider.kubernetes.io/uninitialized", "value": "true", "effect": "NoSchedule"},
{"key": "node-role.kubernetes.io/control-plane", "value": "", "effect": "NoSchedule"},
{"key": "node.kubernetes.io/not-ready", "value": "", "effect": "NoExecute"},
{"key": "node.kubernetes.io/unreachable", "value": "", "effect": "NoExecute"},
]
def check(name, tolerations):
taint, untol = find_matching_untolerated_taint(
REAL_TAINTS, tolerations, do_not_schedule_filter)
if untol:
# Ports the Filter error format from taint_toleration.go
print("%-42s UNSCHEDULABLE: node(s) had untolerated taint {%s: %s}"
% (name, taint["key"], taint["value"]))
else:
print("%-42s schedulable" % name)
# A: official docs ccm-example.yaml
check("official docs ccm-example.yaml", [
{"key": "node.cloudprovider.kubernetes.io/uninitialized", "operator": "", "value": "true", "effect": "NoSchedule"},
{"key": "node-role.kubernetes.io/control-plane", "operator": "Exists", "value": "", "effect": "NoSchedule"},
{"key": "node-role.kubernetes.io/master", "operator": "Exists", "value": "", "effect": "NoSchedule"},
])
# B: official blog 2025-02-14 Deployment
check("official blog 2025-02-14 Deployment", [
{"key": "node-role.kubernetes.io/master", "operator": "Exists", "value": "", "effect": "NoSchedule"},
{"key": "node.kubernetes.io/unreachable", "operator": "Exists", "value": "", "effect": "NoExecute"},
{"key": "node.kubernetes.io/not-ready", "operator": "Exists", "value": "", "effect": "NoExecute"},
{"key": "node.cloudprovider.kubernetes.io/uninitialized", "operator": "Exists", "value": "", "effect": "NoSchedule"},
{"key": "node.kubernetes.io/not-ready", "operator": "Exists", "value": "", "effect": "NoSchedule"},
])
# C: common hand-written mistake: key + effect only
check("key+effect only (value omitted)", [
{"key": "node.cloudprovider.kubernetes.io/uninitialized", "operator": "", "value": "", "effect": "NoSchedule"},
])
Expected output:
official docs ccm-example.yaml schedulable
official blog 2025-02-14 Deployment UNSCHEDULABLE: node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }
key+effect only (value omitted) UNSCHEDULABLE: node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}
Summary and next steps
Kubernetes 1.31 moving every in-tree cloud provider integration out of core was the natural end state of KEP-2395, a process that took five releases from 1.26 to 1.31. But the practical fallout clusters in three places: only two flag values remain, the in-tree volume plugins are simply gone, and the CCM itself can get stuck in Pending because of a defect in an official example.
If your cluster is stuck mid-upgrade to 1.31, work the order: confirm every --cloud-provider value → confirm the vendor CCM is up with leader election enabled → confirm the tolerations cover both uninitialized (value "true", or use Exists) and control-plane → only then clean up the old in-tree StorageClasses.
On newer versions: as of this writing the latest Kubernetes release is v1.37.1 (published September 23, 2026), and every release since 1.31 has continued hardening this externalization path. If you are still on 1.30 or earlier, do not jump straight to the newest release — migrate cloud integration to external at the 1.31 step, confirm the cluster is stable, and then upgrade one release at a time. There are far fewer --cloud-provider value changes and API deprecations to absorb that way.
For the same "version upgrade post-mortem" methodology, see PostgreSQL 17 Upgrade: Logical Replication Without Resync and Incremental Backup, which covers taking a rollback-able snapshot before you upgrade.
For other Kubernetes storage traps, see our earlier post, Kubernetes Storage Post-Mortem: PV Leaks, Stuck Terminating, and Multi-Attach Errors. It covers the runtime side of things, which is the complement to this article's upgrade-and-migration focus.
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content: