Kubernetes Helm stuck with an update in progress
KubernetesKubernetes HelmKubernetes Problem Overview
I tried to run a Helm upgrade before running helm repo update
and now it seems to be permanently stuck in "STATUS: pending-upgrade" and won't let me try to run the upgrade again.
Trying to run:
helm upgrade --namespace coder --install --force --atomic --wait --version 1.13.2 --values ./coder.yaml coder coder/coder
outputs: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Kubernetes Solutions
Solution 1 - Kubernetes
TLDR: You need to rollback to another version first and then helm upgrade
again:
helm rollback <release> <revision> --namespace <namespace>
This can happen for a few reasons, but it ultimately occurs when there's an interruption during the upgrade/install process. Commonly, you SIGKILL
(Ctrl C
) while the deployment is ongoing.
You'll notice that if you helm ls --namespace <namespace>
while it's stuck in STATUS: pending-upgrade
state, you'll see the following without any other information:
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
The best workaround currently is to rollback to another version, and then helm upgrade
again:
helm rollback <release> <revision> --namespace <namespace>
revision
is optional, but you should try to provide it.
more resources:
Solution 2 - Kubernetes
This solution worked for me:
kubectl get secrets
kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>
Following the resolution described in this issue
Solution 3 - Kubernetes
In case is useful to someone, and in response to explicitsoul's comment, what fixed it to me was just:
helm delete <release> -n <namespace>
That removed the pending install (in my case, the first one so I hadn't a previous release to rollback to) and then I was able to run the install again.
What caused the stuck process in my case was a CTRL-C canceling the install command, so don't do that.
Solution 4 - Kubernetes
Here is what worked for me
helm list --all
This will list all the releases with their status
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
rel1 default 1 2021-06-04 14:15:37.652066 +0530 IST deployed rel1-3.32.0 0.46.0
rel2 default 29 2021-06-18 11:02:38.779801 +0530 IST pending-upgrade rel2-0.0.1
rel3 default 3 2021-06-17 11:27:14.608042 +0530 IST deployed rel3-0.0.1
- Notice that rel2 has status as pending-update . This happened because I did a Ctrl+C while upgrade was in progress
- All i had to do was rollback to previous revision in this case 28
helm rollback rel2 28 --namespace default
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
rel1 default 1 2021-06-04 14:15:37.652066 +0530 IST deployed rel1-3.32.0 0.46.0
rel2 default 30 2021-06-18 11:26:07.555547 +0530 IST deployed rel2-0.0.1
rel3 default 3 2021-06-17 11:27:14.608042 +0530 IST deployed rel3-0.0.1
Solution 5 - Kubernetes
I am facing the same. I used helm 3.4.1... It happens when the deployment is in pending and you use --atomic (which in helm3 implies also --wait).
I could not get upgrade working. Worst is that even helm -n code list
did not show anything, so I could not do:
helm -n code code
As helm3 holds such info in secrets, just clean the respective secret(s) and do install (or upgrade --install , but without --atomic). In your case something like
helm delete --namespace code secret sh.helm.release.v1.code.v1
(where last v1 is the release number, so maybe list and delete all if you are ok with that).
and afterwars helm install.
NOTE: old objects (pods,etc) will be there, so the new install will try to merge things. It went OK for me, but note -> It's a HACK :)
Solution 6 - Kubernetes
These are the steps that worked for me:
-
See the status of your deployment (my was pending all the time)
helm list --all
-
Rollback to the previous version, for me was working already here, next step optional
helm rollback <NAMESPACE_NAME> <Previous Version> --namespace <NAMESPACE_NAME>
-
In case you want new/another deploy
helm upgrade . . .
Solution 7 - Kubernetes
kubectl get secrets
kubectl delete secret sh.helm.release.v1.
By using the above command, it will remove the existing secrets in the middle of helm upgrade which will run and delete the stucked helm upgrade and it will generate the new one to proceed with the new upgrade.
Solution 8 - Kubernetes
In order to rollback to previous version you can just pass the release name:
helm rollback <RELEASE_NAME>
Where RELEASE_NAME
can be seen when you run helm list --all
-> under the NAME
column.
(*) Add --namespace <namespace>
if terminal context is not set to namespace.