Nginx Ingress: service "ingress-nginx-controller-admission" not found
KubernetesNginx IngressKubernetes Problem Overview
We created a kubernetes cluster for a customer about one year ago with two environments; staging and production separated in namespaces. We are currently developing the next version of the application and need an environment for this development work, so we've created a beta environment in its own namespace.
This is a bare metal kubernetes cluster with MetalLB and and nginx-ingress. The nginx ingress controllers is installed with helm and the ingresses are created with the following manifest (namespaces are enforced by our deployment pipeline and are not visible in the manifest):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: api-ingress
annotations:
#ingress.kubernetes.io/ssl-redirect: "true"
#kubernetes.io/tls-acme: "true"
#certmanager.k8s.io/issuer: "letsencrypt-staging"
#certmanager.k8s.io/acme-challenge-type: http01
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Robots-Tag: noindex, nofollow";
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, OPTIONS"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
spec:
tls:
- hosts:
- ${API_DOMAIN}
secretName: api-cert
rules:
- host: ${API_DOMAIN}
http:
paths:
- backend:
serviceName: api
servicePort: 80
When applying the manifest kubernetes responds with the following error:
> Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: service "ingress-nginx-controller-admission" not found
I've attempted to update the apiVersion of the ingress manifest to networking.k8s.io/v1beta1
(this is the apiVersion the new nginx-ingress controllers are installed with via helm), but I'm getting the same error.
My initial suspicion is that this is related to a change in the nginx-ingress between the current installation and the installation from one year ago, even if the ingress controllers are separated by namespaces. But i cant find any services called ingress-nginx-controller-admission
in any of my namespaces, so I'm clueless how to proceed.
Kubernetes Solutions
Solution 1 - Kubernetes
I had the same problem and found a solution from another SO thread.
I had previously installed nginx-ingress using the manifests. I deleted the namespace it created, and the clusterrole and clusterrolebinding as noted in the documentation, but that does not remove the ValidatingWebhookConfiguration
that is installed in the manifests, but NOT when using helm by default. As Arghya noted above, it can be enabled using a helm parameter.
Once I deleted the ValidatingWebhookConfiguration
, my helm installation went flawlessly.
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
Solution 2 - Kubernetes
You can check if there is a validation webhook and a service. If they don't exist double check the deployment and add these.
kubectl get -A ValidatingWebhookConfiguration
NAME CREATED AT
ingress-nginx-admission 2020-04-22T15:01:33Z
kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.212.217 <none> 80:32268/TCP,443:32683/TCP 2m34s
ingress-nginx-controller-admission ClusterIP 10.96.151.42 <none> 443/TCP 2m34s
Deployment yamls here have the webhook and service.
Since you have used helm to install it you can enable/disable the webhook via a helm parameter as defined here
Solution 3 - Kubernetes
There is some issue with SSL cert it seems in the webhook.
Chaning failurePolicy: Fail to Ignore worked for me in the
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml
for more info check:
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
Solution 4 - Kubernetes
my problem is proven to be a ssl cert issue. after I delete"ValidatingWebhookConfiguration", the issue is resolved
Solution 5 - Kubernetes
For me issue was with Kubernetes version 1.18 and I upgraded to 1.19.1 and it worked just fine.
Pod status
k get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-cgpj7 0/1 Completed 0 3m44s
ingress-nginx-admission-patch-mksxs 0/1 Completed 0 3m44s
ingress-nginx-controller-5fb6f67b9c-ps67k 0/1 CrashLoopBackOff 5 3m45s
Error logs from pod
I0916 07:15:34.317477 8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
F0916 07:15:34.318721 8 main.go:107] ingress-nginx requires Kubernetes v1.19.0 or higher
k get po -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-2tk8p 0/1 Completed 0 104s
ingress-nginx-admission-patch-nlv5w 0/1 Completed 0 104s
ingress-nginx-controller-79c4d49bb9-7bgcj 1/1 Running 0 105s