Kubernetes certificate errors in GitLab CI: not the code
When kubectl in GitLab CI fails with x509 or expired certificate errors, separate the issue from the code and narrow it to kubeconfig, CA and credentials.
The deploy job that worked yesterday is red today. The code hasn't changed, or the change is harmless. At the end of the log you find one of these lines:
Unable to connect to the server: x509: certificate signed by unknown authority
Unable to connect to the server: x509: certificate has expired or is not yet valid
error: You must be logged in to the server (Unauthorized)
What these errors have in common: the pipeline can't establish a secure connection to the Kubernetes API. The application code never even gets to run. So the first job is to separate the problem from the code.
1. Isolate the problem
Add a minimal check step to the pipeline, before the deploy:
k8s-check:
stage: .pre
image: bitnami/kubectl:latest
script:
- kubectl version
- kubectl auth can-i get pods -n "$KUBE_NAMESPACE"
If this step fails too, the problem is definitely on the connection and identity side.
2. Read the error
| Error | Most likely cause |
|---|---|
signed by unknown authority |
The CA in the kubeconfig (certificate-authority-data) doesn't match the server's certificate, or is missing entirely |
certificate has expired |
The client certificate or the API server certificate has expired |
certificate is valid for …, not … |
The server: address isn't in the certificate's SAN list (DNS name instead of IP, or the other way around) |
Unauthorized |
The token has been revoked, has expired, or belongs to the wrong user |
3. Pass the kubeconfig correctly
The most common mistake is defining the kubeconfig as a regular variable and treating its contents as an environment variable. In GitLab, store the kubeconfig as a CI/CD variable of type File. The variable then gives you the path to the file:
deploy:
script:
- export KUBECONFIG="$KUBECONFIG_FILE"
- kubectl config view --minify # see which cluster and user you are connecting with
- kubectl apply -f k8s/
In the output of kubectl config view --minify, check that the server: address points to the right cluster and that the expected user is in use.
4. Check certificate lifetimes
Many distributions issue client certificates with a limited lifetime (often one year), and they need to be renewed. To see the expiry date of the client certificate in your kubeconfig:
kubectl config view --raw --minify \
-o jsonpath='{.users[0].user.client-certificate-data}' \
| base64 -d | openssl x509 -noout -enddate
If the certificates were renewed on the cluster side, the kubeconfig stored in the CI variable must be updated as well. Otherwise the pipeline keeps trying to connect with the old credentials.
5. The lasting fix
- For CI, use a ServiceAccount that only has deploy permissions instead of a personal kubeconfig.
- Document where the kubeconfig comes from and when it needs to be renewed.
- Add an alert for certificate expiry dates to your monitoring system, so the surprise doesn't land on the next deploy day.
Lesson
A red pipeline doesn't always mean bad code. First, take a look: which layer is the error in? If the connection, identity and certificates aren't sound, it doesn't matter what the code does.