"x509: certificate signed by an unknown authority" when kubelet starts

Question

I am trying to install kubernetes with kubelet 1.4.5 on the beta version of CoreOS (1192.2.0).

I am using a slightly modified version of the controller and working installation scripts from https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic

therefore, in general, I created licenses for Gentoo Linux using the following bash script:

#!/bin/bash export MASTER_HOST=coreos-2.tux-in.com export K8S_SERVICE_IP=10.3.0.1 export WORKER_IP=10.79.218.3 export WORKER_FQDN=coreos-3.tux-in.com openssl genrsa -out ca-key.pem 2048 openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca" openssl genrsa -out apiserver-key.pem 2048 openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile openssl.cnf openssl genrsa -out ${WORKER_FQDN}-worker-key.pem 2048 openssl req -new -key ${WORKER_FQDN}-worker-key.pem -out ${WORKER_FQDN}-worker.csr -subj "/CN=${WORKER_FQDN}" -config worker-openssl.cnf openssl x509 -req -in ${WORKER_FQDN}-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ${WORKER_FQDN}-worker.pem -days 365 -extensions v3_req -extfile worker-openssl.cnf openssl genrsa -out admin-key.pem 2048 openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=kube-admin" openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 365 echo done 

and this is openssl.cnf

 [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = coreos-2.tux-in.com DNS.2 = coreos-3.tux-in.com IP.1 = 10.3.0.1 IP.2 = 10.79.218.2 IP.3 = 10.79.218.3 

and this is my worker-openssl.cnf

 [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] IP.1 = 10.79.218.3 DNS.1 = coreos-3.tux-in.com 

My controller machine is coreos-2.tux-in.com , which allows lan ip 10.79.218.2

my working machine is coreos-3.tux-in.com , which allows lan ip 10.79.218.3

he created the licenses just fine. but when I use them and install the script controller on the main machine, I see that when running journalctl -xef -u kubelet and I noticed the following messages:

 Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.805868 2018 event.go:208] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping) Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.950827 2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461042 2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461340 2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.024366 2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.171170 2018 eviction_manager.go:162] eviction manager: unexpected err: failed GetNode: node '10.79.218.2' not found Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543619 2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543926 2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority 

Answer 1

in general, the solution was to create another etcd2 port, which connects to the loopback device for each machine and runs on http instead of https . more details in calico-policy-controller requests etcd2 certificates of another Coreos server

Answer 2

The kubelet document says that the --tls-cert-file flag requires that the CA be merged after the certificate. In your case, this is apiserver.pem :

- tls-cert-file File containing x509 certificate for HTTPS. (The CA certificate, if any, is combined after the server certificate). If -tls-cert-file and -tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and stored in the directory passed to -cert-dir.

If I read you the creation of certificates apiserver.pem , apiserver.pem does not contain a root directory.

Answer 3

I am using kubelet with rkt on CoreOS 1192.2.0.

This is the unit that I use to launch kubelet on the desktop:

 [Unit] Description=Kubelet via Hyperkube ACI Requires=k8s-assets.target After=k8s-assets.target [Service] EnvironmentFile=/etc/proxy.env Environment="RKT_OPTS=--volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log" Environment=KUBELET_VERSION=v1.4.0_coreos.0 ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests ExecStart=/usr/lib/coreos/kubelet-wrapper \ --api-servers=https://10.203.69.108 \ --register-node=true \ --allow-privileged=true \ --config=/etc/kubernetes/manifests \ --hostname-override=node2.my.domain \ --cluster_dns=10.3.0.10 \ --cluster_domain=cluster.local \ --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \ --tls-cert-file=/etc/kubernetes/ssl/worker.pem \ --tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem Restart=always RestartSec=10 [Install] WantedBy=multi-user.target 

The important thing is that

- api-servers , which must point to the IP address of the master

- tls-cert-file , which should point to the public key of the employee certificate

- tls-private-key-file , which should point to the private key of the working certificate

- kubeconfig , which should point to a valid kubeconfig file

Here is my kubeconfig file (it contains the path to the CAs that signed the certificates):

 apiVersion: v1 kind: Config clusters: - name: local cluster: certificate-authority: /etc/kubernetes/ssl/ca.pem users: - name: kubelet user: client-certificate: /etc/kubernetes/ssl/worker.pem client-key: /etc/kubernetes/ssl/worker-key.pem contexts: - context: cluster: local user: kubelet name: kubelet-context current-context: kubelet-context 

Answer 4

Your OpenSSL certificates are "self-signed":

 openssl genrsa -out ca-key.pem 2048 openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca" 

That is, you sign them instead of a trusted certification authority . It should be completely beautiful and safe if you keep the secret keys safe.

If you want it to be signed by a certification authority, you will need to create a CSR request (certificate signing request).

https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs

Answer 5

 0. if your issue is : : Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" 1. look at your ca.crt openssl x509 -noout -text -in ca.crt, you will find below info : X509v3 Basic Constraints: CA:FLASE X509v3 Basic Constraints means : "Basic Constraints" identifies if the subject of certificates is a CA who is allowed to issue child certificates. For a certificate that can be used to sign certificates, the info is in some sense duplicated: X509v3 Basic Constraints: CA: TRUE --- Can sign certificates. you should modify it to CA:TRUE through vi openssl.conf [ v3_ca ] basicConstraints = CA:true Regenerate your crts.