Kubernetes security

 Security primitives - Authentication

User - managed by kube-apiserver

1. Admins

2. Developers

Static file methods - not recommended authentication

1. static password file

2. static token file (hash the password)

Example static password file

# User File Contents

password123,user1,u0001

password123,user2,u0002

password123,user3,u0003

password123,user4,u0004

password123,user5,u0005

define specific user in curl: curl -v -k https://<service> -u "user1:password"

TLS Certificates

CSR: certificate signing request

Public key: *.crt / *.pem

Private key: *.key / *-key.pem

CA: root certificate - 10 years

Server: server certificate - 1 year

Client: client certificate - 1 year

CA certificate: ca.crt / ca.key

server certificate

1. Master node

Kube-apiserver: apiserver.crt / apiserver.key

Etcd server: etcdsever.crt / etcdserver.key

1. Work node

kubelet server: kubelet.crt / kubectl.key

Client certificate

1. admin: admin.crt / admin.key

2. kube scheduler: scheduler.crt / scheduler.key

3. kube controller: controller-manager.crt / controller-manager.key

4. kube proxy: kube-proxy.crt / kube-proxy.key

5. kube-apiserver: apiserver-kubelet-client.crt / apiserver-kubelet-client.key

6. kubelet: kubelet-client.crt / kubelet-client.key

Certificate generation

1. CA

generate key #openssl genrsa -out ca.key 2048

certificate signing request #openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr

sign certificates #openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

2. client certificate

for example: admin

generate key #openssl genrsa -out admin.key 2048

certificate signing request #openssl req -new -key admin.key -subj "/CN=kube-admin" -out admin.csr

sign certificates #openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

3. Server certificate

kube-apiserver

Generate key #openssl genrsa -out apiserver.key 2048

Certificate signing request #openssl req -new -key apiserver.key -subj="/CN=kube-apiserver" -out apiserver.csr -config openssl.cnf

Sign certificate #openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -out apiserver.crt

openssl.cnf need has alt_name information

Error: ca.srl not found need to add parameter -CAcreateserial

Renew expired key, and csr file exist

#openssl x509 -req -in apiserver-etcd-client.csr -CA ca.crt -Cakey ca.key -CAcreateserial -out apiserver-etcd-client.crt

Example

Check certificate in cluster

Get the idea how the cluster setup

1. Kubeadm --> /etc/kubernetes/manifests/kube-apiserver.yaml

2. Manually setup --> /etc/systemd/system/kube-apiserver.service

kubeadm setup cluster

1. Find the path

#less /etc/kubernetes/manifests/kube-apiserver.yaml

2. Check the certificate file

#openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

Check certificate example

Check certificate issue

Check log #journalctl -u etcd.service -l

Check log #kubectl logs etcd-master

Check log (without kube-apiserver) #docker ps -a

#docker logs <container id>

Certificates API

New user - jane

Generate key #openssl genrsa -out jane.key 2048

Certificate signing request #openssl req -new -key jane.key -subj="/CN=jane" -out jane.csr

Create the yaml file depending on the csr file

Example

Create csr #kubectl create -f <csr file>

Check the csr #kubectl get csr

Approve the csr #kubectl certificate approve <csr name>

Reject the csr #kubectl certificate deny <csr name>

View the certificate #kubectl get csr <csr name> -o yaml

Delete csr #kubectl delete csr <csr name>

Certificate all made by controller manager

Kubectl config file

Default config file: <user home>/.kube/config

Config file format

1. Clusters: environment

2. Contexts: map user and environment

3. Users

View #kubectl config view

Edit #kubectl config -h

API Groups

Check version of kube-apiserver curl https://kube-master:6443/version

Check pods information curl https://kube-master:6443/api/v1/pods

get information curl https://kube-master:6443 -k

get apis information curl https://kube-master:6443/apis -k | grep "name"

if you got forbidden issue, need to add auth in curl

#curl http://localhost:6443 -k --key admin.key --cert admin.crt --cacert ca.crt

Another solution is call kubectl proxy service 

#curl http://localhost:8001 -k

K8S default list

1. /metrics

2. /healthz

3. /version

4. /logs

5. /api

1. /apis

RBAC (Role Based Access Control)

check the kube-apiserver configuration of auth

#kubectl describe pods kube-apiserver-master --namespace=kube-system

Find the key:  --authorization-mode

check namespace resource / no-namespace resource

#kubectl api-resources --namespaced=true

#kubectl api-resources --namespaced=false

Namespace role & rolebinding

Example role yaml file

https://github.com/tomshenhao/kubernetes-learning/blob/master/role.yaml

Example role binding yaml file

https://github.com/tomshenhao/kubernetes-learning/blob/master/rolebinding.yaml

check role #kubectl get roles

create role #kubectl create -f <yaml file>

check rolebinding #kubectl get rolebindings

check role detail #kubectl describe role <role name>

check rolebinding detail #kubectl describe rolebinding <role binding name>

User check own privilege #kubectl auth can-i <verb> <resources>

Example: #kubectl auth can-i create deployments

admin check specific user privilege #kubectl auth can-i <verb> <resource> --as <username>

Example: #kubectl auth can-i create deployments --as dev-user

Cluster roles  & rolebinding

1. cluster admin --> nodes operation

2. storage admin --> PV operation

check cluster role #kubectl get clusterrole

check cluster rolebinding #kubectl get clusterrolebinding

Example cluster role yaml file

https://github.com/tomshenhao/kubernetes-learning/blob/master/cluster-role.yaml

Example cluster rolebinding yaml file

https://github.com/tomshenhao/kubernetes-learning/blob/master/cluster-role-binding.yaml

Image security

image: nginx --> image: docker.io/nginx/nginx

registry: docker.io

user/account: nginx

image/repository: nginx

k8s image:

image: gcr.io/kubernetes-e2e-test-image/dnsutils

yaml file use the image

1. create the secret

#kubectl create secret docker-registry <secret name> --docker -server=<server link> --docker-username=<username> --docker-password=<password> --docker-email=<email address>

2. put in yaml file

apiVersion: v1

kind: Pod

metadata:

  name: nginx-pod

spec:

  containers:

  - name: nginx

    image: private-registry.io/apps/internal-apps/internal-a

  imagePullSecrets:

  - name: regcred

Context security

Add following to spec part or container part

  securityContext:

    runAsUser: 1010

  

Network Security

By default, All allow between pods

Example: db add policy: add ingress traffic from api pod on port 3306

Solutions support network policy: kube-router / calico / romana / weave-net

Solution not support network policy: flannel

https://github.com/tomshenhao/kubernetes-learning/blob/master/networkpolicy.yaml