You probably know what Kubernetes is if you're reading this. For those who don't, its an Open Source container orchestration platform that has sparked hundreds of related Open Source projects and made it possible for organizations small and large to build massive platforms with (relative) ease.
However, what Kubernetes presents in power, it equally presents in complexity. The vast majority of Kubernetes implementations I've seen stay skin deep as far as the totality of what is possible on Kubernetes - and with good reason!
For large organizations using Kubernetes that have the engineering power and desire to customize and optimize, Kubernetes has many lesser known features. In this first installment, we'll review 4 such features.
When extending Kubernetes with custom functionality, the Custom Resource Definition - or CRD for short - is first to mind. Many cloud native projects and products (like Istio) use CRDs in spades, and they have the benefit of being stored in etcd like all the normal Kubernetes resource definitions in a cluster.
However, there's another, lesser known way to extend the scope of resources that the Kubernetes API knows about - Aggregation.
Aggregation, or more specifically the Aggregation Layer, allows us to specify resource types and groups (just like with CRDs) that should be served by a resource completely seperate to the Kubernetes API Server. When requests come into the Kubernetes API that reference these specific resources, Kubernetes will make a request to the aggregated server (which could be running on Kubernetes or somewhere else entirely) and present the response.
Now this is a weird one. All Kubernetes Secrets are encrypted, right?
In fact, by default NO Secret resources on Kubernetes are encrypted. When configuring your Kubernetes cluster, an EncryptionConfiguration must be passed to the kube-apiserver. Some cluster bootstrapping tools will prompt you to do this (and some managed Kubernetes Services, like EKS will have this as a tickbox) - but assuming that Secrets on your cluster are encrypted is dangerous. Even Kubeadm does not do this by default.
Pod Affinities and Anti Affinities on Kubernetes can get really complex. In fact, this even presents a scaling problem where the sheer heft required to compute some of the more complex affinity and anti-affinity combinations can significantly slow down the process of scheduling Pods.
Generally, Pod Affinities and Anti Affinities allow us to provide hard and soft rules to Kubernetes about how to schedule Pods. These rules are very different from Taints and Tolerations, Node Selectors, or even Node Affinities in that they operate not based on the content or labels on the Pods themselves as compared to the Node, but based on the Pod as compared to other Pods on a Node.
Put simply, Pod Affinities let us tell the scheduler to attempt to schedule a Pod on a Node or set of Nodes where other Pods of a certain type also live.
Pod Anti Affinities do the opposite - they let us dictate that the scheduler should attempt to schedule Pods on areas of our cluster where Pods of a certain type do NOT exist.