k8s_cluster_communication

Home

Understanding Kubernetes Cluster Communication

This guide explores the communication pathways between key Kubernetes components, focusing on how the API server, etcd, and kubelet interact. We'll inspect configurations and use curl with appropriate credentials to query these components directly.

(Note: The configurations and specific IPs/hostnames shown are from a particular kubeadm setup. Your paths, IPs, and certificate details might vary. The Kubernetes versions (e.g., API server v1.18.3, etcd 3.4.3-0) are older; concepts are similar, but flags or API paths might differ in newer versions.)

1. API Server Communication

The Kubernetes API server is the central hub for all cluster operations. Control plane components and kubectl interact with it.

1.1 Inspecting API Server Configuration (Master Node)

On a master node set up by kubeadm, the API server runs as a static pod defined by a manifest typically found in /etc/kubernetes/manifests/.

  1. Locate and view the API server manifest:

    cd /etc/kubernetes/manifests
sudo cat ./kube-apiserver.yaml

    Below is an example snippet of such a manifest (details will vary):

    apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.10.2.78:6443 # Example IP
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.10.2.78 # Example IP
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    # ... many other flags ...
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.18.3 # Example version
    # ... probes, volumeMounts, etc. ...
  hostNetwork: true
  # ... volumes ...
status: {}

    This manifest shows how the API server is configured, including network ports, certificate paths, and communication with etcd.

1.2 Querying API Server Endpoints from the Host

You can use curl to interact with the API server directly.

  1. Set your Master IP:

  2. Basic Unauthenticated Queries (Health Checks): Some endpoints like /healthz, /livez, /readyz might be accessible (or partially accessible) without full authentication for health checking. The -k or --insecure flag for curl is used here to bypass TLS certificate verification if you don't specify a CA (common for self-signed certs).

  3. Authenticated Query using Client Certificates: To access most API endpoints, you need to authenticate. One way is using client certificates that the API server trusts (e.g., those used by kubelet or an admin user). The paths to these certificates are often found in kubeconfig files or component configurations. This example uses the apiserver-kubelet-client certificates:

    This should return a JSON response listing available API paths, for example:

    (The extensive JSON output from the original doc is represented by this snippet.)

2. etcd Communication (Master Node)

etcd is the distributed key-value store used by Kubernetes for all cluster data. The API server is its primary client.

2.1 Inspecting etcd Configuration

Like the API server, etcd often runs as a static pod on master nodes in a kubeadm setup.

Example etcd.yaml snippet:

This shows etcd's configuration, including its client and peer ports, and certificate paths.

2.2 Interacting with etcd using etcdctl

etcdctl is the command-line client for etcd.

  1. Obtain etcdctl: If not already installed on the master node, you can copy it from the etcd container.

    (Added command to get etcd pod name. Used sudo kubectl for consistency. Changed chmod 700 to +x.)

  2. Set API Version: etcd v3 uses API version 3.

  3. Query etcd: Commands require specifying endpoints and credentials (certificates) which are defined in the etcd.yaml manifest.

    Check endpoint status:

    (Note: The original used server.crt and server.key for etcdctl. Often, peer.crt/key or specific client certs are used for etcdctl if different from server certs. For simplicity, if server.crt is also a client cert, it can work. Using peer.crt/key as an example here, assuming they might be more appropriate for client auth to etcd.)

    List all keys (use with caution on large clusters):

    Get specific keys (example paths, these will vary):

    (Consolidated duplicated get /registry/namespaces/default and generalized specific key examples.)

    Task: The original document suggests: "find a secret, decode the secret and find the json webtoken..." This involves getting a secret's data, which is base64 encoded within the JSON/protobuf value stored in etcd, then base64 decoding those fields.

3. Kubelet Communication (Worker Node Focus)

The kubelet runs on every node and is responsible for managing pods as instructed by the API server.

3.1 Inspecting Kubelet Configuration (Worker Node)

The kubelet's configuration tells it how to communicate with the API server.

This file is a kubeconfig file used by the kubelet. Example content:

3.2 Kubelet Authenticating to API Server

The kubelet uses its client certificate and key (specified in kubelet.conf) to authenticate to the API server. You can simulate this with curl:

(Corrected variable usage in URL)

3.3 Inspecting Kubelet Process and Certificates

View the running kubelet process:

Example output:

(Formatted as generic text block)

View kubelet client certificate files (often a symlink to the current active cert):

Example output:

(Formatted as generic text block)

3.4 Kubelet API (Read-Only Port and Secure Port)

The kubelet listens on ports for its own API, used by the API server to issue commands to it and retrieve status/logs.

Check listening ports on a node:

Example output snippet showing common Kubelet ports:

(Updated netstat and formatted as generic text block. Added port descriptions.)

Hint: Port 10250 is the main secure Kubelet API. Port 10255 (not shown in example) was historically an unauthenticated read-only port, now often disabled. Port 10248 is often a localhost-only health check port.

3.5 API Server Communicating with Kubelet

The API server communicates with the kubelet's secure port (10250) using client certificates (apiserver-kubelet-client.crt/key) to authenticate itself to the kubelet.

  1. Set Worker Node IP:

  2. Query Kubelet Endpoints (from Master or machine with API server certs): These commands simulate how the API server might query a kubelet.

    Get logs (example):

    (Added CA cert to curl command for Kubelet API, as Kubelet's serving cert needs to be verified.)

    List pods running on the node:

    Execute a command in a pod (example, requires specific pod/container names):(This endpoint is powerful and its direct use is less common than kubectl exec)

    (Commented out the /run example as it's complex, potentially dangerous, and often disabled or restricted by default on modern Kubelet configurations for security reasons. Kept it commented for informational purposes if the user wants to explore it, but it's not a typical interaction method.)

Home

PreviousingressNextkube-bench

Last updated