Note_Tech

All technological notes.

Project maintained by simonangel-fong Hosted on GitHub Pages — Theme by mattgraham

Kubernetes: Container - Logging

Kubernetes: Container - Logging

Metric

Imperative Command

CMD	DESC
`kubectl top node`	Display resource (CPU/memory) usage of nodes
`kubectl top pod`	Display resource (CPU/memory) usage of pods
`kubectl top pod POD_NAME --containers`	Show metrics for a given pod and its containers
`kubectl top pod POD_NAME --show-swap`	Show pod resources related to swap memory
`kubectl top pod POD_NAME --sort-by=cpu`	sort pods list using specified field

Metrics Server

In-memory monitoring solution
- not store metrics on the disk
kubelet
- kubelet has a subcomponent cadvisor, the Container Advisor
cadvisor
- a subcomponent of the kubelet on each pod used to retrieving performance metrics from pods and exposing through the Kubelet API to make the metrics avaialable for the metrics server.

# view the logs of a pod with single container
kubectl logs -f pod_name

kubectl logs -f pod_name | grep WARNING

# view the logs of a pod with multiple containers
kubectl logs -f pod_name con_name

Lab: Enable Metric Server

# deploy metric server: minikube
# minikube addons enable metrics-server
# 💡  metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
# You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
#     ▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.8.0
# 🌟  The 'metrics-server' addon is enabled

# confirm
kubectl get pod -A
# NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE
# kube-system   metrics-server-85b7d694d7-fjbwr    1/1     Running   0               98s

# view node metric
kubectl top node
# NAME       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
# minikube   211m         5%       1394Mi          17%

kubectl top pod
# NAME               CPU(cores)   MEMORY(bytes)
# busybox-minikube   0m           3Mi

Event objects

events
- represented by Event objects that are created and read via the Kubernetes API.
- deleted one hour after its creation to reduce the burden on etcd, the data store for Kubernetes API objects.
Event objects have no spec and status sections
Two types of events exist:
- Normal
- Warning:
  - generated by controllers when something prevents them from reconciling the object.
How it works:
- controller manages the object
  - read spec and update status
- controller generates Event objects, showing actions and state changes
- events are store in etcd
  - automatically purged one hour after created.

Common Commands

CMD	DESC
`kubectl describe OBJECT`	Show details, including events related to this object
`kubectl get events`/`kubectl get ev`	list recent events
`kubectl get events -o wide`	list recent events with additional column
`kubectl get ev --field-selector type=Warning`	display only Warning events

Lab: Get events object

kubectl create deploy warning --image=warning --replicas=3
# deployment.apps/warning created

kubectl get ev
# Warning: short name "ev" could also match lower priority resource events.events.k8s.io
# LAST SEEN   TYPE      REASON              OBJECT                         MESSAGE
# 8m13s       Normal    Killing             pod/nginx-7ccccd94f7-jtbsr     Stopping container nginx
# 8m13s       Normal    SuccessfulDelete    replicaset/nginx-7ccccd94f7    Deleted pod: nginx-7ccccd94f7-jtbsr
# 8m13s       Normal    ScalingReplicaSet   deployment/nginx               Scaled down replica set nginx-7ccccd94f7 from 3 to 2
# 22s         Normal    Scheduled           pod/warning-d874477f8-6qn9h    Successfully assigned default/warning-d874477f8-6qn9h to docker-desktop
# 4s          Normal    Pulling             pod/warning-d874477f8-6qn9h    Pulling image "warning"
# 4s          Warning   Failed              pod/warning-d874477f8-6qn9h    Failed to pull image "warning": Error response from daemon: pull access denied for warning, repository does not exist or may require 'docker login'
# 4s          Warning   Failed              pod/warning-d874477f8-6qn9h    Error: ErrImagePull
# 19s         Normal    BackOff             pod/warning-d874477f8-6qn9h    Back-off pulling image "warning"
# 19s         Warning   Failed              pod/warning-d874477f8-6qn9h    Error: ImagePullBackOff
# 22s         Normal    Scheduled           pod/warning-d874477f8-994bp    Successfully assigned default/warning-d874477f8-994bp to docker-desktop
# 7s          Normal    Pulling             pod/warning-d874477f8-994bp    Pulling image "warning"
# 7s          Warning   Failed              pod/warning-d874477f8-994bp    Failed to pull image "warning": Error response from daemon: pull access denied for warning, repository does not exist or may require 'docker login'
# 7s          Warning   Failed              pod/warning-d874477f8-994bp    Error: ErrImagePull
# 19s         Normal    BackOff             pod/warning-d874477f8-994bp    Back-off pulling image "warning"
# 19s         Warning   Failed              pod/warning-d874477f8-994bp    Error: ImagePullBackOff
# 22s         Normal    Scheduled           pod/warning-d874477f8-ql7d9    Successfully assigned default/warning-d874477f8-ql7d9 to docker-desktop
# 6s          Normal    Pulling             pod/warning-d874477f8-ql7d9    Pulling image "warning"
# 6s          Warning   Failed              pod/warning-d874477f8-ql7d9    Failed to pull image "warning": Error response from daemon: pull access denied for warning, repository does not exist or may require 'docker login'
# 6s          Warning   Failed              pod/warning-d874477f8-ql7d9    Error: ErrImagePull
# 19s         Normal    BackOff             pod/warning-d874477f8-ql7d9    Back-off pulling image "warning"
# 19s         Warning   Failed              pod/warning-d874477f8-ql7d9    Error: ImagePullBackOff
# 22s         Normal    SuccessfulCreate    replicaset/warning-d874477f8   Created pod: warning-d874477f8-994bp
# 22s         Normal    SuccessfulCreate    replicaset/warning-d874477f8   Created pod: warning-d874477f8-ql7d9
# 22s         Normal    SuccessfulCreate    replicaset/warning-d874477f8   Created pod: warning-d874477f8-6qn9h
# 22s         Normal    ScalingReplicaSet   deployment/warning             Scaled up replica set warning-d874477f8 from 0 to 3


# event object info for type of event
kubectl explain events
  # type  <string>
  #   Type of this event (Normal, Warning), new types could be added in the future

# filter only warning event
kubectl get ev --field-selector type=Warning
# Warning: short name "ev" could also match lower priority resource events.events.k8s.io
# LAST SEEN   TYPE      REASON   OBJECT                        MESSAGE
# 5s          Warning   Failed   pod/warning-d874477f8-6qn9h   Failed to pull image "warning": Error response from daemon: pull access denied for warning, repository does not exist or may require 'docker login'
# 5s          Warning   Failed   pod/warning-d874477f8-6qn9h   Error: ErrImagePull
# 21s         Warning   Failed   pod/warning-d874477f8-6qn9h   Error: ImagePullBackOff
# 49s         Warning   Failed   pod/warning-d874477f8-994bp   Failed to pull image "warning": Error response from daemon: pull access denied for warning, repository does not exist or may require 'docker login'
# 49s         Warning   Failed   pod/warning-d874477f8-994bp   Error: ErrImagePull
# 9s          Warning   Failed   pod/warning-d874477f8-994bp   Error: ImagePullBackOff
# 46s         Warning   Failed   pod/warning-d874477f8-ql7d9   Failed to pull image "warning": Error response from daemon: pull access denied for warning, repository does not exist or may require 'docker login'
# 46s         Warning   Failed   pod/warning-d874477f8-ql7d9   Error: ErrImagePull
# 12s         Warning   Failed   pod/warning-d874477f8-ql7d9   Error: ImagePullBackOff

Event object cannot be delete using kubectl delete all -all

kubectl get event
# LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
# 27m         Normal   Scheduled   pod/init-containers-demo   Successfully assigned default/init-containers-demo to docker-desktop
# 27m         Normal   Pulling     pod/init-containers-demo   Pulling image "busybox"
# 27m         Normal   Pulled      pod/init-containers-demo   Successfully pulled image "busybox" in 1.018s (1.018s including waiting). Image size: 2224358 bytes.
# ...

kubectl  delete all --all

kubectl get event
# LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
# 27m         Normal   Scheduled   pod/init-containers-demo   Successfully assigned default/init-containers-demo to docker-desktop
# 27m         Normal   Pulling     pod/init-containers-demo   Pulling image "busybox"
# 27m         Normal   Pulled      pod/init-containers-demo   Successfully pulled image "busybox" in 1.018s (1.018s including waiting). Image size: 2224358 bytes.
# ...

# delete all event
kubectl delete event --all
# event "init-containers-demo.18835cddd8eb5ed5" deleted from default namespace
# event "init-containers-demo.18835cddfc62a87b" deleted from default namespace
# event "init-containers-demo.18835cde391aa01d" deleted from default namespace
# event "init-containers-demo.18835cde4681a7a7" deleted from default namespace
# event "init-containers-demo.18835cde4cb35ff5" deleted from default namespace
# event "init-containers-demo.18835ce0b85eac47" deleted from default namespace
# ...

kubectl get event
# No resources found in default namespace.