15.深入k8s:Event事件处理及其源码分析

74623200_p0_master1200

转载请声明出处哦~,本篇文章发布于luozhiyun的博客:https://www.luozhiyun.com

源码版本是1.19

概述

k8s的Event事件是一种资源对象,用于展示集群内发生的情况,k8s系统中的各个组件会将运行时发生的各种事件上报给apiserver 。可以通过kubectl get event 或 kubectl describe pod podName 命令显示事件,查看k8s集群中发生了哪些事件。

apiserver 会将Event事件存在etcd集群中,为避免磁盘空间被填满,故强制执行保留策略:在最后一次的事件发生后,删除1小时之前发生的事件。

如:

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 19s default-scheduler Successfully assigned default/hpatest-bbb44c476-8d45v to 192.168.13.130 Normal Pulled 15s kubelet, 192.168.13.130 Container image "nginx" already present on machine Normal Created 15s kubelet, 192.168.13.130 Created container hpatest Normal Started 13s kubelet, 192.168.13.130 Started container hpatest

当集群中的 node 或 pod 异常时,大部分用户会使用 kubectl 查看对应的 events,我们通过前面章节的代码分析可以看到这样的代码:

recorder.Eventf(cj, v1.EventTypeWarning, "FailedNeedsStart", "Cannot determine if job needs to be started: %v", err)

通过查找也可以确认基本上与node 或 pod相关的模块都会涉及到事件,如:controller-manage、kube-proxy、kube-scheduler、kubelet 等。

Event事件管理机制主要有三部分组成:

EventRecorder:是事件生成者,k8s组件通过调用它的方法来生成事件;

EventBroadcaster:事件广播器,负责消费EventRecorder产生的事件,然后分发给broadcasterWatcher;

broadcasterWatcher:用于定义事件的处理方式,如上报apiserver;

整个事件管理机制的流程大致如图:

image-20201011221745830

下面我们以kubelet 中的Event事件来作为分析的例子进行讲解。

源码分析

kubelet 在初始化的时候会调用makeEventRecorder进行Event初始化。

makeEventRecorder

文件位置:cmd/kubelet/app/server.go

func makeEventRecorder(kubeDeps *kubelet.Dependencies, nodeName types.NodeName) { if kubeDeps.Recorder != nil { return } // 初始化 EventBroadcaster eventBroadcaster := record.NewBroadcaster() // 初始化 EventRecorder kubeDeps.Recorder = eventBroadcaster.NewRecorder(legacyscheme.Scheme, v1.EventSource{Component: componentKubelet, Host: string(nodeName)}) //记录Event到log eventBroadcaster.StartStructuredLogging(3) if kubeDeps.EventClient != nil { klog.V(4).Infof("Sending events to api server.") //上报Event到apiserver并存储至etcd集群 eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: kubeDeps.EventClient.Events("")}) } else { klog.Warning("No api server defined - no events will be sent to API server.") } }

这个方法创建了一个EventBroadcaster,这是一个事件广播器,会消费EventRecorder记录的事件并通过StartStructuredLogging和StartRecordingToSink分别将event发送给log和apiserver;EventRecorder,用作事件记录器,k8s系统组件通过它记录关键性事件;

EventRecorder记录事件 type EventRecorder interface { Event(object runtime.Object, eventtype, reason, message string) Eventf(object runtime.Object, eventtype, reason, messageFmt string, args ...interface{}) AnnotatedEventf(object runtime.Object, annotations map[string]string, eventtype, reason, messageFmt string, args ...interface{}) }

EventRecorder接口非常的简单,就3个方法。其中Event是可以用来记录刚发生的事件;Eventf通过使用fmt.Sprintf格式化输出事件的格式;AnnotatedEventf功能和Eventf一致,但是附加了注释字段。

我们记录事件的时候上面也提到了,一般如下记录:

recorder.Eventf(cj, v1.EventTypeWarning, "FailedNeedsStart", "Cannot determine if job needs to be started: %v", err)

Eventf会调用到EventRecorder的实现类recorderImpl中去,最后调用到generateEvent方法中:

Event

文件位置:client-go/tools/record/event.go

func (recorder *recorderImpl) Event(object runtime.Object, eventtype, reason, message string) { recorder.generateEvent(object, nil, metav1.Now(), eventtype, reason, message) } func (recorder *recorderImpl) Eventf(object runtime.Object, eventtype, reason, messageFmt string, args ...interface{}) { recorder.Event(object, eventtype, reason, fmt.Sprintf(messageFmt, args...)) }

generateEvent

func (recorder *recorderImpl) generateEvent(object runtime.Object, annotations map[string]string, timestamp metav1.Time, eventtype, reason, message string) { ... //实例化Event event := recorder.makeEvent(ref, annotations, eventtype, reason, message) event.Source = recorder.source //异步调用Action方法将事件写入到incoming中 go func() { // NOTE: events should be a non-blocking operation defer utilruntime.HandleCrash() recorder.Action(watch.Added, event) }() }

generateEvent方法会异步的调用Action方法,将事件写入到incoming中:

func (m *Broadcaster) Action(action EventType, obj runtime.Object) { m.incoming <- Event{action, obj} }

调用步骤如下:

image-20201011170747803

EventBroadcaster事件广播

EventBroadcaster初始化的时候会调用NewBroadcaster方法:

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/zypzwy.html