Analysis of KubeDiag framework technology

design goal

KubeDiag's design objectives include:

• Portability: it can run in a Linux standard environment where Kubernetes is deployed.
• Scalability: users can integrate customized diagnostic functions. The modules interact through loose coupling interface, and each functional module is pluggable.
• Automation: greatly reduce the labor cost of problem diagnosis. Users can define diagnostic workflows through declarative API s and run them automatically when problems occur.
• Ease of use: built in common problem diagnosis logic to provide out of the box experience.

architecture design

KubeDiag consists of Master and Agent, and obtains data from APIServer, Prometheus and other components.

KubeDiag Master design

KubeDiag Master is responsible for managing Operation, OperationSet, Trigger and Diagnosis objects. After the OperationSet is created, the KubeDiag Master will check the legitimacy and generate a directed acyclic graph based on the user definition, and all diagnostic paths will be updated to the metadata of the OperationSet. If the pre dependent diagnostic Operation of an Operation in the OperationSet does not exist, the OperationSet will be marked as an exception.

KubeDiag Master will verify whether the PodReference or NodeName of diagnosis exists. If only PodReference is defined in diagnosis, NodeName will be calculated and updated according to the PodReference. KubeDiag Master will query the status of the OperationSet referenced by diagnosis. If the referenced OperationSet is abnormal, it will mark diagnosis failed. Diagnosis can be created manually or automatically by configuring Prometheus alarm template or Kubernetes event template.

The KubeDiag Master consists of the following parts:

• Graph builder
• Prometheus alarm manager (Alertmanager)
• Kafka Message Manager (KafkaConsumer)
• Event manager

The diagram builder generates a diagnostic running flowchart based on the OperationSet object. The graph builder generates a directed acyclic graph according to the edges contained in the OperationSet and calculates all diagnostic paths.

The Prometheus alarm manager receives Prometheus alarms and creates a Diagnosis object. The Prometheus alarm manager can receive Prometheus alarms and match them with the templates defined in Trigger. If the matching is successful, a Diagnosis object will be created according to the metadata of Trigger.

The Kafka message manager receives Kafka messages and creates a Diagnosis object. The value of the Kafka message must be a JSON object and contain the meta information required to create the Diagnosis.

The event manager receives the Kubernetes event and creates a Diagnosis object. The event manager can receive the Kubernetes event and match it with the template defined in Trigger. If the match is successful, the Diagnosis object is created according to the metadata of Trigger.

KubeDiag Agent design

KubeDiag Agent is responsible for the execution of actual Diagnosis work and built-in multiple common diagnostic operations. After Diagnosis is created, KubeDiag Agent will execute the Diagnosis workflow according to the OperationSet referenced by Diagnosis. The Diagnosis workflow is a collection of multiple Diagnosis operations.

The KubeDiag Agent component consists of the following parts:

• Actuator

The actuator is responsible for executing the diagnostic workflow. The OperationSet metadata referenced by Diagnosis contains a directed acyclic graph representing the Diagnosis workflow and all Diagnosis paths. The Diagnosis path represents the troubleshooting path in the Diagnosis process. The problem can be troubleshooting by performing the Diagnosis operation of each vertex in a Diagnosis path. If all diagnostic operations of a diagnostic path are successful, the Diagnosis is marked as successful. If all diagnostic paths fail, the Diagnosis is marked as failed.

Implementation details

KubeDiag abstracts the process of an Operation and maintenance Diagnosis by implementing Operation, OperationSet, Trigger and Diagnosis custom resources.

Management of diagnostics

KubeDiag meets the needs of user management diagnostics by supporting the following functions:

• Specify the target Node or Pod for the diagnosis.
• View the phase of the current diagnosis.
• Extend the status of the diagnosis through parameters.
• When the diagnosis is successful, view the diagnosis results and troubleshooting path.
• When the diagnosis fails, view the reason for the failure and the troubleshooting path.
• View details of a stage in the diagnostic process.

The data structure of the Diagnosis object is as follows:

// Diagnosspec defines the target state of Diagnosis. type DiagnosisSpec struct { // OperationSet is the OperationSet name of the diagnostic pipeline to be executed. OperationSet string `json:"operationSet"` // Either NodeName or PodReference field must be specified. // NodeName is the node where the diagnosis is performed. NodeName string `json:"nodeName,omitempty"` // The PodReference contains the details of the target Pod. PodReference *PodReference `json:"podReference,omitempty"` // Parameters contains parameters that need to be passed in during diagnosis. // Generally, the key of this field is the sequence number of the vertex in the OperationSet, and the value is the parameter required to perform the vertex diagnosis operation. // Parameters and OperationResults are serialized as JSON objects and sent to the fault processor during running diagnosis. Parameters map[string]string `json:"parameters,omitempty"` } // The PodReference contains the details of the target Pod. type PodReference struct { NamespacedName `json:",inline"` // Container is the name of the destination container. Container string `json:"container,omitempty"` } // NamespacedName represents the Kubernetes API object. type NamespacedName struct { // Namespace is the Kubernetes API object namespace. Namespace string `json:"namespace"` // Namespace is the Kubernetes API object name. Name string `json:"name"` } // The diagnosistatus defines the actual status of the Diagnosis. type DiagnosisStatus struct { // Phase is a simple macro overview of where Diagnosis is in its lifecycle. The status list contains more information about the status of Diagnosis. // There may be five different values for phases: // // Pending: Diagnosis has been accepted by the system, but the preparation before Diagnosis execution has not been completed. // Running: Diagnosis has been bound to a node, and at least one diagnostic operation is running. // Succeeded: all diagnostic operations in a path in the diagnostic pipeline are executed successfully. // Failed: failed to diagnose all paths in the pipeline. That is, the return code of the last diagnostic operation performed in all paths is not 200. // Unknown: the status of Diagnosis cannot be obtained for some reason. This situation is usually caused by communication failure with the host where Diagnosis is located. Phase DiagnosisPhase `json:"phase,omitempty"` // Conditions contain the current service status of Diagnosis. Conditions []DiagnosisCondition `json:"conditions,omitempty"` // StartTime is the date and time when the object received RFC 3339 by the system. StartTime metav1.Time `json:"startTime,omitempty"` // FailedPaths contains paths that diagnose all run failures in the pipeline. The last vertex of the path is the vertex where the operation failed. FailedPaths []Path `json:"failedPath,omitempty"` // SucceededPath is the path that runs successfully in the diagnostic pipeline. SucceededPath Path `json:"succeededPath,omitempty"` // OperationResults contains the results of operations during the diagnostic run. // Parameters and OperationResults are serialized as JSON objects and sent to the fault processor during running diagnosis. OperationResults map[string]string `json:"operationResults,omitempty"` // Checkpoint is a checkpoint to recover incomplete diagnostics. Checkpoint *Checkpoint `json:"checkpoint,omitempty"` } // DiagnosisCondition contains the current service status of Diagnosis. type DiagnosisCondition struct { // Type is the type of condition. Type DiagnosisConditionType `json:"type"` // Status is the status of the status. // Can be True, False, Unknown. Status corev1.ConditionStatus `json:"status"` // LastTransitionTime describes the last time you migrated from one situation to another. LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"` // Reason is the description of the last condition transfer, which is unique, contains only a single word and conforms to the hump nomenclature. Reason string `json:"reason,omitempty"` // Message is the information describing the details of the last status migration. Message string `json:"message,omitempty"` } // Checkpoint is a checkpoint to recover incomplete diagnostics. type Checkpoint struct { // PathIndex is the sequence number of the current path in the OperationSet state. PathIndex int `json:"pathIndex"` // NodeIndex is the sequence number of the current vertex in the path. NodeIndex int `json:"nodeIndex"` } // DiagnosisPhase is a label that describes the current Diagnosis status. type DiagnosisPhase string // DiagnosisConditionType is the legal value of the Diagnosis condition type. type DiagnosisConditionType string // API object for Diagnosis. type Diagnosis struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec DiagnosisSpec `json:"spec,omitempty"` Status DiagnosisStatus `json:"status,omitempty"` }

Diagnosis is actually a stateful task. Its state may migrate many times in the life cycle of diagnosis. The ability to manage diagnosis state migration is essential in many scenarios.

An operation may depend on input in a specific format. The. spec.parameters field in Diagnosis is used to specify the parameters to be passed in during Diagnosis. This field is a key value pair. Both keys and values must be of String type. When the operation to be performed depends on the input of a specific format, the user can define the parameters to be input during operation execution in this field, and the operation processor will execute the diagnostic operation after obtaining the parameters.

An operation may depend on the output of a previous operation. The. status.operationResults field in Diagnosis is used to record the results of operations during the Diagnosis run. This field is a key value pair. Both keys and values must be of String type. The execution result of the current operation must be returned in the form of JSON object, and the returned result will be updated to this field. If the execution of subsequent operations depends on the output of the current operation, the subsequent operation processor can obtain the result of the current operation from. status.operationResults. It should be noted that if two identical operations update the same key in the troubleshooting path, the result of the later operation will overwrite the result of the first operation.

Users need to analyze and optimize the results of an Operation in the troubleshooting path. The. status.failedPath field and. status.succeededPath field in Diagnosis record all failed paths and successful paths respectively. Each path is represented by an array, and the elements of the array contain the sequence number and Operation name of the vertex. The order of Operation execution can be restored by traversing the path, and the access information of each Operation result is recorded in the. spec.storage field in the Operation.

Diagnosis contains the. status.phase field, which is a simple macro overview of where diagnosis is in its lifecycle. This stage is not a comprehensive summary of the diagnosis state, nor is it intended to become a complete state machine. The number and meaning of the diagnosis phase are strictly defined. It should not be assumed that diagnosis has other phase values other than those listed in this document.

The following are the possible values of. status.phase:

• Pending: Diagnosis has been accepted by the system, but the preparation before Diagnosis execution has not been completed.
• Running: Diagnosis has been bound to a node, and at least one diagnostic operation is running.
• Succeeded: all diagnostic operations in a path in the diagnostic pipeline are executed successfully.
• Failed: failed to diagnose all paths in the pipeline. That is, the return code of the last diagnostic operation performed in all paths is not 200.
• Unknown: the status of Diagnosis cannot be obtained for some reason. This situation is usually caused by communication failure with the host where Diagnosis is located.

Graph based diagnostic pipeline

When designing the graph based diagnosis pipeline, we mainly consider the following assumptions:

• Finite termination: the whole pipeline is not executed indefinitely and can terminate within a certain time and space complexity.
• Process traceability: after diagnosis, you can view the results of a vertex during operation.
• State machine extensible: support adding new processing vertices to the pipeline.

KubeDiag implements a graph based diagnostic pipeline by introducing the following API objects:

• Operation: describes how to add a processing vertex to the diagnostic pipeline and how to store the results generated by the processing vertex.
• OperationSet: represents the directed acyclic graph of the diagnostic process state machine.
• Trigger: describes how to trigger a diagnosis through a Prometheus alarm or Kubernetes event.

The data structure of the Operation object is as follows:

// OperationSpec defines the target state of Operation. type OperationSpec struct { // Processor describes how to register an operation processor in KubeDiag. Processor Processor `json:"processor"` // Dependencies is a list of all dependent diagnostic actions that must be performed in advance. Dependences []string `json:"dependences,omitempty"` // Storage indicates the storage type of the operation processing result. // If this field is empty, the operation processing result will not be saved. Storage *Storage `json:"storage,omitempty"` } // Processor describes how to register an operation processor in KubeDiag. type Processor struct { // ExternalAddress is the listening address of the operation processor. // If this field is empty, it defaults to the address of KubeDiag Agent. ExternalAddress *string `json:"externalAddress,omitempty"` // ExternalPort is the service port of the operation processor. // If this field is empty, it defaults to the service port of KubeDiag Agent. ExternalPort *int32 `json:"externalPort,omitempty"` // Path is the HTTP path of the operation processor service. Path *string `json:"path,omitempty"` // Scheme is the protocol that operates the processor service. Scheme *string `json:"scheme,omitempty"` // The number of seconds the operation processor timed out. // The default is 30 seconds. The minimum value is 1. TimeoutSeconds *int32 `json:"timeoutSeconds,omitempty"` } // Storage indicates the storage type of the operation result. type Storage struct { // HostPath represents the directory on the host. HostPath *HostPath `json:"hostPath,omitempty"` } // HostPath represents the directory on the host. type HostPath struct { // The path to the directory on the host. // If this field is empty, it defaults to the data root directory of KubeDiag Agent. Path string `json:"path"` } // API object for Operation. type Operation struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec OperationSpec `json:"spec,omitempty"` }

The data structure of the OperationSet object is as follows:

// OperationSetSpec defines the target state of OperationSet. type OperationSetSpec struct { // Adjacency list contains all vertices representing diagnostic operations in a directed acyclic graph. // The first vertex of the array represents the beginning of the diagnosis rather than a specific diagnostic operation. AdjacencyList []Node `json:"adjacencyList"` } // Node is a vertex in a directed acyclic graph. It contains the sequence number and the operation name. type Node struct { // ID is the unique identifier of the vertex. ID int `json:"id"` // To is a list of vertex sequence numbers that can be reached directly from the vertex. To NodeSet `json:"to,omitempty"` // Operation is the name of the operation that runs at the vertex. Operation string `json:"operation"` // Dependencies is a list of all dependent diagnostic operation ID s that must be performed in advance. Dependences []int `json:"dependences,omitempty"` } // NodeSet is a set of vertex sequence numbers. type NodeSet []int // OperationSetStatus defines the actual status of the OperationSet. type OperationSetStatus struct { // Paths is a collection of all diagnostic paths in a directed acyclic graph. Paths []Path `json:"paths,omitempty"` // Indicates whether the vertices provided in the definition can generate a legal directed acyclic graph. Ready bool `json:"ready,omitempty"` } // Path represents the linear order of vertices consistent with all edge directions. type Path []Node // API object for OperationSet. type OperationSet struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec OperationSetSpec `json:"spec,omitempty"` Status OperationSetStatus `json:"status,omitempty"` }

The data structure of Trigger object is as follows:

// TriggerSpec defines the target status of the Trigger. type TriggerSpec struct { // OperationSet is the OperationSet name referenced in the build Diagnosis. OperationSet string `json:"operationSet"` // SourceTemplate is the template source used to generate Diagnosis. SourceTemplate SourceTemplate `json:"sourceTemplate"` } // SourceTemplate describes the information used to generate a Diagnosis. type SourceTemplate struct { // A unique template source must be specified in the following sources. // Prometheus alerttemplate declares a template for creating a Diagnosis based on Prometheus alarms. PrometheusAlertTemplate *PrometheusAlertTemplate `json:"prometheusAlertTemplate,omitempty"` // KubernetesEventTemplate declares a template for creating Diagnosis based on Event. KubernetesEventTemplate *KubernetesEventTemplate `json:"kubernetesEventTemplate,omitempty"` } // Prometheus alerttemplate declares a template for creating a Diagnosis based on Prometheus alarms. type PrometheusAlertTemplate struct { // Regexp is a regular expression used to match the Prometheus alarm template. Regexp PrometheusAlertTemplateRegexp `json:"regexp"` // NodeNameReferenceLabel specifies the label key used to set the ". spec.nodeName" field of Diagnosis. NodeNameReferenceLabel model.LabelName `json:"nodeNameReferenceLabel"` // PodNamespaceReferenceLabel specifies the label key used to set the ". spec.podReference.namespace" field of Diagnosis. PodNamespaceReferenceLabel model.LabelName `json:"podNamespaceReferenceLabel,omitempty"` // PodNameReferenceLabel specifies the label key used to set the ". spec.podReference.name" field of Diagnosis. PodNameReferenceLabel model.LabelName `json:"podNameReferenceLabel,omitempty"` // ContainerReferenceLabel specifies the label key used to set the ". spec.podReference.container" field of Diagnosis. ContainerReferenceLabel model.LabelName `json:"containerReferenceLabel,omitempty"` // ParameterInjectionLabels specifies the list of label keys that need to be injected into the ". spec.parameters" field. ParameterInjectionLabels []model.LabelName `json:"parameterInjectionLabels,omitempty"` } // Prometheus alerttemplateregexp is a regular expression used to match Prometheus alarm templates. // All regular expressions must follow the RE2 specification. For details, please refer to https://golang.org/s/re2syntax . type PrometheusAlertTemplateRegexp struct { // AlertName is a regular expression used to match the AlertName field of the Prometheus alarm. AlertName string `json:"alertName,omitempty"` // Labels is a regular expression used to match the labels field of the Prometheus alarm. // A successful match is only possible if the value of the tag is a regular expression and all tags are consistent with the Prometheus alarm. Labels model.LabelSet `json:"labels,omitempty"` // Annotations is a regular expression used to match the annotations field of the Prometheus alarm. // Only when the value of the annotation is a regular expression and all annotations are consistent with the Prometheus alarm can it be successfully matched. Annotations model.LabelSet `json:"annotations,omitempty"` // StartsAt is a regular expression used to match the StartsAt field of the Prometheus alarm. StartsAt string `json:"startsAt,omitempty"` // EndsAt is a regular expression used to match the EndsAt field of the Prometheus alarm. EndsAt string `json:"endsAt,omitempty"` // GeneratorURL is a regular expression used to match the GeneratorURL field of the Prometheus alarm. GeneratorURL string `json:"generatorURL,omitempty"` } // KubernetesEventTemplate declares a template for creating Diagnosis based on Event. type KubernetesEventTemplate struct { // Regexp is a regular expression used to match the Event template. Regexp KubernetesEventTemplateRegexp `json:"regexp"` } // KubernetesEventTemplateRegexp is a regular expression used to match Event templates. // All regular expressions must follow the RE2 specification. For details, please refer to https://golang.org/s/re2syntax . type KubernetesEventTemplateRegexp struct { // Name is the regular expression used to match the name field of the Event. Name string `json:"name,omitempty"` // Namespace is a regular expression used to match the namespace field of an Event. Namespace string `json:"namespace,omitempty"` // Reason is a regular expression used to match the reason field of an Event. Reason string `json:"reason,omitempty"` // Message is a regular expression used to match the message field of an Event. Message string `json:"message,omitempty"` // Source is a regular expression used to match the source field of an Event. // All fields in the Source are regular expressions. Source corev1.EventSource `json:"source,omitempty"` } // Trigger API object. type Trigger struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec TriggerSpec `json:"spec,omitempty"` }

Diagnostic Operation refers to a logic running in the diagnostic pipeline. It is the smallest unit for the management of the diagnostic pipeline, such as obtaining node information, matching keywords in the log, performance analysis of the process, etc. You can register diagnostic operations by creating an Operation object. The back end of the diagnostic Operation is an HTTP server. When registering diagnostic operations, you need to specify the address, path, storage type of diagnostic results, etc. monitored by the HTTP server.

The diagnosis pipeline is a collection of multiple diagnosis operations. Usually, a diagnosis may have multiple troubleshooting paths, so the state machine of a diagnosis is abstracted through a directed acyclic graph. By creating an OperationSet object, you can define a directed acyclic graph representing the diagnostic state machine. The start state of diagnosis is the starting point of the directed acyclic graph. The paths in the directed acyclic graph are the troubleshooting paths in the diagnosis process. When a path can run successfully to the end point, it means that the diagnosis runs successfully. The generation logic of the diagnosis pipeline is as follows:

1. The user creates an OperationSet resource and defines all edges in the directed acyclic graph.
2. Construct a directed acyclic graph according to the definition of OperationSet.
3. If a legal directed acyclic graph cannot be built, the registration failure status and failure reason are updated to the OperationSet.
4. Enumerate all diagnostic paths in the OperationSet and update them to the OperationSet.

The directed acyclic graph representing the diagnosis pipeline must contain only one Source Node, which is used to represent the start state of diagnosis and does not contain any diagnosis operations. The diagnostic path is any path from the source vertex to any Sink Node. The diagnosis path does not include source vertices that represent the start state of the diagnosis. The graph builder searches all diagnostic paths in the directed acyclic graph and updates them to the. status.paths field of the OperationSet.

The metadata of the diagnosis object contains the OperationSet to be executed. Trigger diagnosis includes manual and automatic modes. Diagnosis can be triggered directly by manually creating a diagnosis object. By creating a trigger object and configuring the Prometheus alarm template or Event template, you can automatically generate a diagnosis based on Prometheus or Event to trigger the diagnosis pipeline.

The running status of the Diagnosis pipeline is recorded in the metadata of the Diagnosis object. The operation logic of the Diagnosis pipeline is as follows:

1. Gets all diagnostic paths in the OperationSet referenced by Diagnosis.
2. According to the diagnostic operations defined by Operation in the diagnostic execution path, update the results of Operation operation to Diagnosis and persist them to the corresponding storage type in Operation.
3. If a diagnostic operation defined in the path fails, the next diagnostic path is executed.
4. If all diagnostic operations defined in the path are successful, the diagnosis is successful.
5. If all paths fail, the diagnosis fails.

The above is a directed acyclic graph representing the diagnosis pipeline. Each node represents an Operation. The diagnosis path of the graph represents multiple executable troubleshooting paths:

• Data collection 1, data analysis 1, recovery 1
• Data collection 1, data analysis 1, recovery 2
• Data collection 2, data analysis 2, recovery 2
• Data collection 3. Data collection 4

Typical use case

The following file defines an OperationSet for handling Docker problems:

apiVersion: diagnosis.kubediag.org/v1 kind: OperationSet metadata: name: docker-debugger spec: adjacencyList: - id: 0 to: - 1 - id: 1 operation: docker-info-collector to: - 2 - id: 2 operation: dockerd-goroutine-collector to: - 3 - id: 3 operation: containerd-goroutine-collector to: - 4 - id: 4 operation: node-cordon

The process of triggering Docker problem diagnosis through KubeletPlegDurationHigh alarm is as follows:

• The user successfully created an OperationSet to handle Docker problems.
• The user successfully created a Trigger based on KubeletPlegDurationHigh alarm Trigger diagnosis
• The KubeletPlegDurationHigh alarm is sent to the KubeDiag Master.
• KubeDiag Master creates a Diagnosis based on the information in the alarm.
• Diagnosis is performed on the node that triggered the alarm.
  • Collect Docker related information.
  • Collect the Goroutine of Dockerd.
  • Collect Goroutine from container.
  • Set the node triggering the alarm as non schedulable.
• End of diagnostic execution.

Welcome to the community

Finally, KubeDiag community is still in the initial stage. Welcome to join KubeDiag community to jointly improve the automation of Kubernetes, solve the use problems in the post container era, and promote the global application of cloud native technology.

KubeDiag project home page: https://kubediag.org/

KubeDiag project address: https://github.com/kubediag/kubediag

Scan code to join KubeDiag wechat group: https://kubediag.nos-eastchina1.126.net/QR%20Code.jpeg

Author: Huang Jiuyuan, Netease digital sail cloud native technology expert, KubeDiag Maintainer.

Add new comment

0 comments