Data flow monitoring visualization 1 tree structure

The data flow here refers to all components or services in the process of data flow from the front end to the back end. For example, the user's http request goes first to Nginx, then to back-end service 1, then to discovery service, then to cache service, and then to back-end service 2, Then there is the database, and other calls. Generally speaking, it's a path to request access. If such a process is visualized, I think it's good, and such a visualization can also be made into a monitoring visualization, to monitor whether the connection of the test can be completed, and to find that there is a problem with that component.

The front-end and back-end here are not limited to the front-end and back-end in development. Only the place where there is data flow is data flow, such as the network flow between different physical machines and switches, routers, or the data flow between containers. Generally speaking, there is always a flow method for all data. If you can obtain the signs of each node through certain technologies, then This path can be visualized dynamically.

Demonstration effect

Technical framework

JavaScript and SVG are selected here. The reason why SVG is selected is that D3 and its corresponding layout are available, so it is convenient to visualize the data in tree.

  • D3.js

data

Let's assume that the data flow architecture we want to monitor is as follows

                                            |---> backend11
           |---> nginx1 ---> backend1 --->  |
           |                                |---> backend12
client --> |
           |                                |---> backend21
           |---> nginx2 ---> backend2 --->  |
                                            |---> backend22

So we can express it in two ways

# Take each row as a node, and point to the corresponding parent node through the parent
rawData1 = [
    {name: "client", parent: "", status: 1},
    {name: "nginx1", parent: "client", status: 1},
    {name: "nginx2", parent: "client", status: 1},
    {name: "backend1", parent: "nginx1", status: 1},
    {name: "backend2", parent: "nginx2", status: 1},
    {name: "backend11", parent: "backend1", status: 1},
    {name: "backend12", parent: "backend1", status: 1},   
    {name: "backend21", parent: "backend2", status: 1},
    {name: "backend22", parent: "backend2", status: 1},   
]

# Represented by a nested json data structure
rawData2 = {data:{id: "client"},
    children: [
        {data: {id: "nginx1"},
         children: [
             {data: {id: "backend1"},
              children: [
                  {data: {id: "backend11"}},
                  {data: {id: "backend12"}}
              ]
            }
         ]
        },
        {data: {id: "nginx2"},
         children: [
             {data: {id: "backend2"},
              children: [
                  {data: {id: "backend21"}},
                  {data: {id: "backend22"}}
              ]
            }
         ]
        }
]};

Draw tree view

Drawing tree view through D3.js is mainly divided into two parts

  1. Convert the data into a form recognized by D3.js. Both of the above mentioned forms D3.js can be handled well.
  2. Draw all parts of the tree view, nodes, connections between nodes, annotation text of each node.

    Each part of the tree view can see that it needs to draw corresponding parts, such as not drawing node identification or text.

Processing data

Through the strategy method of d3, the first form of data can be processed into a data form that can be directly passed to the d3.tree object

var rawData = [
    {name: "client", parent: "", status: 1},
    {name: "nginx1", parent: "client", status: 1},
    {name: "nginx2", parent: "client", status: 1},
    {name: "backend1", parent: "nginx1", status: 1},
    {name: "backend2", parent: "nginx2", status: 1},
    {name: "backend11", parent: "backend1", status: 1},
    {name: "backend12", parent: "backend1", status: 1},   
    {name: "backend21", parent: "backend2", status: 1},
    {name: "backend22", parent: "backend2", status: 1},   
]

// stratify needs at least two fields to process the completed data: id, children
// Through the chain call id method, you can pass in a callback function to process each row of data, and the function should return this row of data as the value of id
// Through the chained call of parent id, you can pass in a callback function to process each row of data. The function should return this row of data and the parent id it points to
const data = d3.stratify()
                 .id(d => d.name)
                 .parentId(d => d.parent)
                 (rawData);

Through the hierarchy method of d3, the second form of data is processed into a data form that can be directly passed to the d3.tree object

var rawData = {data:{id: "client"},
                    children: [
                        {data: {id: "nginx1"},
                         children: [
                             {data: {id: "backend1"},
                              children: [
                                  {data: {id: "backend11"}},
                                  {data: {id: "backend12"}}
                              ]
                            }
                         ]
                        },
                        {data: {id: "nginx2"},
                         children: [
                             {data: {id: "backend2"},
                              children: [
                                  {data: {id: "backend21"}},
                                  {data: {id: "backend22"}}
                              ]
                            }
                         ]
                        }
                    ]};

// Since the above data already has the corresponding ID and children fields, no additional processing is required, and it can be directly transferred to hierarchy
const data = d3.hierarchy(rawData)

Both hierarchy and stratify transform data into a nested object, whose structure is a tree.

The effect is as follows

Drawing parts of the tree view

Initialize the tree object. The data processed above can be directly transferred to the initialized d3.tree object. After passing the data call, you will get a tree object (root in the code). This object has two important attributes, links and descendants

They correspond to the path of each side of the tree structure and the location of the nodes.

const myTree = d3.tree().size([innerWidth - 400, innerHeight]);
const root = myTree(data);
const links = root.links();
const nodes = root.descendants();

The reason for subtracting 400 from innerWidth is that the location calculation method of the built-in tree will calculate the scale of the graph according to the data structure, rather than the size of the canvas. In general, if you need to put the visualized graph of the tree in a proper position, the width and height need to be adjusted according to the data structure, so you can learn more.

Draw each edge of the tree

// Drawing links
svg.selectAll("path").data(links)
    .enter().append("path")
        .attr("d", d3.linkHorizontal()
            .x(d => d.y)
            .y(d => d.x)
        )

Draw each node of the tree and text

// Draw nodes
const circles = g.selectAll("circle").data(nodes, d=>d.data.id)
const circleFill = d => d.data.status ? successColor:failedColor

circles
    .enter().append("circle")
        .attr("r", 0)
        // Attention! x is wonderful for y
        .attr("cx", d => d.y)
        .attr("cy", d => d.x)
        .attr("fill", circleFill)

// Draw text
g.selectAll("text").data(nodes)
    .enter().append("text")
        .attr("x", d => d.y)
        .attr("y", d => d.x)
        // If the node has children field, it means that there is a child node, and the position in x direction does not change
        // Instead, the position moves 10 pixels to the right
        .attr("dx", d => d.children? 0: 10)
        // If the node has children field, it means there are child nodes, then the position in y direction moves up 10 pixels
        // Instead, the position moves down 5 pixels
        .attr("dy", d => d.children? -10: 5)
        // If the node has no parent, it means that it is the root node and the text method is end
        // If the node has children field, it means there is a child node, and the way of text is middle
        // On the contrary, the way of text is middle
        .attr("text-anchor", d => {
            if (!d.parent) {
                return "end"
            } else if (d.children) {
                return "middle"
            } else {
                return "start"
            }
        })
        .attr("font-size",  "1em")
        .text(d=> d.data.name)

Let the sides of the tree move

The animation effect is implemented in two parts: one is to add a path to the original path, and the other is to make the path move through css.

Draw flowing lines

// Draw flowing lines
g.selectAll("path.flow").data(links)
    .enter().append("path")
        // Attention! x is wonderful for y
        // Convert the data in links into the drawing path of the corresponding path in the graph through linkHorizontal
        .attr("d", d3.linkHorizontal()
            .x(d => d.y)
            .y(d => d.x)
        )
        .attr("class", "flow")

Configure css Style

path.flow {
    fill:transparent;
    stroke:#6D9459;
    stroke-dasharray: 20 4;
    animation: flowpath 4s linear infinite;
}

@keyframes flowpath {
    from {
        stroke-dashoffset: 100;
    }

    to {
        stroke-dashoffset: 0;
    }
}

The final effect will animate the beginning of the article

Listening to mouse zooming and dragging events

The size of the graph can be zoomed by the middle mouse button, and the svg object can also be moved by the mouse.

const zoomG = svg.append("g");
const g = zoomG.append("g").attr("transform", `translate(${margin.left} ${margin.top})`);

      // Increase zoom drag  
      svg.call(d3.zoom().on("zoom", () => {
        zoomG.attr("transform", d3.event.transform)            
    }));

summary

Because the code part of the text cuts off or replaces the relevant variables as needed, it is a problem only to copy the code in the text. For complete code, please refer to the complete source code.

source code

https://github.com/youerning/blog/tree/master/dataflow-vis

If you look forward to the following articles, you can pay attention to my WeChat official account (ear notes), headline (and ears note), github.

Q&A

Q: How to get the path of data flow?

Here, two data flow monitoring are provided as reference:

  1. http request

The back-end can configure a unique ID for a specific request, which will be passed in each component, so each component can send data to its own monitoring center when it monitors the specific ID, and the monitoring center provides processed data for the front-end.

  1. Connectivity between networks

Each node needs to set up an Agent or Agent. The monitoring center sends ping or specific request to its directly connected Agent according to a specific time interval to check the network connectivity.

  1. What's the use of this?

If you think it's useful, you can use it. If you think it's useless, you can't use it.

  1. In most cases, the data structure between nodes is not a tree structure but a network structure. What should I do?

Let's talk about the non tree structure in the following article,

Tags: Programming network github Nginx Database

Posted on Sat, 21 Mar 2020 10:18:16 -0400 by suave4u