16_Netdata Embedded Configuration

Netdata Embedded Configuration

background

Netdata can indeed be called a very excellent project. At present, there are not many monitoring systems that can operate stably on embedded devices, and even fewer that really have strong commercial value. For example, Baota Qinglong also provides functions similar to netdata, but it is a little Pediatrics if it is really used as a monitoring means for operation and maintenance personnel, More like personal amateur development and use.

Netdata can be called the most perfect embedded monitoring scheme at present. Even so, it still has many defects in embedded devices, most of which come from the embedded devices themselves; Although netdata can run on embedded devices, it is more aimed at servers or even server clusters; In other words, netdata targets more server operation and maintenance personnel than embedded operation and maintenance personnel.

Although the requirements for embedded applications are not as harsh as the server, they are also much more complex than the client program. An excellent embedded monitoring program should run stably for 7 * 24 hours and reduce the system overhead (CPU, disk, network and memory) as much as possible. To do this, personalized customization of Netdata is an indispensable process.

This section will first explain the configuration rules of Netdata configuration file, then give a configuration practice, and finally give the effect diagram; One of the principles of embedded devices is to save as much as possible without unintentional overhead, but it can not lack necessary functions. In short, it is more fast and better to save, so how to configure Netdata needs to be realistic, and there is no absolute answer.

Netdata configuration rules

Netdata is a highly configurable monitoring system. Most of its configuration files exist in / etc/netdata, and most of the configuration files begin with *. conf. it is worth mentioning that netdata follows the principle that the Convention is better than the configuration. Therefore, if there is no configuration file, it will start the default configuration at compile time.

The main configurations of Netdata are apps+groups.conf, charts.d.conf and health_alarm_notify.conf and netdata.conf, of which the key is netdata.conf, which is the main configuration file of Netdata.

Netdata.conf is the main configuration file of netdata. By modifying it, nearly 90% of netdata can be personalized. Therefore, understanding how to configure netdata.conf is a very important and necessary skill. If you access netdata configuration through IP:Port/netdata.conf, you will find that there are many configuration items, and everyone is different, There are more than 4000 lines here, as shown in the following figure:

Modifying the configuration of Netdata depends on skills, not mussels; The configuration of Netdata adopts ini format, and its configuration principle is similar to the form of multi-level domain name. For example:

[plugins] -> [plugin:proc] -> [plugin:proc:diskspace] -> [plugin:proc:diskspace:/mnt/disk/disk1]

This is the multi-level domain name system. Blocking anywhere on the link will cut off the whole link. For example, if I disable [plugins], then [plugin:proc] is useless, so [plugin:proc:diskspace] is also invalid, let alone [plugin:proc:diskspace:/mnt/disk/disk1].

Generally speaking, you don't need to set it to such a deep level. Generally, it's like the above example. It's often enough to set it to [plugin:proc:diskspace].

netdata main configuration modification

For embedded devices, some main modifications are given here, that is, the first several levels of Netdata configuration:

# netdata configuration
# global netdata configuration
[global]
    run as user = root
    hostname = unknown
    update every = 10
    config directory = /var/netdata/etc/netdata
    stock config directory = /var/netdata/lib/netdata/conf.d
    log directory = /var/netdata/var/log/netdata
    web files directory = /var/netdata/share/netdata/web
    cache directory = /var/netdata/var/cache/netdata
    lib directory = /var/netdata/var/lib/netdata
    home directory = /var/netdata/var/lib/netdata
    lock directory = /var/netdata/var/lib/netdata/lock
    plugins directory = "/var/netdata/libexec/netdata/plugins.d" "/var/netdata/etc/netdata/custom-plugins.d"
    memory mode = dbengine
    page cache size = 4
    dbengine disk space = 128
    timezone = Asia/Shanghai
    OOM score = 0
    process scheduling policy = nice
    pthread stack size = 1048576

[web]
    default port = 25000
    enable gzip compression = no

[plugins]
    PATH environment variable = /usr/bin:/usr/sbin:/bin:/sbin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
    proc = yes
    diskspace = yes
    timex = no
    cgroups = no
    tc = no
    idlejitter = no
    enable running new plugins = yes
    check for new plugins every = 60
    slabinfo = no
    apps = yes
    fping = no
    charts.d = yes
    ioping = no
    perf = no
    node.d = no
    python.d = no

[health]
    enabled = no

[backend]
    enabled = no

[statsd]
    enable = no

[plugin:proc]
    netdata server resources = no
    /proc/pagetypeinfo = no
    /proc/stat = yes
    /proc/uptime = yes
    /proc/loadavg = yes
    /proc/sys/kernel/random/entropy_avail = yes
    /proc/pressure = yes
    /proc/interrupts = yes
    /proc/softirqs = yes
    /proc/vmstat = yes
    /proc/meminfo = yes
    /sys/kernel/mm/ksm = yes
    /sys/block/zram = yes
    /sys/devices/system/edac/mc = yes
    /sys/devices/system/node = yes
    /proc/net/dev = yes
    /proc/net/wireless = no
    /proc/net/sockstat = no
    /proc/net/sockstat6 = no
    /proc/net/netstat = yes
    /proc/net/snmp = no
    /proc/net/snmp6 = no
    /proc/net/sctp/snmp = no
    /proc/net/softnet_stat = no
    /proc/net/ip_vs/stats = no
    /sys/class/infiniband = no
    /proc/net/stat/conntrack = no
    /proc/net/stat/synproxy = no
    /proc/diskstats = yes
    /proc/mdstat = no
    /proc/net/rpc/nfsd = no
    /proc/net/rpc/nfs = no
    /proc/spl/kstat/zfs/arcstats = no
    /proc/spl/kstat/zfs/pool/state = no
    /sys/fs/btrfs = no
    ipc = yes
    /sys/class/power_supply = no

[plugin:apps]
    command options =  without-users without-groups

There are several important core modifications:

update every: set to 10, i.e. collect data every 10 seconds. Reduce netdata CPU utilization and disk storage by reducing the data collection frequency of netdata; Reducing disk storage can greatly reduce the IO operation, memory read-write frequency and bandwidth overhead caused by netdata querying previous data.
dbengine disk space: set the disk storage space to 128MB to prevent it from occupying too much storage space. Under this configuration, this size can store more than a week of data.
OOM score: set it to 0 to prevent the automatic destruction of netdata due to the shortage of system resources.
process scheduling policy: modify the thread scheduling policy so that netdata can better collect data and prevent data from not being collected when the system resource is tight because the scheduling policy is idel (after all, this is an important time for monitoring).
pthread stack size: modify the stack space of the thread to 1MB.
Compression: enable gzip compression and netdata.compression_ratio is all disabled to no, that is, the stored data is not compressed, which reduces the burden of CPU during compression and decompression.
plugin:apps: application programs disable grouping. Generally, there is only one root user on the embedded system. You don't need to know the information of users and groups, just one person.
plugin:proc: tailor some collection indicators to reduce the amount of data collection.

Through the above modifications, netdata will be more suitable for the operation of embedded devices.

The complete configuration file is given below:

# netdata configuration
# global netdata configuration
[global]
    run as user = root
    hostname = unknown
    update every = 10
    config directory = /var/netdata/etc/netdata
    stock config directory = /var/netdata/lib/netdata/conf.d
    log directory = /var/netdata/var/log/netdata
    web files directory = /var/netdata/share/netdata/web
    cache directory = /var/netdata/var/cache/netdata
    lib directory = /var/netdata/var/lib/netdata
    home directory = /var/netdata/var/lib/netdata
    lock directory = /var/netdata/var/lib/netdata/lock
    plugins directory = "/var/netdata/libexec/netdata/plugins.d" "/var/netdata/etc/netdata/custom-plugins.d"
    memory mode = dbengine
    page cache size = 8
    dbengine disk space = 128
    timezone = Asia/Shanghai
    OOM score = 0
    process scheduling policy = nice
    pthread stack size = 1048576

[web]
    default port = 25000
    enable gzip compression = no

[plugins]
    PATH environment variable = /usr/bin:/usr/sbin:/bin:/sbin:/sbin:/usr   /sbin:/usr/local/bin:/usr/local/sbin
    proc = yes
    diskspace = yes
    timex = no
    cgroups = no
    tc = no
    idlejitter = no
    enable running new plugins = yes
    check for new plugins every = 60
    slabinfo = no
    apps = yes
    fping = no
    charts.d = yes
    ioping = no
    perf = no
    node.d = no
    python.d = no

[health]
    enabled = no

[backend]
    enabled = no

[statsd]
    enable = no

[netdata.compression_ratio]
    enabled = no

[plugin:proc]
    netdata server resources = no
    /proc/pagetypeinfo = no
    /proc/stat = yes
    /proc/uptime = yes
    /proc/loadavg = yes
    /proc/sys/kernel/random/entropy_avail = yes
    /proc/pressure = yes
    /proc/interrupts = yes
    /proc/softirqs = yes
    /proc/vmstat = yes
    /proc/meminfo = yes
    /sys/kernel/mm/ksm = yes
    /sys/block/zram = yes
    /sys/devices/system/edac/mc = yes
    /sys/devices/system/node = yes
    /proc/net/dev = yes
    /proc/net/wireless = no
    /proc/net/sockstat = no
    /proc/net/sockstat6 = no
    /proc/net/netstat = yes
    /proc/net/snmp = no
    /proc/net/snmp6 = no
    /proc/net/sctp/snmp = no
    /proc/net/softnet_stat = no
    /proc/net/ip_vs/stats = no
    /sys/class/infiniband = no
    /proc/net/stat/conntrack = no
    /proc/net/stat/synproxy = no
    /proc/diskstats = yes
    /proc/mdstat = no
    /proc/net/rpc/nfsd = no
    /proc/net/rpc/nfs = no
    /proc/spl/kstat/zfs/arcstats = no
    /proc/spl/kstat/zfs/pool/state = no
    /sys/fs/btrfs = no
    ipc = yes
    /sys/class/power_supply = no    
[plugin:apps]
    command options =  without-users without-groups

[plugin:proc:/proc/diskstats:mtdblock0]
    enable = no

[plugin:proc:/proc/diskstats:mtdblock2]
    enable = no

[plugin:proc:/proc/diskstats:mtdblock3]
    enable = no

[plugin:proc:/proc/diskstats:mtdblock4]
    enable = no

[plugin:proc:/proc/diskstats:mtdblock5]
    enable = no

[plugin:proc:/proc/diskstats:mtdblock1]
    enable = no



[plugin:proc:/proc/net/dev:ip6gre0]
    enabled = no

[plugin:proc:/proc/net/dev:sit0]
    enabled = no

[plugin:proc:/proc/net/dev:lo]
    enabled = no

[plugin:proc:/proc/net/dev:ip6tnl0]
    enabled = no

[plugin:proc:/proc/net/netstat]
    bandwidth = auto
    input errors = auto
    multicast bandwidth = auto
    broadcast bandwidth = auto
    multicast packets = auto
    broadcast packets = auto
    ECN packets = no
    TCP reorders = no
    TCP SYN cookies = no
    TCP out-of-order queue = auto
    TCP connection aborts = auto
    TCP memory pressures = auto
    TCP SYN queue = no
    TCP accept queue = no
    filename to monitor = /proc/net/netstat