netdata cross compilation

preface

netdata is a system monitoring program basically developed by C/C + +, which is mainly used for server monitoring; Because its technical implementation is basically implemented by C/C + +, it is more suitable for embedded device monitoring. Other monitoring integration schemes often cannot be transplanted to embedded devices due to the problem of technical stack.

Netdata can run on embedded devices. Although some aspects are not perfect, after all, netdata is not specifically developed for embedded products, but mainly serves servers; Transplantation on embedded has always been a problem, which is limited by various problems of the target machine itself, such as the low version of cross compilation tool chain, the lack of some environment dependent libraries, and the incompatibility between 32-bit compilation chain and 64 bit compilation chain, which often leads to cross compilation failure.

This paper only makes a complete record of the cross compilation process of netdata, that is, the transplantation and deployment of netdata.

The cross compilation process of netdata is very complex and needs to be carried out carefully. It is recommended to save the snapshot of the virtual machine in advance to prevent it from being difficult to trace back. This installation is only applicable to the Ubuntu system. Due to the use of the apt package management tool of Ubuntu.

ubuntu environment preparation phase

Do not switch terminals during installation

Switch to root

su

Setting environment variables

# PATH and CrossComplie shall be filled in according to their own conditions
export PATH=/opt/hisi-linux/x86-arm/arm-hisiv600-linux/target/bin:${PATH}        # Import cross compilation tool chain installation path
export WorkSpace=$(pwd)                                                          # working space
export CrossComplie="arm-hisiv600-linux"                                         # Cross compilation tool chain

To facilitate execution, you need to record the path of the current folder and your cross compilation tool chain. All the downloaded command lines will use these temporarily entered variables

Installing the Ubuntu tool

# Install the automake plug-in
apt install autoconf automake libtool pkg-config
# Install perl
apt install perl
# Install uuid
# Hint: it's essentially unnecessary. netdata's configure is not written properly
apt install uuid-dev
# Installing 32-bit Libraries
sudo apt install lib32z1-dev
sudo apt install lib32z1 lib32ncurses5
sudo apt install lib32stdc++6
sudo apt install gcc-multilib

Configure cross compilation environment

# Configure cross compilation environment
export CC="${CrossComplie}-gcc"
export AR="${CrossComplie}-ar"
export CXX="${CrossComplie}-c++"
export RANLIB="${CrossComplie}-ranlib"

# Clear environment
rm -rf install
rm -rf netdata

# Build environment
mkdir install
mkdir install/libuv
mkdir install/zlib
mkdir install/liblz4
mkdir install/openssl
mkdir install/libuuid
mkdir install/libuv
mkdir install/netdata
mkdir install/judy

Dependent Library Acquisition

# Basically, it was bad last time. After all, it is difficult to access in China, but replacement is not allowed. These libraries have been tested
cd "${WorkSpace}" || exit
rm -rf depend
mkdir depend
cd "${WorkSpace}"/depend || exit
## Download uuid source code
wget https://nchc.dl.sourceforge.net/project/libuuid/libuuid-1.0.3.tar.gz
## Download zlib source code
git clone https://github.com/madler/zlib.git
## Download libuv source code
git clone https://github.com/libuv/libuv.git
## Download openssl source code
git clone https://github.com/openssl/openssl.git
## Download lz4 source code
git clone https://github.com/lz4/lz4.git
## Download LibJudy source code
wget https://nchc.dl.sourceforge.net/project/judy/judy/Judy-1.0.5/Judy-1.0.5.tar.gz
## Download netdata source code
git clone https://github.com/netdata/netdata.git

Cross compile dependency

Cross compile Judy

cd "${WorkSpace}"/depend || exit
tar zxvf Judy-1.0.5.tar.gz
cd "${WorkSpace}"/depend/judy-1.0.5 || exit
./configure --host="${CrossComplie}" --prefix="${WorkSpace}"/install/judy
### Modify Judy Makefile: modify its cross compilation bug
sed -i "563s/\$(CC)/gcc -m32/g" "${WorkSpace}"/depend/judy-1.0.5/src/JudyL/Makefile
sed -i "563s/\$(CC)/gcc -m32/g" "${WorkSpace}"/depend/judy-1.0.5/src/Judy1/Makefile
make
make install
rm -rf "${WorkSpace}"/depned/judy-1.0.5

Cross compile zlib

cd "${WorkSpace}"/depend/zlib || exit
git checkout v1.2.9
./configure --prefix="${WorkSpace}"/install/zlib
make
make install
make clean

Cross compile libuuid

cd "${WorkSpace}"/depend || exit
tar zxvf libuuid-1.0.3.tar.gz
cd "${WorkSpace}"/depend/libuuid-1.0.3 || exit
./configure --host="${CrossComplie}" --prefix="${WorkSpace}"/install/libuuid
make
make install
rm -rf "${WorkSpace}"/depned/libuuid-1.0.3

Cross compile libuv

cd "${WorkSpace}"/depend/libuv || exit
git checkout v1.42.0
sh autogen.sh
./configure --host="${CrossComplie}" --prefix="${WorkSpace}"/install/libuv
make
make install
make clean

Cross compile openssl

cd "${WorkSpace}"/depend/openssl || exit
git checkout OpenSSL_1_1_0-stable
./config no-asm no-shared --prefix="${WorkSpace}"/install/openssl
### Modify oepnssl Makefile: Remove "- m64"
sed -i "s/ -m64/  /g" "${WorkSpace}"/depend/openssl/Makefile
make
make install
make clean

Cross compile liblz4

cd "${WorkSpace}"/depend/lz4 || exit
git checkout r131
make CC="${CrossComplie}-gcc"
make DESTDIR="${WorkSpace}"/install/liblz4 install

netdata installation

Copy library and header files

cd "${WorkSpace}"/depend/netdata || exit
git checkout v1.31.0
autoreconf -ivf
## Copy dependent libraries and header files
rm -rf 3rdLibrary
mkdir 3rdLibrary
mkdir 3rdLibrary/lib
mkdir 3rdLibrary/include
### Copy judy
cp "${WorkSpace}"/install/judy/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp "${WorkSpace}"/install/judy/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy libzl4
cp "${WorkSpace}"/install/liblz4/usr/local/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp "${WorkSpace}"/install/liblz4/usr/local/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy libuuid
cp "${WorkSpace}"/install/libuuid/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp -rf "${WorkSpace}"/install/libuuid/include/uuid "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy libuv
cp "${WorkSpace}"/install/libuv/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp -rf "${WorkSpace}"/install/libuv/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy oepnssl
cp "${WorkSpace}"/install/openssl/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp -rf "${WorkSpace}"/install/openssl/include/openssl "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy zlib
cp "${WorkSpace}"/install/zlib/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp "${WorkSpace}"/install/zlib/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include

Cross compile netdata

cd "${WorkSpace}"/depend/netdata || exit
./configure --prefix="${WorkSpace}"/install/netdata --host=arm-hisiv600-linux \
    --disable-cloud --disable-ebpf \
    CFLAGS="-I${WorkSpace}/depend/netdata/3rdLibrary/include" \
    LDFLAGS="-L${WorkSpace}/depend/netdata/3rdLibrary/lib" \
    LIBS="-lcrypto -llz4 -luuid -lz -lJudy -lssl -luv -ldl"
make
make install

Deploy netdata

# Configure netdata
cd "${WorkSpace}" || exit
cp -rf "${WorkSpace}"/install/netdata "${WorkSpace}"
mkdir -p "${WorkSpace}"/netdata/var/cache/netdata
mkdir -p "${WorkSpace}"/netdata/var/lib/netdata
mkdir -p "${WorkSpace}"/netdata/var/log/netdata
touch "${WorkSpace}"/netdata/etc/netdata/netdata.conf
## Write default profile
cat>"${WorkSpace}"/netdata/etc/netdata/netdata.conf<<EOF
# netdata configuration
# global netdata configuration
[global]
    run as user = root
    hostname = unknown
    update every = 1
    config directory = /var/netdata/etc/netdata
    stock config directory = /var/netdata/lib/netdata/netdata/conf.d
    log directory = /var/netdata/var/log/netdata
    web files directory = /var/netdata/share/netdata/web
    cache directory = /var/netdata/var/cache/netdata
    lib directory = /var/netdata/var/lib/netdata
    home directory = /var/netdata/var/lib/netdata
    lock directory = /var/netdata/var/lib/netdata/lock
    plugins directory = "/var/netdata/libexec/netdata/plugins.d" "/var/netdata/etc/netdata/custom-plugins.d"
[web]
    default port = 25000
EOF
chown root "${WorkSpace}"/netdata -R
chgrp root "${WorkSpace}"/netdata -R
chmod 0777 "${WorkSpace}"/netdata -R

netdata program running instructions

echo "Program operation instructions"
echo "The generated in the current directory netdata Copy to target /var Directory"
echo "Enter into netdata Directory, use ./sbin/netdata -c ./etc/netdata.conf Run program"
echo "use killall netdata close program"
echo "use [Target machine ip]:25000/netdata.conf see netdata Configure and modify"
echo "The default running directory of the program is configured in /var Directory"
echo ""

appendix

One click installation script

#! /bin/bash
#
# author : haorui
# date : 2021-10-27
# script : netdata_install.h
#
# Note: it needs to run in privileged mode (sudo), that is, under root
# Script description: automatically install netdata
# Check whether it runs under root

function help() {
    echo "netdata Cross compile script"
    echo "This script records netdata All steps in,It is recommended that you follow the script steps to install it manually"
    echo "There are two main reasons for script execution failure::"
    echo "1 - Cloning remote warehouse failed(It is recommended to download the source code manually)"
    echo "2 - There are some differences between different cross compilation tool chains"
    echo "Hint : This script can only be used in ubuntu function"
    echo "Hint : This script is for the 32-bit cross compilation tool chain"
    echo ""
    echo ""

    echo "Script instructions"
    echo "1 - stay sudo Run script under"
    echo "2 - take CrossComplie Modify the cross compilation tool chain for yourself"
    echo "3 - export PATH Own cross compilation tool chain"
    echo "4 - ResourceLocation stay local In this case, the source code will not be downloaded again and built locally"
    echo "It is recommended to download the source code manually and follow the path specified in the script"
    echo ""
    echo ""

    echo "Program operation instructions"
    echo "The generated in the current directory netdata Copy to target /var Directory"
    echo "Enter into netdata Directory, use ./sbin/netdata -c ./etc/netdata.conf Run program"
    echo "use killall netdata close program"
    echo "use [Target machine ip]:25000/netdata.conf see netdata Configure and modify"
    echo "The default running directory of the program is configured in /var Directory"
    echo ""
    echo ""

    echo "Please read carefully. After pressing any key, the script starts to execute:"
    read -r
}

set-x

# Help documentation
help

if [ "$(id -u)" != 0 ]; then
    echo "use sudo Run this script"；
    exit 255
fi

# Script environment variable related
export PATH=/opt/hisi-linux/x86-arm/arm-hisiv600-linux/target/bin:${PATH} # Import cross compilation tool chain installation path
WorkSpace=$(pwd)                                                          # working space
ResourceLocation="local"                                                  # Resource acquisition location
CrossComplie="arm-hisiv600-linux"                                         # Cross compilation tool chain

# Install the automake plug-in
apt install autoconf automake libtool pkg-config

# Install perl
apt install perl

# Install uuid
# Hint: it's essentially unnecessary. netdata's configure is not written properly
apt install uuid-dev

# Installing 32-bit Libraries
sudo apt install lib32z1-dev
sudo apt install lib32z1 lib32ncurses5
sudo apt install lib32stdc++6
sudo apt install gcc-multilib


# Configure cross compilation environment
export CC="${CrossComplie}-gcc"
export AR="${CrossComplie}-ar"
export CXX="${CrossComplie}-c++"
export RANLIB="${CrossComplie}-ranlib"

# Clear environment
rm -rf install
rm -rf netdata

# Build environment
mkdir install
mkdir install/libuv
mkdir install/zlib
mkdir install/liblz4
mkdir install/openssl
mkdir install/libuuid
mkdir install/libuv
mkdir install/netdata
mkdir install/judy

# Download dependency
if [ ${ResourceLocation} == "remote" ]; then
    cd "${WorkSpace}" || exit
    rm -rf depend
    mkdir depend
    cd "${WorkSpace}"/depend || exit
    ## Download uuid source code
    wget https://nchc.dl.sourceforge.net/project/libuuid/libuuid-1.0.3.tar.gz
    ## Download zlib source code
    git clone https://github.com/madler/zlib.git
    ## Download libuv source code
    git clone https://github.com/libuv/libuv.git
    ## Download openssl source code
    git clone https://github.com/openssl/openssl.git
    ## Download lz4 source code
    git clone https://github.com/lz4/lz4.git
    ## Download LibJudy source code
    wget https://nchc.dl.sourceforge.net/project/judy/judy/Judy-1.0.5/Judy-1.0.5.tar.gz
    ## Download netdata source code
    git clone https://github.com/netdata/netdata.git
fi

# Cross compile dependency

## Cross compile Judy
cd "${WorkSpace}"/depend || exit
tar zxvf Judy-1.0.5.tar.gz
cd "${WorkSpace}"/depend/judy-1.0.5 || exit
./configure --host="${CrossComplie}" --prefix="${WorkSpace}"/install/judy
### Modify Judy Makefile: modify its cross compilation bug
sed -i "563s/\$(CC)/gcc -m32/g" "${WorkSpace}"/depend/judy-1.0.5/src/JudyL/Makefile
sed -i "563s/\$(CC)/gcc -m32/g" "${WorkSpace}"/depend/judy-1.0.5/src/Judy1/Makefile
make
make install
rm -rf "${WorkSpace}"/depned/judy-1.0.5

## Cross compile zlib
cd "${WorkSpace}"/depend/zlib || exit
git checkout v1.2.9
./configure --prefix="${WorkSpace}"/install/zlib
make
make install
make clean

## Cross compile libuuid
cd "${WorkSpace}"/depend || exit
tar zxvf libuuid-1.0.3.tar.gz
cd "${WorkSpace}"/depend/libuuid-1.0.3 || exit
./configure --host="${CrossComplie}" --prefix="${WorkSpace}"/install/libuuid
make
make install
rm -rf "${WorkSpace}"/depned/libuuid-1.0.3

## Cross compile libuv
cd "${WorkSpace}"/depend/libuv || exit
git checkout v1.42.0
sh autogen.sh
./configure --host="${CrossComplie}" --prefix="${WorkSpace}"/install/libuv
make
make install
make clean

## Cross compile openssl
cd "${WorkSpace}"/depend/openssl || exit
git checkout OpenSSL_1_1_0-stable
./config no-asm no-shared --prefix="${WorkSpace}"/install/openssl
### Modify oepnssl Makefile: Remove "- m64"
sed -i "s/ -m64/  /g" "${WorkSpace}"/depend/openssl/Makefile
make
make install
make clean

## Cross compile liblz4
cd "${WorkSpace}"/depend/lz4 || exit
git checkout r131
make CC="${CrossComplie}-gcc"
make DESTDIR="${WorkSpace}"/install/liblz4 install

# Cross compile netdata
cd "${WorkSpace}"/depend/netdata || exit
git checkout v1.31.0
autoreconf -ivf
## Copy dependent libraries and header files
rm -rf 3rdLibrary
mkdir 3rdLibrary
mkdir 3rdLibrary/lib
mkdir 3rdLibrary/include
### Copy judy
cp "${WorkSpace}"/install/judy/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp "${WorkSpace}"/install/judy/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy libzl4
cp "${WorkSpace}"/install/liblz4/usr/local/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp "${WorkSpace}"/install/liblz4/usr/local/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy libuuid
cp "${WorkSpace}"/install/libuuid/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp -rf "${WorkSpace}"/install/libuuid/include/uuid "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy libuv
cp "${WorkSpace}"/install/libuv/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp -rf "${WorkSpace}"/install/libuv/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy oepnssl
cp "${WorkSpace}"/install/openssl/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp -rf "${WorkSpace}"/install/openssl/include/openssl "${WorkSpace}"/depend/netdata/3rdLibrary/include
### Copy zlib
cp "${WorkSpace}"/install/zlib/lib/*.a "${WorkSpace}"/depend/netdata/3rdLibrary/lib
cp "${WorkSpace}"/install/zlib/include/* "${WorkSpace}"/depend/netdata/3rdLibrary/include
## Officially started cross compiling netdata
cd "${WorkSpace}"/depend/netdata || exit
./configure --prefix="${WorkSpace}"/install/netdata --host=arm-hisiv600-linux \
    --disable-cloud --disable-ebpf \
    CFLAGS="-I${WorkSpace}/depend/netdata/3rdLibrary/include" \
    LDFLAGS="-L${WorkSpace}/depend/netdata/3rdLibrary/lib" \
    LIBS="-lcrypto -llz4 -luuid -lz -lJudy -lssl -luv -ldl"
make
make install

# Configure netdata
cd "${WorkSpace}" || exit
cp -rf "${WorkSpace}"/install/netdata "${WorkSpace}"
mkdir -p "${WorkSpace}"/netdata/var/cache/netdata
mkdir -p "${WorkSpace}"/netdata/var/lib/netdata
mkdir -p "${WorkSpace}"/netdata/var/log/netdata
touch "${WorkSpace}"/netdata/etc/netdata/netdata.conf
## Write default profile
cat>"${WorkSpace}"/netdata/etc/netdata/netdata.conf<<EOF
# netdata configuration
# global netdata configuration
[global]
    run as user = root
    hostname = unknown
    update every = 1
    config directory = /var/netdata/etc/netdata
    stock config directory = /var/netdata/lib/netdata/netdata/conf.d
    log directory = /var/netdata/var/log/netdata
    web files directory = /var/netdata/share/netdata/web
    cache directory = /var/netdata/var/cache/netdata
    lib directory = /var/netdata/var/lib/netdata
    home directory = /var/netdata/var/lib/netdata
    lock directory = /var/netdata/var/lib/netdata/lock
    plugins directory = "/var/netdata/libexec/netdata/plugins.d" "/var/netdata/etc/netdata/custom-plugins.d"
[web]
    default port = 25000

EOF
chown root "${WorkSpace}"/netdata -R
chgrp root "${WorkSpace}"/netdata -R
chmod 0777 "${WorkSpace}"/netdata -R


echo "Program operation instructions"
echo "The generated in the current directory netdata Copy to target /var Directory"
echo "Enter into netdata Directory, use ./sbin/netdata -c ./etc/netdata.conf Run program"
echo "use killall netdata close program"
echo "use [Target machine ip]:25000/netdata.conf see netdata Configure and modify"
echo "The default running directory of the program is configured in /var Directory"
echo ""
echo ""

other

In the process of cross compiling netdata, Judy is probably the most difficult to compile.

Tags: Linux Operation & Maintenance Ubuntu

Posted on Wed, 27 Oct 2021 04:30:50 -0400 by ManOnScooter

Programmer Help

netdata cross compilation

netdata cross compilation

preface

ubuntu environment preparation phase

Switch to root

Setting environment variables

Installing the Ubuntu tool

Configure cross compilation environment

Dependent Library Acquisition

Cross compile dependency

Cross compile Judy

Cross compile zlib

Cross compile libuuid

Cross compile libuv

Cross compile openssl

Cross compile liblz4

netdata installation

Copy library and header files

Cross compile netdata

Deploy netdata

netdata program running instructions

appendix

One click installation script

other

Hot Tags