xmake v2.5.9 release, improve C++20 module, and support Nim, Keil MDK and Unity Build

xmake It is a lightweight cross platform construction tool based on Lua. It uses xmake.lua to maintain project construction. Compared with makefile/CMakeLists.txt, the configuration syntax is more concise and intuitive. It is very friendly to novices. It can get started quickly in a short time, allowing users to focus more on actual project development.

In this version, we have added a large number of heavyweight new features, such as the construction support of Nim language project, Keil MDK, Circle and Wasi tool chain support.

In addition, we have greatly improved C++20 Modules, which not only supports the latest gcc-11, clang and msvc compilers, but also automatically analyzes the dependencies between modules to achieve the maximum degree of parallel compilation support.

Finally, another useful feature is Unity Build support, through which we can greatly improve the compilation speed of C + + code.

Introduction to new features

Nimlang project construction

Recently, we added a new build support for Nimlang project. See the following for relevant issues: #1756

Create an empty project

We can use the xmake create command to create an empty project.

xmake create -l nim -t console test
xmake create -l nim -t static test
xmake create -l nim -t shared test

Console Application

add_rules("mode.debug", "mode.release")

target("test")
    set_kind("binary")
    add_files("src/main.nim")
$ xmake -v
[ 33%]: linking.release test
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/test/macosx/x86_64/release/nimcache -o:b
uild/macosx/x86_64/release/test src/main.nim
[100%]: build ok!

Static library program

add_rules("mode.debug", "mode.release")

target("foo")
    set_kind("static")
    add_files("src/foo.nim")

target("test")
    set_kind("binary")
    add_deps("foo")
    add_files("src/main.nim")
$ xmake -v
[ 33%]: linking.release libfoo.a
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/foo/macosx/x86_64/release/nimcache --app
:staticlib --noMain --passC:-DNimMain=NimMain_B6D5BD02 --passC:-DNimMainInner=NimMainInner_B6D5B
D02 --passC:-DNimMainModule=NimMainModule_B6D5BD02 --passC:-DPreMain=PreMain_B6D5BD02 --passC:-D
PreMainInner=PreMainInner_B6D5BD02 -o:build/macosx/x86_64/release/libfoo.a src/foo.nim
[ 66%]: linking.release test
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/test/macosx/x86_64/release/nimcache --pa
ssL:-Lbuild/macosx/x86_64/release --passL:-lfoo -o:build/macosx/x86_64/release/test src/main.nim
[100%]: build ok!

Dynamic library program

add_rules("mode.debug", "mode.release")

target("foo")
    set_kind("shared")
    add_files("src/foo.nim")

target("test")
    set_kind("binary")
    add_deps("foo")
    add_files("src/main.nim")
$ xmake -rv
[ 33%]: linking.release libfoo.dylib
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/foo/macosx/x86_64/release/nimcache --app
:lib --noMain -o:build/macosx/x86_64/release/libfoo.dylib src/foo.nim
[ 66%]: linking.release test
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/test/macosx/x86_64/release/nimcache --pa
ssL:-Lbuild/macosx/x86_64/release --passL:-lfoo -o:build/macosx/x86_64/release/test src/main.nim
[100%]: build ok!

C code mixed compilation

add_rules("mode.debug", "mode.release")

target("foo")
    set_kind("static")
    add_files("src/*.c")

target("test")
    set_kind("binary")
    add_deps("foo")
    add_files("src/main.nim")

Nimble dependent package integration

For a complete example, see: Nimble Package Example

add_rules("mode.debug", "mode.release")

add_requires("nimble::zip >0.3")

target("test")
    set_kind("binary")
    add_files("src/main.nim")
    add_packages("nimble::zip")

main.nim

import zip/zlib

echo zlibVersion()

Native dependent package integration

For a complete example, see: Native Package Example

add_rules("mode.debug", "mode.release")

add_requires("zlib")

target("test")
    set_kind("binary")
    add_files("src/main.nim")
    add_packages("zlib")

main.nim

proc zlibVersion(): cstring {.cdecl, importc}

echo zlibVersion()

Unity Build acceleration

As we know, C + + code compilation is usually very slow, because each code file needs to parse the introduced header file.

Through Unity Build, we speed up the compilation of the project by combining multiple cpp files into one. The main advantage is to reduce the repeated work of parsing and compiling the contents of header files contained in multiple source files. The contents of header files usually account for most of the code in the source files after preprocessing.

Unity build also reduces the overhead caused by having a large number of small source files by reducing the number of target files created and processed by the compilation chain, and allows interprocedural analysis and optimization (similar to optimization when effects link) across files forming a unified build task.

It can greatly improve the compilation speed of C/C + + code, usually by 30%. However, according to the complexity of the project, its benefits still depend on its own project.

xmake also supports this build pattern in v2.5.9. For relevant issues, see #1019.

How do I enable it?

We provide two built-in rules to deal with Unity Build for C and C + + code respectively.

add_rules("c.unity_build")
add_rules("c++.unity_build")

Batch mode

By default, as long as the above rules are set, Unity Build in Batch mode will be enabled, that is, xmake will automatically organize and merge according to the project code file.

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2})
    add_files("src/*.c", "src/*.cpp")

We can additionally specify the size and quantity of each merged Batch by setting the {batchsize = 2} parameter to the rule. Here, every two C + + files are automatically merged and compiled.

The compilation effect is as follows:

$ xmake -r
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_642A245F.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_bar.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_73161A20.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_F905F036.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_foo.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/main.c
[ 77%]: linking.release test
[100%]: build ok

Since we only enabled Unity Build of C + +, the C code is compiled one by one normally. In addition, in Unity Build mode, we can still accelerate parallel compilation as much as possible without conflict with each other.

If the batchsize parameter is not set, all files will be merged into one file for compilation by default.

Group mode

If the automatic merging effect of the above Batch mode is not ideal, we can also use custom grouping to manually configure which files are merged together for compilation, which makes users more flexible and controllable.

target("test")
    set_kind("binary")
    add_rules("c++.unity_build", {batchsize = 0}) -- disable batch mode
    add_files("src/*.c", "src/*.cpp")
    add_files("src/foo/*.c", {unity_group = "foo"})
    add_files("src/bar/*.c", {unity_group = "bar"})

We use {unity_group = "foo"} to specify the name of each group and which files are included. The files of each group will be merged into a code file separately.

In addition, batchsize = 0 also forcibly disables the Batch mode, that is, unit is not set_ For code files grouped by group, we will compile them separately, and automatic merging will not be started automatically.

Batch and Group mixed mode

As long as we change the above batchsize = 0 to a non-0 value, we can enable the remaining code files to continue to open the Batch mode for automatic merge and compilation in the grouping mode.

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2})
    add_files("src/*.c", "src/*.cpp")
    add_files("src/foo/*.c", {unity_group = "foo"})
    add_files("src/bar/*.c", {unity_group = "bar"})

Ignore specified file

In Batch mode, all files will be merged by default because it is an automatic merge operation. However, if we do not want some code files to participate in the merge, we can also ignore them through {unity_ignored = true}.

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2})
    add_files("src/*.c", "src/*.cpp")
    add_files("src/test/*.c", {unity_ignored = true}) -- ignore these files

Unique ID

Although the benefits brought by Unity Build are good, we will still encounter some unexpected situations. For example, there are global variables and functions with the same name in our two code files under the global namespace.

Then, merge compilation will lead to compilation conflict. The compiler usually reports global variable redefinition errors.

In order to solve this problem, we need to make some changes in the user code, and then cooperate with the construction tool to solve it.

For example, both foo.cpp and bar.cpp have global variables i.

foo.cpp

namespace {
    int i = 42;
}

int foo()
{
    return i;
}

bar.cpp

namespace {
    int i = 42;
}

int bar()
{
    return i;
}

Then, our merge compilation will conflict. We can introduce a Unique ID to isolate the global anonymous space.

foo.cpp

namespace MY_UNITY_ID {
    int i = 42;
}

int foo()
{
    return MY_UNITY_ID::i;
}

bar.cpp

namespace MY_UNITY_ID {
    int i = 42;
}

int bar()
{
    return MY_UNITY_ID::i;
}

Next, we also need to ensure that after the code is merged, my_ UNITY_ The definitions of ID in foo and bar are completely different. A unique ID value can be calculated according to the file name, which does not conflict with each other, that is, the following merging effect can be achieved:

#define MY_UNITY_ID <hash(foo.cpp)>
#include "foo.c"
#undef MY_UNITY_ID
#define MY_UNITY_ID <hash(bar.cpp)>
#include "bar.c"
#undef MY_UNITY_ID

This seems troublesome, but the user doesn't need to care about these. xmake will automatically process them when merging. The user only needs to specify the name of the Unique ID, such as the following:

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2, uniqueid = "MY_UNITY_ID"})
    add_files("src/*.c", "src/*.cpp")

This method can be used to deal with global variables, global duplicate names, macro definitions and functions to avoid conflicts.

C++20 Modules

xmake uses. mpp as the default module extension, but also supports. Ixx,. CPPM,. Mxx and other extensions.

In the early days, xmake tentatively supported C++ Modules TS, but at that time, gcc was not well supported, and the dependencies between modules were not supported.

Recently, we have made a lot of improvements to xmake. We have fully supported the C++20 Modules construction support of gcc-11/clang/msvc, and can automatically analyze the dependencies between modules to maximize parallel compilation.

At the same time, the new version of clang/msvc is also handled better.

set_languages("c++20")
target("test")
    set_kind("binary")
    add_files("src/*.cpp", "src/*.mpp")

For more examples: C++ Modules

Lua5.4 runtime support

In the last version, we added support for Lua 5.3 runtime. In this version, we further upgraded Lua runtime to 5.4. Compared with 5.3, the runtime performance and memory utilization are greatly improved.

However, at present, the default runtime of xmake is luajit. It is expected that version 2.6.1 (that is, the next version) will officially switch to Lua5.4 as the default runtime.

Although the Lua runtime is switched, it is completely insensitive to the client and fully compatible with the existing project configuration, because xmake originally provides a layer of encapsulation for the exposed api,
Interfaces with compatibility problems between lua versions, such as setfenv, ffi, etc., are hidden internally and are not exposed to users.

Keil MDK tool chain support

In this version, we also added Keil/MDK embedded compilation tool chain support. Related example projects: Example

xmake will automatically detect the compiler installed by Keil/MDK and related issues #1753.

Compile using armcc

$ xmake f -p cross -a cortex-m3 --toolchain=armcc -c
$ xmake

Compile with armclang

$ xmake f -p cross -a cortex-m3 --toolchain=armclang -c
$ xmake

Console Application

target("hello")
    add_deps("foo")
    add_rules("mdk.console")
    add_files("src/*.c", "src/*.s")
    add_defines("__EVAL", "__MICROLIB")
    add_includedirs("src/lib/cmsis")

Static library program

add_rules("mode.debug", "mode.release")

target("foo")
    add_rules("mdk.static")
    add_files("src/foo/*.c")

Wasi tool chain support

Previously, we supported the wasm platform emcc tool chain to build wasm programs. Here, we added another wasm tool chain with WASI enabled to replace emcc.

$ xmake f -p wasm --toolchain=wasi
$ xmake

Circle tool chain support

We also added support for the circle compiler, a new C++20 compiler with some interesting compile time metaprogramming features. Interested students can check it on the official website: https://www.circle-lang.org/

$ xmake f --toolchain=circle
$ xmake

gcc-8/9/10/11 specific version support

If the user additionally installs specific versions of GCC tool chains such as gcc-11 and gcc-10, the local GCC program name may be / usr/bin/gcc-11.

One way is to switch through xmake f --cc=gcc-11 --cxx=gcc-11 --ld=g++-11 one by one, but it is very cumbersome.

Therefore, xmake also provides a faster switching method:

$ xmake f --toolchain=gcc-11 -c
$ xmake

You can quickly switch the entire GCC tool chain by specifying the version name corresponding to gcc-11.

Characteristic detection of C++17/20 compiler

xmake provides check_features auxiliary interface to detect compiler features.

includes("check_features.lua")

target("test")
    set_kind("binary")
    add_files("*.c")
    add_configfiles("config.h.in")
    configvar_check_features("HAS_CONSTEXPR", "cxx_constexpr")
    configvar_check_features("HAS_CONSEXPR_AND_STATIC_ASSERT", {"cxx_constexpr", "c_static_assert"}, {languages = "c++11"})

config.h.in

${define HAS_CONSTEXPR}
${define HAS_CONSEXPR_AND_STATIC_ASSERT}

config.h

/* #undef HAS_CONSTEXPR */
#define HAS_CONSEXPR_AND_STATIC_ASSERT 1

In version 2.5.9, we added c++17 feature detection:

Property name
cxx_aggregate_bases
cxx_aligned_new
cxx_capture_star_this
cxx_constexpr
cxx_deduction_guides
cxx_enumerator_attributes
cxx_fold_expressions
cxx_guaranteed_copy_elision
cxx_hex_float
cxx_if_constexpr
cxx_inheriting_constructors
cxx_inline_variables
cxx_namespace_attributes
cxx_noexcept_function_type
cxx_nontype_template_args
cxx_nontype_template_parameter_auto
cxx_range_based_for
cxx_static_assert
cxx_structured_bindings
cxx_template_template_args
cxx_variadic_using

c++20 feature detection is also added:

Property name
cxx_aggregate_paren_init
cxx_char8_t
cxx_concepts
cxx_conditional_explicit
cxx_consteval
cxx_constexpr
cxx_constexpr_dynamic_alloc
cxx_constexpr_in_decltype
cxx_constinit
cxx_deduction_guides
cxx_designated_initializers
cxx_generic_lambdas
cxx_impl_coroutine
cxx_impl_destroying_delete
cxx_impl_three_way_comparison
cxx_init_captures
cxx_modules
cxx_nontype_template_args
cxx_using_enum

Xrepo package virtual environment management

Enter virtual environment

xmake's xrepo package management tool can now support package virtual machine environment management, similar to nixos's nixpkgs.

We can customize some package configurations by adding the xmake.lua file in the current directory, and then enter the specific package virtual environment.

add_requires("zlib 1.2.11")
add_requires("python 3.x", "luajit")
$ xrepo env shell
> python --version
> luajit --version

We can also configure and load the corresponding tool chain environment in xmake.lua, such as the compilation environment for loading vs.

set_toolchains("msvc")

Manage virtual environments

We can use the following command to globally register the specified virtual environment configuration into the system to facilitate fast switching.

$ xrepo env --add /tmp/base.lua

At this time, we save a global virtual environment called base, which we can view through the list command.

$ xrepo env --list
/Users/ruki/.xmake/envs:
  - base
envs(1) found!

We can also delete it.

$ xrepo env --remove base

Switch Global Virtual Environment

If we register multiple virtual environments, we can also quickly switch them.

$ xrepo env -b base shell
> python --version

Or directly load the specified virtual environment and run specific commands

$ xrepo env -b base python --version

xrepo env -b/--bind is to bind the specified virtual environment. For more details, see: #1762

Header Only target type

For target, we have added the headeronly target type. For this type of target program, we will not actually compile them because it has no source file to be compiled.

However, it contains a header file list, which is usually used for the installation of headeronly library project, the generation of file list of IDE project, and the generation of cmake/pkgconfig import file in the installation phase.

For example:

add_rules("mode.release", "mode.debug")

target("foo")
    set_kind("headeronly")
    add_headerfiles("src/foo.h")
    add_rules("utils.install.cmake_importfiles")
    add_rules("utils.install.pkgconfig_importfiles")

For more details, see: #1747

Find package from CMake

Now cmake is the de facto standard, so the find provided by cmake_ Package can find a large number of libraries and modules. We completely reuse this part of cmake ecology to expand xmake's package integration.

We can find_package("cmake::xxx") uses cmake to find some packages. xmake will automatically generate a cmake script to call cmake's find_package to find some packages and get the package information.

For example:

$ xmake l find_package cmake::ZLIB
{
  links = {
    "z"
  },
  includedirs = {
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.
15.sdk/usr/include"
  },
  linkdirs = {
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.
15.sdk/usr/lib"
  }
}
$ xmake l find_package cmake::LibXml2
{
  links = {
    "xml2"
  },
  includedirs = {
    "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2"
  },
  linkdirs = {
    "/usr/lib"
  }
}

Specify version

find_package("cmake::OpenCV", {required_version = "4.1.1"})

assignment component

find_package("cmake::Boost", {components = {"regex", "system"}})

Preset switch

find_package("cmake::Boost", {components = {"regex", "system"}, presets = {Boost_USE_STATIC_LIB = true}})
set(Boost_USE_STATIC_LIB ON) -- will be used in FindBoost.cmake
find_package(Boost REQUIRED COMPONENTS regex system)

Setting environment variables

find_package("cmake::OpenCV", {envs = {CMAKE_PREFIX_PATH = "xxx"}})

Specify the custom FindFoo.cmake module script directory

mydir/cmake_modules/FindFoo.cmake

find_package("cmake::Foo", {moduledirs = "mydir/cmake_modules"})

Package dependent integration

package("xxx")
    on_fetch(function (package, opt)
         return package:find_package("cmake::xxx", opt)
    end)
package_end()

add_requires("xxx")

Package dependent integration (optional component)

package("boost")
    add_configs("regex",   { description = "Enable regex.", default = false, type = "boolean"})
    on_fetch(function (package, opt)
         opt.components = {}
         if package:config("regex") then
             table.insert(opt.components, "regex")
         end
         return package:find_package("cmake::Boost", opt)
    end)
package_end()

add_requires("boost", {configs = {regex = true}})

Related issues: #1632

Add custom commands to CMakelists.txt

We have further improved the cmake generator. Now we can sequence the custom scripts in the rule into a command list and generate them together into CMakelists.txt

However, currently only serialization of batch CMDS series scripts is supported.

rule("foo")
    after_buildcmd(function (target, batchcmds, opt)
        batchcmds:show("hello xmake!")
        batchcmds:cp("xmake.lua", "/tmp/")
        -- batchcmds:execv("echo", {"hello", "world!"})
        -- batchcmds:runv("echo", {"hello", "world!"})
    end)

target("test")
    set_kind("binary")
    add_rules("foo")
    add_files("src/*.c")

It will generate CMakelists.txt similar to the following

# ...
add_custom_command(TARGET test
    POST_BUILD
    COMMAND echo hello xmake!
    VERBATIM
)
add_custom_command(TARGET test
    POST_BUILD
    COMMAND cp xmake.lua /tmp/
    VERBATIM
)
target_sources(test PRIVATE
    src/main.c
)

But cmake's add_ CUSTOM_ COMMAND PRE_ The actual effect of build is quite different on different generators, which can not meet our needs. Therefore, we have done a lot of processing to support it.

Related issues: #1735

Improved installation support for NixOS

We also improved the get.sh installation script to better support nixOS.

Update content

New features

  • #1736 : support Wasi SDK tool chain
  • Support Lua 5.4 runtime
  • Add gcc-8, gcc-9, gcc-10, gcc-11 tool chain
  • #1623 : support find_package find package from cmake
  • #1747 : add set_kind("headeronly") can better handle the installation of headeronly library
  • #1019 : support Unity build
  • #1438 : added xmake L cli.amalmate command to support code merging
  • #1765 : support nim language
  • #1762 : Specifies the environment configuration for xrepo env management and switching
  • #1767 : support Circle compiler
  • #1753 : armcc/armclang tool chain supporting Keil/MDK
  • #1774 : add table.contains api
  • #1735 : add custom commands to cmake generator
  • #1781 : improved get.sh installation script to support nixos

improvement

  • #1528 : detect c++17/20 features
  • #1729 : improve the support of C++20 modules for clang/gcc/msvc, and support inter module dependent compilation and parallel optimization
  • #1779 : improved ml.exe/x86 to remove the built-in - Gd option

Posted on Sun, 31 Oct 2021 10:37:16 -0400 by cueball2000uk