# MultiArch cross-compilation with Ubuntu/Debian{#tutorial_crosscompile_with_multiarch} @prev_tutorial{tutorial_arm_crosscompile_with_cmake} @next_tutorial{tutorial_building_tegra_cuda} [TOC] | | | | -: | :- | | Original author | Kumataro | | Compatibility | Ubuntu >=23.04 | |^ | OpenCV >=4.8.0 | @warning This tutorial may contain obsolete information. ## What is "MultiArch" OpenCV may use a lot of 3rdparty libraries for video and image decoding, rendering, acceleration and complex math algorithms. The 3rd party components are found by CMake on the build host cross-compilation allows to build OpenCV for foreign architecture or OS, but we loose that large world of components and have to cross-compile each dependency separately and point to it during OpenCV build. Debian/Ubuntu MultiArch helps to fix this. It allows to install several foreign architecture libraries on host system and use them during OpenCV dependencies resolution. @warning - Following these steps will make your Linux environment a little dirty. If possible, it is better to use VMs or Container(e.g. Docker). - This tutorial expects host and target uses same Ubuntu version. Do not use/mix different versions for external library dependency. - Good: Host and Target are 23.04. - Good: Host and Target are 23.10. - Not Good: Host is 23.04, and Target is 23.10. - Not Good: Host is 23.10, and Target is 23.04. - This tutorial may be used for Debian and its derivatives like Raspberry Pi OS. Please make any necessary changes. ## Download tools Install necessary tools and toolchains for cross-compilation. - git, cmake, pkgconf and build-essential are required basically. - ninja-build is to reduce compilation time(option). - crossbuild-essential-armhf is toolchain package for armv7 target. - crossbuild-essential-arm64 is toolchain package for aarch64 target. @code{.bash} sudo apt update -y sudo apt install -y \ git \ cmake \ pkgconf \ build-essential \ ninja-build \ crossbuild-essential-armhf \ crossbuild-essential-arm64 @endcode If you want to enable Python 3 wrapper, install these packages too. @code{.bash} sudo apt install -y \ python3-minimal \ python3-numpy @endcode ## Working folder structure In this tutorial, following working folder structure are used. @code{.unparsed} /home + kmtr - please replace your account name. + work + opencv - source, cloned from github + opencv_contrib - source, cloned from github + build4-full_arm64 - artifact(for aarch64 target), created by cmake + build4-full_armhf - artifact(for armhf target), created by cmake @endcode 1. Create working folder under your home directory. 2. Clone OpenCV and OpenCV Contrib from repository to work directory. @code{.bash} cd ~ mkdir work cd work git clone --depth=1 https://github.com/opencv/opencv.git git clone --depth=1 https://github.com/opencv/opencv_contrib.git @endcode ## Update apt and dpkg settings These steps are on host. `apt` and `dpkg` are package management systems used in Ubuntu and Debian. Following are setup steps to use MultiArch. ### Step 1. Add apt source for arm64 and armhf Execute `sudo apt edit-sources` to add foreign arch libraries at end of file. Example 1: arm64 and armv7 for Ubuntu 23.04 @code{.unparsed} deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar main restricted deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-updates main restricted deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar universe deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-updates universe deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar multiverse deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-updates multiverse deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-backports main restricted universe multiverse deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-security main restricted deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-security universe deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports lunar-security multiverse @endcode Example 2: arm64 and armv7 for Ubuntu 23.10 @code{.unparsed} deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic main restricted deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-updates main restricted deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic universe deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-updates universe deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic multiverse deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-updates multiverse deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-backports main restricted universe multiverse deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-security main restricted deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-security universe deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports mantic-security multiverse @endcode ### Step 2. Update apt database Update apt database to apply new apt sources. Execute `sudo apt update`. @code{.bash} sudo apt update @endcode ### Step 3. Update dpkg settings Update dpkg settings to support foreign architectures. Execute `sudo dpkg --add-architecture arm64` and/or `sudo dpkg --add-architecture armhf`. @code{.bash} sudo dpkg --add-architecture arm64 sudo dpkg --add-architecture armhf @endcode `sudo dpkg --print-architecture` shows what is host architecture. @code{.bash} sudo dpkg --print-architecture amd64 @endcode And `sudo dpkg --print-foreign-architectures` shows what foreign architectures are supported. @code{.bash} sudo dpkg --print-foreign-architectures arm64 armhf @endcode ### Confirm working pkg-config With MultiArch, several shared libraries and pkg-config information for each architectures are stored into /usr/lib. @code{.unparsed} /usr + lib + aarch64-linux-gnu - shared libraries for arm64 + pkgconfig - pkg-config files for arm64 libraries + arm-linux-gnueabihf - shared libraries for armhf + pkgconfig - pkg-config files for armhf libraries + share + pkgconfig - pkg-config files(for header files) @endcode Confirm to work `pkg-config` using `PKG_CONFIG_PATH`, `PKG_CONFIG_LIBDIR` and `PKG_CONFIG_SYSROOT_DIR` options. for aarch64: @code{.bash} PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu \ PKG_CONFIG_SYSROOT_DIR=/ \ pkg-config --list-all @endcode for armv7: @code{.bash} PKG_CONFIG_PATH=/usr/lib/arm-linux-gnueabihf/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/arm-linux-gnueabihf \ PKG_CONFIG_SYSROOT_DIR=/ \ pkg-config --list-all @endcode ## Cross-compile for aarch64 Following is to compile for target (aarch64) at host (x86-64). ### Step 1. Install external libraries for target into host This step is on host. Install libfreetype-dev, libharfbuzz-dev and FFmpeg packages for target (arm64) into host (x86-64). @code{.bash} sudo apt install -y \ libavcodec-dev:arm64 \ libavformat-dev:arm64 \ libavutil-dev:arm64 \ libswscale-dev:arm64 \ libfreetype-dev:arm64 \ libharfbuzz-dev:arm64 @endcode If you want to enable Python 3 wrapper, install these packages too. @code{.bash} sudo apt install -y \ libpython3-dev:arm64 @endcode If succeed, pkg-config can show information about these packages. For Freetype2 and Harfbuzz: @code{.bash} PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu \ PKG_CONFIG_SYSROOT_DIR=/ \ pkg-config freetype2 harfbuzz --cflags --libs -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -L/usr/lib/aarch64-linux-gnu -lfreetype -lharfbuzz @endcode For FFmpeg: @code{.bash} PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu \ PKG_CONFIG_SYSROOT_DIR=/ \ pkg-config libavcodec libavformat libavutil libswscale --cflags --libs -I/usr/include/aarch64-linux-gnu -L/usr/lib/aarch64-linux-gnu -lavcodec -lavformat -lavutil -lswscale @endcode ### Step 2. Configure OpenCV Settings This step is on host. Execute `cmake` to make cross-compile configuration for aarch64. @note `-DCMAKE_TOOLCHAIN_FILE` should be absolute/real file path, not relative path. @code{.bash} PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu \ PKG_CONFIG_SYSROOT_DIR=/ \ cmake -S opencv \ -B build4-full_arm64 \ -DCMAKE_TOOLCHAIN_FILE=/home/kmtr/work/opencv/platforms/linux/aarch64-gnu.toolchain.cmake \ -DOPENCV_EXTRA_MODULES_PATH=opencv_contrib/modules \ -GNinja @endcode If you want to enable Python 3 wrapper, extra options are needed. @code{.bash} PYTHON3_REALPATH=`realpath /usr/bin/python3` PYTHON3_BASENAME=`basename ${PYTHON3_REALPATH}` PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/aarch64-linux-gnu \ PKG_CONFIG_SYSROOT_DIR=/ \ cmake -S opencv \ -B build4-full_arm64 \ -DCMAKE_TOOLCHAIN_FILE=/home/kmtr/work/opencv/platforms/linux/aarch64-gnu.toolchain.cmake \ -DOPENCV_EXTRA_MODULES_PATH=opencv_contrib/modules \ -DPYTHON3_NUMPY_INCLUDE_DIRS="/usr/local/lib/${PYTHON3_BASENAME}/dist-packages/numpy/core/include/" \ -DPYTHON3_INCLUDE_PATH="/usr/include/${PYTHON3_BASENAME};/usr/include/" \ -DPYTHON3_LIBRARIES=`find /usr/lib/aarch64-linux-gnu/ -name libpython*.so` \ -DPYTHON3_EXECUTABLE="/usr/bin/${PYTHON3_BASENAME}" \ -DPYTHON3_CVPY_SUFFIX=".so" \ -GNinja @endcode @note @parblock Lastly, "python3.XX" string is needed. So this script generate it. - Get real path from "/usr/bin/python3" to "/usr/bin/python3.xx". - Get base name from "/usr/bin/python3.xx" to "pyhton3.xx". @endparblock Following is cmake outputs. - `Host` is `Linux x86_64`. - `Target` is `Linux aarch64`. - FFmpeg is available. @code{.unparsed} -- General configuration for OpenCV 4.8.0-dev ===================================== -- Version control: 408730b -- -- Extra modules: -- Location (extra): /home/kmtr/work/opencv_contrib/modules -- Version control (extra): faa5468 -- -- Platform: -- Timestamp: 2023-12-01T22:02:14Z -- Host: Linux 6.5.0-13-generic x86_64 -- Target: Linux 1 aarch64 -- CMake: 3.27.4 -- CMake generator: Ninja -- CMake build tool: /usr/bin/ninja -- Configuration: Release -- -- CPU/HW features: -- Baseline: NEON FP16 -- required: NEON -- disabled: VFPV3 -- Dispatched code generation: NEON_DOTPROD NEON_FP16 NEON_BF16 -- requested: NEON_FP16 NEON_BF16 NEON_DOTPROD -- NEON_DOTPROD (1 files): + NEON_DOTPROD -- NEON_FP16 (2 files): + NEON_FP16 -- NEON_BF16 (0 files): + NEON_BF16 -- -- C/C++: -- Built as dynamic libs?: YES -- C++ standard: 11 -- C++ Compiler: /usr/bin/aarch64-linux-gnu-g++ (ver 13.2.0) : : -- -- Video I/O: -- DC1394: NO -- FFMPEG: YES -- avcodec: YES (60.3.100) -- avformat: YES (60.3.100) -- avutil: YES (58.2.100) -- swscale: YES (7.1.100) -- avresample: NO -- GStreamer: NO -- v4l/v4l2: YES (linux/videodev2.h) -- @endcode If enabling Python 3 wrapper is succeeded, `Python 3:` section shows more. @code{.unparsed} -- -- Python 3: -- Interpreter: /usr/bin/python3.11 (ver 3.11.6) -- Libraries: /usr/lib/aarch64-linux-gnu/libpython3.11.so -- numpy: /usr/local/lib/python3.11/dist-packages/numpy/core/include/ (ver undefined - cannot be probed because of the cross-compilation) -- install path: lib/python3.11/dist-packages/cv2/python-3.11 -- -- Python (for build): /usr/bin/python3.11 -- @endcode ### Step 3. Build and archive OpenCV libraries and headers This step in in host. Build and install. (This `install` means only that copying artifacts to `install` folder.) @code{.bash} cmake --build build4-full_arm64 sudo cmake --install build4-full_arm64 @endcode Archive artifacts(built libraries and headers) to `opencv_arm64.tgz` with tar command. @code{.bash} tar czvf opencv_arm64.tgz -C build4-full_arm64/install . @endcode And send `opencv_arm64.tgz` to target. ### Step 4. Install dependency libraries at target This step is executed on the target system. Install dependency run-time libraries for OpenCV/OpenCV contrib libraries at target. @code{.bash} sudo apt install -y \ libavcodec60 \ libavformat60 \ libavutil58 \ libswscale7 \ libfreetype6 \ libharfbuzz0b sudo ldconfig @endcode If you want to enable Python 3 wrapper, install these packages too. @code{.bash} sudo apt install -y \ python3-minimal \ python3-numpy @endcode @warning @parblock If version of runtime libraries and/or programs are incremented, apt package names may be changed (e.g. `libswscale6` is used for Ubuntu 23.04, but `libswscale7` is used for Ubuntu 23.10). Looking for it with `apt search` command or https://packages.ubuntu.com/ . @endparblock @warning @parblock External library version between host and target should be same. Please update to the latest version libraries at the same time as possible. Even if the OS versions are the same between the Host and Target, the versions may differ due to additional updates to the libraries. This will cause unexpected problems. For example) - On Host, OpenCV has been build with external libA (v1.0) for target. - libA (v1.1) may be updated. - On Target, libA (v1.1) is installed to use OpenCV. - In this case, versions of libA is difference between compiling and running. @endparblock @warning @parblock If you forget/mismatch to install some necessary libraries, OpenCV will not works well. `ldd` command can detect dependency. If there are any "not found", please install necessary libraries. @code{.bash} ldd /usr/local/lib/libopencv_freetype.so @endcode (Not Good) `freetype module` requires `libharfbuzz.so.0`, but it has not been installed. @code{.unparsed} linux-vdso.so.1 (0xABCDEFG01234567) libopencv_imgproc.so.408 => /usr/local/lib/libopencv_imgproc.so.408 (0xABCDEF001234567) libfreetype.so.6 => /lib/aarch64-linux-gnu/libfreetype.so.6 (0xABCDEF001234567) libharfbuzz.so.0 => not found libopencv_core.so.408 => /usr/local/lib/libopencv_core.so.408 (0xABCDEF001234567) : @endcode (Good) All libraries which are required from `freetype modules` are installed. @code{.unparsed} linux-vdso.so.1 (0xABCDEFG01234567) libopencv_imgproc.so.408 => /usr/local/lib/libopencv_imgproc.so.408 (0xABCDEF001234567) libfreetype.so.6 => /lib/aarch64-linux-gnu/libfreetype.so.6 (0xABCDEF001234567) libharfbuzz.so.0 => /lib/aarch64-linux-gnu/libharfbuzz.so.0 (0xABCDEF001234567) libopencv_core.so.408 => /usr/local/lib/libopencv_core.so.408 (0xABCDEF001234567) : @endcode @endparblock ### Step 5. Install OpenCV libraries to target This step is on target. Receive `opencv_arm64.tgz` from host (generated at Step3), and extract to `/usr/local`. @code{.bash} sudo tar zxvf opencv_arm64.tgz -C /usr/local sudo ldconfig @endcode You can use OpenCV libraries same as self-compiling. Following is OpenCV sample code. Compile and run it on target. Makefile @code{.make} a.out : main.cpp g++ main.cpp -o a.out \ -I/usr/local/include/opencv4 \ -lopencv_core @endcode main.cpp @code{.cpp} #include #include int main(void) { std::cout << cv::getBuildInformation() << std::endl; return 0; } @endcode Execute `make` and run it. @code{.bash} make a.out ./a.out @endcode If you want to enable Python 3 wrapper, execute following command to confirm. @code{.bash} python3 -c "import cv2; print(cv2.getBuildInformation())" @endcode ## Cross-compile for armv7 Following is to compile for target (armhf) at host (x86-64). - To resolve dependencies, `linux-libc-dev:armhf` is required. - To optimize with neon, `-DENABLE_NEON=ON` is needed. @code{.bash} sudo apt install -y \ linux-libc-dev:armhf \ libavcodec-dev:armhf \ libavformat-dev:armhf \ libavutil-dev:armhf \ libswscale-dev:armhf \ libfreetype-dev:armhf \ libharfbuzz-dev:armhf PKG_CONFIG_PATH=/usr/lib/arm-linux-gnueabihf/pkgconfig:/usr/share/pkgconfig \ PKG_CONFIG_LIBDIR=/usr/lib/arm-linux-gnueabihf \ PKG_CONFIG_SYSROOT_DIR=/ \ cmake -S opencv \ -B build4-full_armhf \ -DENABLE_NEON=ON \ -DCMAKE_TOOLCHAIN_FILE=/home/kmtr/work/opencv/platforms/linux/arm-gnueabi.toolchain.cmake \ -DOPENCV_EXTRA_MODULES_PATH=opencv_contrib/modules \ -GNinja cmake --build build4-full_armhf sudo cmake --install build4-full_armhf tar czvf opencv_armhf.tgz -C build4-full_armhf/install . @endcode Following is cmake outputs. - `Host` is `Linux x86_64`. - `Target` is `Linux arm`. - FFmpeg is available. @code{.unparsed} -- General configuration for OpenCV 4.8.0-dev ===================================== -- Version control: 408730b -- -- Extra modules: -- Location (extra): /home/kmtr/work/opencv_contrib/modules -- Version control (extra): faa5468 -- -- Platform: -- Timestamp: 2023-12-02T03:39:58Z -- Host: Linux 6.5.0-13-generic x86_64 -- Target: Linux 1 arm -- CMake: 3.27.4 -- CMake generator: Ninja -- CMake build tool: /usr/bin/ninja -- Configuration: Release -- -- CPU/HW features: -- Baseline: NEON -- requested: DETECT -- required: NEON -- disabled: VFPV3 -- -- C/C++: -- Built as dynamic libs?: YES -- C++ standard: 11 -- C++ Compiler: /usr/bin/arm-linux-gnueabihf-g++ (ver 13.2.0) : : -- -- Video I/O: -- DC1394: NO -- FFMPEG: YES -- avcodec: YES (60.3.100) -- avformat: YES (60.3.100) -- avutil: YES (58.2.100) -- swscale: YES (7.1.100) -- avresample: NO -- GStreamer: NO -- v4l/v4l2: YES (linux/videodev2.h) -- @endcode