vcpkg/scripts/azure-pipelines/linux/provision-image.sh

98 lines
4.0 KiB
Bash
Raw Normal View History

#!/bin/bash
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: MIT
#
sudo apt -y update
sudo apt -y dist-upgrade
# Install common build dependencies
APT_PACKAGES="at curl unzip tar libxt-dev gperf libxaw7-dev cifs-utils \
build-essential g++ gfortran zip libx11-dev libxkbcommon-x11-dev libxi-dev \
libgl1-mesa-dev libglu1-mesa-dev mesa-common-dev libxinerama-dev libxxf86vm-dev \
libxcursor-dev yasm libnuma1 libnuma-dev python-six python3-six python-yaml \
flex libbison-dev autoconf libudev-dev libncurses5-dev libtool libxrandr-dev \
[opentracing] Update VMs 2020-12 (#15151) * Add meson from https://github.com/microsoft/vcpkg/pull/12860/ * Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/ * Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/ * Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/ * Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/ * Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774 * Remove no longer necessary apt-mark calls. * Update nasm on Linux. * Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share * Delete no longer necessary azure storage firewall rules. * Install the newer Windows SDK with the VS installer instead of manually. * Install the VS2015 and VS2017 compilers. * Update Powershell-Core to 7.1.0. * Update source of WDK. * Update pools. * [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled. Fixes: C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001). Fixes: D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error] try { ^ * [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX. Note that this port sets /GS-, possibly because it may be for authoring shellcode. * Bump storage API version to 2020-04-08.
2020-12-23 06:52:58 +08:00
xutils-dev dh-autoreconf autoconf-archive libgles2-mesa-dev ruby-full \
pkg-config meson"
2020-06-02 02:15:11 +08:00
# Additionally required by qt5-base
APT_PACKAGES="$APT_PACKAGES libxext-dev libxfixes-dev libxrender-dev \
[opentracing] Update VMs 2020-12 (#15151) * Add meson from https://github.com/microsoft/vcpkg/pull/12860/ * Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/ * Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/ * Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/ * Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/ * Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774 * Remove no longer necessary apt-mark calls. * Update nasm on Linux. * Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share * Delete no longer necessary azure storage firewall rules. * Install the newer Windows SDK with the VS installer instead of manually. * Install the VS2015 and VS2017 compilers. * Update Powershell-Core to 7.1.0. * Update source of WDK. * Update pools. * [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled. Fixes: C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001). Fixes: D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error] try { ^ * [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX. Note that this port sets /GS-, possibly because it may be for authoring shellcode. * Bump storage API version to 2020-04-08.
2020-12-23 06:52:58 +08:00
libxcb1-dev libx11-xcb-dev libxcb-glx0-dev libxcb-util0-dev"
# Additionally required by qt5-base for qt5-x11extras
APT_PACKAGES="$APT_PACKAGES libxkbcommon-dev libxcb-keysyms1-dev \
libxcb-image0-dev libxcb-shm0-dev libxcb-icccm4-dev libxcb-sync0-dev \
libxcb-xfixes0-dev libxcb-shape0-dev libxcb-randr0-dev \
libxcb-render-util0-dev libxcb-xinerama0-dev libxcb-xkb-dev libxcb-xinput-dev"
# Additionally required by libhdfs3
APT_PACKAGES="$APT_PACKAGES libkrb5-dev"
[opentracing] Update VMs 2020-12 (#15151) * Add meson from https://github.com/microsoft/vcpkg/pull/12860/ * Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/ * Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/ * Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/ * Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/ * Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774 * Remove no longer necessary apt-mark calls. * Update nasm on Linux. * Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share * Delete no longer necessary azure storage firewall rules. * Install the newer Windows SDK with the VS installer instead of manually. * Install the VS2015 and VS2017 compilers. * Update Powershell-Core to 7.1.0. * Update source of WDK. * Update pools. * [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled. Fixes: C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001). Fixes: D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error] try { ^ * [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX. Note that this port sets /GS-, possibly because it may be for authoring shellcode. * Bump storage API version to 2020-04-08.
2020-12-23 06:52:58 +08:00
# Additionally required by kf5windowsystem
APT_PACKAGES="$APT_PACKAGES libxcb-res0-dev"
# Additionally required by mesa
APT_PACKAGES="$APT_PACKAGES python3-setuptools python3-mako"
# Additionally required by some packages to install additional python packages
APT_PACKAGES="$APT_PACKAGES python3-pip"
[opentracing] Update VMs 2020-12 (#15151) * Add meson from https://github.com/microsoft/vcpkg/pull/12860/ * Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/ * Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/ * Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/ * Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/ * Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774 * Remove no longer necessary apt-mark calls. * Update nasm on Linux. * Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share * Delete no longer necessary azure storage firewall rules. * Install the newer Windows SDK with the VS installer instead of manually. * Install the VS2015 and VS2017 compilers. * Update Powershell-Core to 7.1.0. * Update source of WDK. * Update pools. * [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled. Fixes: C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001). Fixes: D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error] try { ^ * [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX. Note that this port sets /GS-, possibly because it may be for authoring shellcode. * Bump storage API version to 2020-04-08.
2020-12-23 06:52:58 +08:00
# Additionally required by rtaudio
APT_PACKAGES="$APT_PACKAGES libasound2-dev"
# Additionally required by qtwebengine
APT_PACKAGES="$APT_PACKAGES nodejs"
# Additionally required by qtwayland
APT_PACKAGES="$APT_PACKAGES libwayland-dev"
# Additionally required by all GN projects
APT_PACKAGES="$APT_PACKAGES python2 python-is-python3"
# Additionally required/installed by Azure DevOps Scale Set Agents
APT_PACKAGES="$APT_PACKAGES liblttng-ust0 libkrb5-3 zlib1g libicu66"
sudo apt -y install $APT_PACKAGES
# Install newer version of nasm than the apt package, required by intel-ipsec
mkdir /tmp/nasm
cd /tmp/nasm
[opentracing] Update VMs 2020-12 (#15151) * Add meson from https://github.com/microsoft/vcpkg/pull/12860/ * Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/ * Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/ * Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/ * Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/ * Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774 * Remove no longer necessary apt-mark calls. * Update nasm on Linux. * Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share * Delete no longer necessary azure storage firewall rules. * Install the newer Windows SDK with the VS installer instead of manually. * Install the VS2015 and VS2017 compilers. * Update Powershell-Core to 7.1.0. * Update source of WDK. * Update pools. * [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled. Fixes: C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001). Fixes: D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error] try { ^ * [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX. Note that this port sets /GS-, possibly because it may be for authoring shellcode. * Bump storage API version to 2020-04-08.
2020-12-23 06:52:58 +08:00
curl -O https://www.nasm.us/pub/nasm/releasebuilds/2.15.05/nasm-2.15.05.tar.gz
tar -xf nasm-2.15.05.tar.gz
cd nasm-2.15.05/
./configure --prefix=/usr && make -j
sudo make install
cd ~
# Install the latest Haskell stack
curl -sSL https://get.haskellstack.org/ | sudo sh
# Install CUDA
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt -y update
Update VMs, CMake to 3.20.1, CUDA to 11.3, and pwsh to 7.1.3 (#17331) * Update to CUDA 11.3 on Windows. * Update PowerShell to 7.1.3. * Update CUDA to 11.3 on Linux. * "Explode" VM provisioning scripts for more consistent feedback during deploy. This resolves the deployment script often hanging with no feedback as to why. * [libdatachannel] Fix use of deprecated result_type typedef. Submitted upstream as https://github.com/paullouisageneau/libdatachannel/pull/413 * [libvpx] Get the libvpx outputs from the correct place. (Perhaps VS2019 version 16.10 moved where these are placed? I've been defensive and left an attempt to find from the old location in place.) * [chromaprint] Support implementations where lrintf is an intrinsic. * Add provision-entire-image script. * [cudnn] Disable download-on-the-fly due to licensing concerns. * Add libnccl to Linux VMs. * [wangle] Disable x64-windows due to an ICE. * [cmake] Update cmake to 3.20.1 to avoid https://gitlab.kitware.com/cmake/cmake/-/issues/21571 race * [libudis86] Fix passing a bogus working directory which fails on CMake 3.20.x * [dartsim] Disable unit tests, examples, and tutorials, some of which have CMake authoring errors rejected by 3.20.1. * Add thrust to the cuda installees. * [tensorflow] Put .bzl in CURRENT_BUILDTREES_DIR to avoid running out of disk space in CI and to respect --clean-after-build. * [dimcli] Skip port broken by changes in VS2019 project system. * [upb] Disable an additional warning. * [libhv] Fix typo DISABLE_PARALLEL => DISABLE_PARALLEL_CONFIGURE * Update pools
2021-04-27 01:27:45 +08:00
sudo apt install -y --no-install-recommends cuda-compiler-11-3 cuda-libraries-dev-11-3 cuda-driver-dev-11-3 \
cuda-cudart-dev-11-3 libcublas-11-3 libcurand-dev-11-3 libcudnn8-dev libnccl2 libnccl-dev
# Install PowerShell
wget -q https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt update
sudo add-apt-repository universe
sudo apt install -y powershell
# Write script to provision disks used by cloud-init
echo "if [ ! -d \"/mnt/vcpkg-ci\" ]; then" > /etc/provision-disks.sh
echo "sudo parted /dev/sdb mklabel gpt" >> /etc/provision-disks.sh
echo "sudo parted /dev/sdb mkpart cidisk ext4 0% 100%" >> /etc/provision-disks.sh
echo "sudo mkfs -t ext4 /dev/sdb1" >> /etc/provision-disks.sh
echo "sudo mkdir /mnt/vcpkg-ci -m=777" >> /etc/provision-disks.sh
echo "echo \"/dev/sdb1 /mnt/vcpkg-ci ext4 barrier=0 0 0\" | sudo tee -a /etc/fstab" >> /etc/provision-disks.sh
echo "sudo mount -a" >> /etc/provision-disks.sh
echo "sudo chmod 777 /mnt/vcpkg-ci" >> /etc/provision-disks.sh
echo "fi" >> /etc/provision-disks.sh
sudo chmod 700 /etc/provision-disks.sh
[opentracing] Update VMs 2020-12 (#15151) * Add meson from https://github.com/microsoft/vcpkg/pull/12860/ * Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/ * Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/ * Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/ * Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/ * Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774 * Remove no longer necessary apt-mark calls. * Update nasm on Linux. * Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share * Delete no longer necessary azure storage firewall rules. * Install the newer Windows SDK with the VS installer instead of manually. * Install the VS2015 and VS2017 compilers. * Update Powershell-Core to 7.1.0. * Update source of WDK. * Update pools. * [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled. Fixes: C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001). Fixes: D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error] try { ^ * [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX. Note that this port sets /GS-, possibly because it may be for authoring shellcode. * Bump storage API version to 2020-04-08.
2020-12-23 06:52:58 +08:00
# provision-image.ps1 will append installation of the SAS token here