Commit Graph

45 Commits

Author SHA1 Message Date
Kai Pastor
827a54c404
[ci] Don't emit warnings for absence of failure logs ()
* No warning for absence of failure logs

* Update condition for osx and linux

Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
2021-09-28 14:00:33 -07:00
Billy O'Neal
93a8320399
Back out "cascades" enforcement because it caused too many stealth merge conflicts. ()
Mostly reverts https://github.com/microsoft/vcpkg/pull/20140

See https://github.com/microsoft/vcpkg/discussions/20180 for further discussion.
2021-09-15 11:39:39 -07:00
Billy O'Neal
5395ad9a4f
Add "cascades" enforcement ()
* Add "cascades" enforcement as requested by https://github.com/microsoft/vcpkg/pull/20074#issuecomment-916565430

* Add counts.

* Fix arm-uwp count.

* Update counts.

* Update counts
2021-09-14 17:46:38 -07:00
Billy O'Neal
c26477297c
Update pwsh to 7.1.14, separate image minting from vmss minting, and update VMs. ()
* Update Powershell-Core to 7.1.4.

* Add deployment of .NET 4.7.2 targeting pack, extracted from https://github.com/microsoft/vcpkg/pull/19320

* Separate image minting from vmss minting.

* Update pools.

* Fix image minting typo on Windows.
2021-09-11 13:43:19 -07:00
Billy O'Neal
f78c96f706
Decouple the storage account used for binary caching from the images. ()
* Decouple the storage account used for binary caching from the images.

We are being forced to change infrastructure somewhat more frequently than we would like, so change the binary caches to use a shared storage account like we already use for asset caching.

* blob blob blob

* Add quotes

* Avoid name stomping.
2021-09-03 00:02:29 -07:00
Billy O'Neal
403c952a0c
Clean the downloads directory every CI run. ()
Now that vcpkg has artifact caching, meaning downloading artifacts usually doesn't have to leave the datacenter, there's no need to potentially dirty different runs with leftover downloads from each other.

This resulted in some "impossible" results in https://github.com/microsoft/vcpkg/pull/18687 where a previous PR run accidentially downloaded the Windows version of nodejs to /mnt/vcpkg-ci/downloads/node-v14.17.4-linux-x64.tar.xz
2021-08-02 14:15:48 -07:00
Billy O'Neal
f698b0e8be
Change VM size to Standard_D32ds_v4. For some reason they are cheaper than D16 right now??? ()
Remove no longer used "unstable" scaffolding.
Use ephemeral OS disks for better latency.
Add ability to change whether temp disks are used or explicitly provisioned disks are used; explicitly provisioned disks turned out to be more expensive than expected before.
Disable Windows Updates in the scale set because the machines are very short lived; working on a mechanism of patching the images instead.
2021-07-15 13:47:58 -07:00
Billy O'Neal
957d2024b8
Update to Ubuntu 20.04. ()
* Update Ubuntu to 20.04.

* [tfhe] Disable -Wall -Werror

* [sophus] Disable -Werror.

* Fix typo in deployment of CUDA.

* Add libxxf86vm-dev for opensubdiv.

* [opensubdiv] Fix OpenCL search on Ubuntu 20.04, and document libxxf86vm-dev dependency

* [msgpack11] Disable -Werror.

* [qtwayland, qt5wayland] Add system library notice.

* Add python-is-python3 to unbreak tensorflow, v8, and others

* [graphqlparser] Remove directive rejected by Bison 3.5.1

* [ogdf] Mark as failing in ci.baseline.txt. Issue filed https://github.com/microsoft/vcpkg/issues/18936

* [bde] Mark failing in ci.baseline.txt, issue filed https://github.com/microsoft/vcpkg/issues/18937

* [soem] Patch out -Werror

* Update pool.
2021-07-14 12:46:05 -07:00
Billy O'Neal
75beef03d6
Update vcpkg tool to resolve leak of the asset cache token () 2021-06-29 13:47:39 -07:00
Billy O'Neal
2ed44b4546
Update VMs and pick up VS2019 16.10 ()
* Cherry-pick https://github.com/microsoft/vcpkg/pull/15598

* Hook deploy-inteloneapi into create-vmss.ps1.

* Add script to resolve https://github.com/microsoft/vcpkg/issues/17521

* Move tls settings deployment to the front and respond to script triggering a reboot.

* Go back to provisioning an extra disk to workaround https://github.com/microsoft/vcpkg/issues/18379

* Disallow public access to blob storage and require TLS 1.2.

* Update Pools.

* Update tool to 2021-06-19

* [simage] Skip simage on uwp platforms as it appears broken by 16.10.

* [tensorflow-cc] Skip because changes in our MacOS hardware broke the port.
2021-06-22 19:16:04 -07:00
ras0219
8c497bb647
[vcpkg] Add experimental $X_VCPKG_ASSET_SOURCES for source caching ()
* [vcpkg] Add experimental $VCPKG_X_READWRITE_MIRROR_URL_TEMPLATE for source caching

* [vcpkg] Use prerelease version for testing

* [docs] Add assetcaching.md

* Revert bootstrap changes

* [vcpkg_download_distfile] Deprecate SILENT_EXIT

* cmake_parse_arguments() always defines option arguments

* Adjust CI baseline

* [vcpkg_download_distfile] Address CR comments, restore ARIA2

* [CI] Add source caching to Linux and OSX CI passes

* [ci.baseline.txt] Skip chartdir on more platforms
2021-06-04 16:48:26 -07:00
Billy O'Neal
a9b27ed5df
Update VMs, CMake to 3.20.1, CUDA to 11.3, and pwsh to 7.1.3 ()
* Update to CUDA 11.3 on Windows.

* Update PowerShell to 7.1.3.

* Update CUDA to 11.3 on Linux.

* "Explode" VM provisioning scripts for more consistent feedback during deploy. This resolves the deployment script often hanging with no feedback as to why.

* [libdatachannel] Fix use of deprecated result_type typedef.

Submitted upstream as https://github.com/paullouisageneau/libdatachannel/pull/413

* [libvpx] Get the libvpx outputs from the correct place.

(Perhaps VS2019 version 16.10 moved where these are placed? I've been defensive and left an attempt to find from the old location in place.)

* [chromaprint] Support implementations where lrintf is an intrinsic.

* Add provision-entire-image script.

* [cudnn] Disable download-on-the-fly due to licensing concerns.

* Add libnccl to Linux VMs.

* [wangle] Disable x64-windows due to an ICE.

* [cmake] Update cmake to 3.20.1 to avoid https://gitlab.kitware.com/cmake/cmake/-/issues/21571 race

* [libudis86] Fix passing a bogus working directory which fails on CMake 3.20.x

* [dartsim] Disable unit tests, examples, and tutorials, some of which have CMake authoring errors rejected by 3.20.1.

* Add thrust to the cuda installees.

* [tensorflow] Put .bzl in CURRENT_BUILDTREES_DIR to avoid running out of disk space in CI and to respect --clean-after-build.

* [dimcli] Skip port broken by changes in VS2019 project system.

* [upb] Disable an additional warning.

* [libhv] Fix typo DISABLE_PARALLEL => DISABLE_PARALLEL_CONFIGURE

* Update pools
2021-04-26 10:27:45 -07:00
Billy O'Neal
aa60b7efa5
[vcpkg] Download vcpkg.exe rather than building it in bootstrap on Windows. ()
This reduces bootstrap cost for Windows customers, resolving the issue initially submitted as  .

The `toolsrc` tree was extracted to https://github.com/microsoft/vcpkg-tool. `bootstrap.sh` was changed to download the right source tarball, extract, and build it. This was chosen over the previous attempt, a submodule, over concerns of accidentally destroying people's local modifications.
2021-02-04 10:15:44 -08:00
ras0219
5793c4bd9f
[vcpkg] Miscellaneous internal improvements extracted from ()
* [vcpkg] Miscellaneous internal improvements extracted from 

* [vcpkg] CR comments

* [armadillo] Use vcpkg_from_git() to workaround gitlab missing archive

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
2021-02-02 15:00:33 -08:00
Billy O'Neal
4d136ef25f
[vcpkg] Add vcpkg_minimum_required as a replacement for VERSION.txt. () 2021-01-20 12:07:41 -08:00
Billy O'Neal
229f537c94
[opentracing] Update VMs 2020-12 ()
* Add meson from https://github.com/microsoft/vcpkg/pull/12860/

* Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/

* Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/

* Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/

* Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/

* Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774

* Remove no longer necessary apt-mark calls.

* Update nasm on Linux.

* Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share

* Delete no longer necessary azure storage firewall rules.

* Install the newer Windows SDK with the VS installer instead of manually.

* Install the VS2015 and VS2017 compilers.

* Update Powershell-Core to 7.1.0.

* Update source of WDK.

* Update pools.

* [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled.

Fixes:
C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001).

Fixes:
D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error]
    try {
    ^

* [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX.

Note that this port sets /GS-, possibly because it may be for authoring shellcode.

* Bump storage API version to 2020-04-08.
2020-12-22 14:52:58 -08:00
Billy O'Neal
bf7dbde05e
[vcpkg] Always publish failure logs, even on success. ()
This way we get failure logs for things excluded by ci.baseline.txt in the nightly builds.
2020-10-27 15:35:28 -07:00
Billy O'Neal
0890b5a25f
[vcpkg] Add build scripts to produce signed vcpkg binaries. () 2020-09-21 17:27:35 -07:00
Billy O'Neal
8054263f15
[skia, ompl, capnproto] Update VMs for 2020-08-12 (including VS 16.7) ()
Co-authored-by: Nicole Mazzuca <mazzucan@outlook.com>
2020-08-27 12:27:39 -07:00
Billy O'Neal
4e65ce52ab
Sort file lists and failure logs together. () 2020-08-22 22:23:59 -07:00
ras0219
481738beae
[vcpkg] Add vcpkg export to E2E tests. Enable E2E tests on all platforms. ()
* [vcpkg] Add `vcpkg export` to E2E tests. Enable E2E tests on MacOS.

* [vcpkg] Fix export --raw --output-dir=/path/ by changing directory to new export root

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
2020-08-10 10:22:51 -07:00
ras0219
bc07f4b052
[vcpkg-ci] Always publish file lists ()
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
2020-07-13 12:14:51 -07:00
Leo Yao
aa2b9ef32e
[vcpkg] Added python script to generate all packages file list, added to azur… ()
* Added python script to generate all packages file list, added to azure pipeline to build at the end of the run

* fixed an issue that causes some leading slashes were being removed

* Header Database now includes hpp files for C++ header files

* Changed Header Database to include all files under \include\ folder

* Apply suggestions from code review, changing condition from eq to ne

Co-authored-by: Billy O'Neal <bion@microsoft.com>

* Update last condition from eq to ne

Co-authored-by: Billy O'Neal <bion@microsoft.com>
2020-07-06 13:38:22 -07:00
Billy O'Neal
a28bfe7674
[vcpkg] Remove the tombstones and 'ignore' baseline concepts. ()
This changes our PR builds to treat 'fail' in the ci.baseline.txt as 'skip' instead of using tombstones.

We currently have large numbers of spurious failures that get enshrined in PRs through no fault of a PR author, removing the tombstones concept will fix those by allowing the user to retry. This does mean we accept some risk of not detecting when a port is 'fixed', but that failure is reasonable for us to handle after we see it in CI, but that seems worth it given that it lets us get rid of the tombstone concept.

This also helps out the binary caching feature, because we don't have to figure out how to productize tombstones.
2020-07-02 20:20:07 -07:00
Billy O'Neal
532db1d8f7
[vcpkg] VM Update ()
* [qt5] Add more dependent Linux packages for https://github.com/microsoft/vcpkg/pull/11596/

* [vcpkg] Add LLVM 10 to Windows CI machines for https://github.com/microsoft/vcpkg/pull/11655/

* [vcpkg] Add missing Param block to InstallLLVM function.

* [vcpkg] Add throw after all Write-Error calls for consistency with https://github.com/microsoft/vcpkg/pull/12172
2020-07-01 10:53:10 -07:00
Billy O'Neal
0d37525d75
[vcpkg] Commonize pipelines build powershell and don't use binary caching in CI (rather than PR) builds () 2020-06-25 16:36:28 -07:00
Billy O'Neal
2bf1f30deb
[vcpkg] Use SSH keys instead of password authentication when minting Linux scale sets () 2020-06-21 19:13:13 -07:00
Billy O'Neal
55d6bd1b7d
[vcpkg] [cudnn] [msmpi] [openmpi] Update VMSS ()
Co-authored-by: @JackBoosY 
Co-authored-by: Stefano Sinigardi <stesinigardi@hotmail.com>
Co-authored-by: Voskrese <10104740+Voskrese@users.noreply.github.com>
2020-06-11 11:13:09 -07:00
Billy O'Neal
1d3e985e4f
[vcpkg] Fix OSX CI by ensuring the downloads directory exists ()
* [vcpkg] Remove do-nothing Set-Content from Windows azure-pipelines.yml.
* [vcpkg] Fix OSX CI by ensuring the downloads directory exists in advance, and extract common command line parameters with powershell splatting.
* [tensorflow-cc] Prevent hang building tensorflow-cc asking to configure iOS.
* Skip ignition-msgs5:x64-osx
2020-06-08 22:46:18 -07:00
Alexander Neumann
b4f3e5854a
[CI|gfortran] Install gfortran for OSX and Linux CI ()
* [CI|gfortran] Install gfortran for OSX and Linux CI

* [osx] use cask
2020-06-05 12:13:32 -07:00
Billy O'Neal
3eeeb1b0d3
[vcpkg] Add directories to x-ci-clean lost in merge conflict resolution. () 2020-06-03 22:50:07 -07:00
Billy O'Neal
4fb2256085
[vcpkg] Allow CI to pass in all relevant directories and remove use of symbolic links () 2020-06-03 19:31:28 -07:00
Jack·Boos·Yu
47a883b395
[unixodbc/nanodbc] Fix unixodbc build failure and set it as a dependency on nanodbc () 2020-06-01 15:56:10 -07:00
Alexander Neumann
2c3ffa3b19
[CI|python3] add setuptools () 2020-06-01 11:15:11 -07:00
Billy O'Neal
6d36e2a86b
[vcpkg] Use a crypto RNG to generate admin passwords ()
* [vcpkg] Use a crypto RNG to generate admin passwords

* Apply code review comments from Stephan and Casey.

* Extract functions into create-vmss-helpers.psm1.

* Put Wait-Shutdown back.
2020-05-29 22:46:05 -07:00
Billy O'Neal
d9b4acf02d
[vcpkg] Move CI cleaning back out of the 'ci' command into a separate command to restore cross-compilation preinstalls. () 2020-05-27 18:40:23 -07:00
Jack·Boos·Yu
b271aeffa2
[VCPKG baseline] Fix many ports () 2020-05-25 11:52:48 -07:00
Jack·Boos·Yu
f23ebed76f
[cmake] Update to 3.17.2 ()
* [cmake] Update to 3.17.1

* update cmake directory name

* [cpuid] Fix install headers.

* [stormlib] Fix install headers

* [murmurhash] Fix install headers

* [metrohash] Fix install headers

* update baseline

* update baseline

* [otl] update hash

* update baseline

* [gts] Do not use ninja to avoid empty implib issue

* update baseline

* [dmlc] Re-trigger ci test

* [replxx] Re-trigger ci test

* update baseline

* [osg] Re-trigger ci test

* [replxx] Fix build failure

* Set cmake min version to 3.17.1

* update baseline

* [libnice] Re-trigger ci test

* [mlpack] Re-trigger ci test

* update to 3.17.2

* update cmake hash

* update baseline

* update baseline

* update baseline

* update baseline

* Update scripts/ci.baseline.txt

Co-authored-by: Billy O'Neal <billy.oneal@gmail.com>

* update baseline

* update baseline

* [magnum] Set magnum:arm64-windows to skip in baseline

* [nanogui] Set nanogui:arm64-windows to fail in baseline

* [nettle] Set nettle:x64-windows to fail, waiting for fix this issue in another PR

* re-trigger CI test

* update baseline

* Install unixODBC in Linux

* [nanodbc] Re-trigger CI test

* update baseline

* Remove space

* update baseline
2020-05-19 14:47:19 -07:00
Billy O'Neal
3b0080e3b0
[vcpkg] Harden file removals and clean directory contents in "CI" inside vcpkg itself. ()
`files.h/files.cpp`:
* Add end and else comments to all macros.
* Add "remove_all_inside" function which empties a directory without actually deleting the directory. This is necessary to handle the case where the directory is actually a directory symlink.
* Change remove_all to use std::remove when VCPKG_USE_STD_FILESYSTEM is set; this will engage POSIX delete support available on current Win10.

`commands.ci.cpp`: empty "installed".

`*/initialize_environment.*`: No longer clean the directories outside the tool.

`ci-step.ps1`: Remove unused console output tee-ing.
2020-05-19 13:43:30 -07:00
nicole mazzuca
e806b2ed38
[vcpkg CI] Clean git directory before clone () 2020-05-19 12:23:39 -07:00
Billy O'Neal
5504dfa7da
[vcpkg] Turn on tests and PREfast in CI, and fix tests to pass. ()
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
2020-05-14 12:49:31 -07:00
Billy O'Neal
6920b49047
[vcpkg] [llvm] [mpir] Bump Linux VM memory size and do all operations on the temporary disk. ()
* [vcpkg] Bump Linux VM memory size and do all operations on the temporary disk.

* [llvm] disable optional dependencies to libxml2 and zlib

* [vcpkg] Reduce quotas on the share to something reasonable.

* [mpir] Skip on Linux because it conflicts with libgmp.

Co-authored-by: yurybura <yurybura@gmail.com>
2020-05-10 15:17:27 -07:00
Billy O'Neal
63e1d87432
[vcpkg] Add disk space report on Linux and MacOS. () 2020-05-04 12:12:07 -07:00
Billy Robert O'Neal III
6aab6e0d3d [OpenBLAS] Upgrade to 0.3.9 + a patch to fix AVX512 (which is now present on the VMs) 2020-04-30 21:51:31 -07:00
Billy Robert O'Neal III
b8755728ab [vcpkg] Onboard Linux to VMSS, open 'git' port, and switch back to Azure Spot
* Adds scripts to generate scale sets for testing Linux.
    * Note workaround for https://github.com/microsoft/azure-pipelines-agent/pull/2929
* Switches Windows validation to 'Spot' VMs.
* Opens the git port 9418.
* Removes provisioning of the no longer used 'logs' file share.
* Changes Azure region to 'westus2', which is cheaper.
* Adds +x to all the scripts in scripts/azure-pipelines.
* Use 'xml-results' for all platforms instead of 'raw xml results' on Windows.
2020-04-30 21:51:31 -07:00