* Add the ability to do full rebuilds with updated copies of vcpkg-tool.
Resurrection of https://github.com/microsoft/vcpkg/pull/23753/
* Extract all scripts out of inline blocks.
* Replicate all changes to the "unstable" testing.
* Fix bootstrap :)
* Fix version files validation path.
* Malformed.
* Paths and permissions.
* Fixed file name typo
* Update scripts/azure-pipelines/bootstrap-from-source.sh
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* Install Ubuntu nasm package rather than building from source; 2.14 which is available in Ubuntu 20.04 is sufficient for intel-ipsec.
* Add cmake and ninja.
* Update CUDA signing key.
* Update pwsh to 7.2.3
* Remove clean downloads step obsoleted since we ripped out pacman.
* Note that haskell-stack is for bond.
* Cherry pick from https://github.com/microsoft/vcpkg/pull/23569 : add cuda-nvml-dev-11-6
* Update pools.
* Baseline updates
PASSING, REMOVE FROM FAIL LIST: mesa:x64-windows-static-md (C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt).
:)
```
REGRESSION: cppgraphqlgen:arm64-windows failed with BUILD_FAILED. If expected, add cppgraphqlgen:arm64-windows=fail to C:\a\1\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: cppgraphqlgen:x64-windows failed with BUILD_FAILED. If expected, add cppgraphqlgen:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: cppgraphqlgen:x64-windows-static failed with BUILD_FAILED. If expected, add cppgraphqlgen:x64-windows-static=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: cppgraphqlgen:x64-windows-static-md failed with BUILD_FAILED. If expected, add cppgraphqlgen:x64-windows-static-md=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: cppgraphqlgen:x86-windows failed with BUILD_FAILED. If expected, add cppgraphqlgen:x86-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
```
This is a compiler behavior change or bug in VS 2022 16.2:
```
C:\PROGRA~1\MICROS~1\2022\ENTERP~1\VC\Tools\MSVC\1432~1.313\bin\Hostx64\x86\cl.exe /TP -DGRAPHQL_DLLEXPORTS -DIMPL_GRAPHQLSERVICE_DLL -Dgraphqlservice_EXPORTS -ID:\buildtrees\cppgraphqlgen\src\v4.3.1-9d04ffd723.clean\src\..\include -ID:\buildtrees\cppgraphqlgen\src\v4.3.1-9d04ffd723.clean\src\..\PEGTL\include -external:ID:\installed\x86-windows\include -external:W0 /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 -MDd /W4 /WX /permissive- -std:c++20 /showIncludes /Fosrc\CMakeFiles\graphqlservice.dir\GraphQLService.cpp.obj /Fdsrc\CMakeFiles\graphqlservice.dir\ /FS -c D:\buildtrees\cppgraphqlgen\src\v4.3.1-9d04ffd723.clean\src\GraphQLService.cpp
cl : Command line warning D9025 : overriding '/W3' with '/W4'
D:\installed\x86-windows\include\tao\pegtl/demangle.hpp(147): error C2338: static_assert failed: 'internal::dependent_true< T > && ( begin != std::string_view::npos )'
D:\buildtrees\cppgraphqlgen\src\v4.3.1-9d04ffd723.clean\include\graphqlservice/internal/SyntaxTree.h(120): note: see reference to function template instantiation 'std::string_view tao::graphqlpeg::demangle<graphql::peg::variable_value>(void) noexcept' being compiled
D:\buildtrees\cppgraphqlgen\src\v4.3.1-9d04ffd723.clean\include\graphqlservice/internal/SyntaxTree.h(53): note: see reference to function template instantiation 'std::string_view graphql::peg::ast_node::type_name<graphql::peg::variable_value>(void) noexcept' being compiled
D:\buildtrees\cppgraphqlgen\src\v4.3.1-9d04ffd723.clean\src\GraphQLService.cpp(310): note: see reference to function template instantiation 'bool graphql::peg::ast_node::is_type<graphql::peg::variable_value>(void) noexcept const' being compiled
ninja: build stopped: subcommand failed.
```
This is a compiler behavior change in 17.2. @wravery do you have comments?
```
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\1\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\1\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: qtwebengine:x64-windows failed with BUILD_FAILED. If expected, add qtwebengine:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
```
qtwebengine doesn't report logs, but I'm pretty sure it's this ICE: https://developercommunity.visualstudio.com/t/Visual-Studio-2022-v1720-reports-fata/10039296 @Neumann-A
```
REGRESSION: rsocket:x64-windows failed with BUILD_FAILED. If expected, add rsocket:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: rsocket:x64-windows-static failed with BUILD_FAILED. If expected, add rsocket:x64-windows-static=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
REGRESSION: rsocket:x64-windows-static-md failed with BUILD_FAILED. If expected, add rsocket:x64-windows-static-md=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt.
```
ICE :( https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/1490389
Other changes:
* Removed qt5-webengine skips because they are now skipped by a "supports" clause.
* Removed ctp skips because asset caching exists.
* [stxxl] Guard definition of log2 for current MSVCs.
* Update Linux VM CUDA to 11.6.
* Update Intel OneAPI
* Update Visual Studio to 2022, and remove deployment of v140 and v141 compilers.
* Update CUDA to 11.6 on Windows.
* Update pools.
* cix ci.baseline.txt
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Add triplet to name of main CI task
* Fix osx bootstrap task name
* Hide output from switch to parent hash
* Bump test port version
* Hardcode osx+linux triplet names
* Improve logging order
* Add error handling to each tool download
* Revert "Bump test port version"
* Don't prefetch 7zip
* Revert "incorporate changes from microsoft:master"
* Revert "Revert "incorporate changes from microsoft:master""
* preliminary approach to solve pip issue
* fix python venv
* fix macOS build command again
* bump version
* x-add-version
* Fix spelling of prerequisites.
* Add python3-venv
* Update linux pool.
Co-authored-by: Joachim Gehweiler <joachim@Joachims-iMac.local>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* Decouple the storage account used for binary caching from the images.
We are being forced to change infrastructure somewhat more frequently than we would like, so change the binary caches to use a shared storage account like we already use for asset caching.
* blob blob blob
* Add quotes
* Avoid name stomping.
Now that vcpkg has artifact caching, meaning downloading artifacts usually doesn't have to leave the datacenter, there's no need to potentially dirty different runs with leftover downloads from each other.
This resulted in some "impossible" results in https://github.com/microsoft/vcpkg/pull/18687 where a previous PR run accidentially downloaded the Windows version of nodejs to /mnt/vcpkg-ci/downloads/node-v14.17.4-linux-x64.tar.xz
Remove no longer used "unstable" scaffolding.
Use ephemeral OS disks for better latency.
Add ability to change whether temp disks are used or explicitly provisioned disks are used; explicitly provisioned disks turned out to be more expensive than expected before.
Disable Windows Updates in the scale set because the machines are very short lived; working on a mechanism of patching the images instead.
* Cherry-pick https://github.com/microsoft/vcpkg/pull/15598
* Hook deploy-inteloneapi into create-vmss.ps1.
* Add script to resolve https://github.com/microsoft/vcpkg/issues/17521
* Move tls settings deployment to the front and respond to script triggering a reboot.
* Go back to provisioning an extra disk to workaround https://github.com/microsoft/vcpkg/issues/18379
* Disallow public access to blob storage and require TLS 1.2.
* Update Pools.
* Update tool to 2021-06-19
* [simage] Skip simage on uwp platforms as it appears broken by 16.10.
* [tensorflow-cc] Skip because changes in our MacOS hardware broke the port.
* Update to CUDA 11.3 on Windows.
* Update PowerShell to 7.1.3.
* Update CUDA to 11.3 on Linux.
* "Explode" VM provisioning scripts for more consistent feedback during deploy. This resolves the deployment script often hanging with no feedback as to why.
* [libdatachannel] Fix use of deprecated result_type typedef.
Submitted upstream as https://github.com/paullouisageneau/libdatachannel/pull/413
* [libvpx] Get the libvpx outputs from the correct place.
(Perhaps VS2019 version 16.10 moved where these are placed? I've been defensive and left an attempt to find from the old location in place.)
* [chromaprint] Support implementations where lrintf is an intrinsic.
* Add provision-entire-image script.
* [cudnn] Disable download-on-the-fly due to licensing concerns.
* Add libnccl to Linux VMs.
* [wangle] Disable x64-windows due to an ICE.
* [cmake] Update cmake to 3.20.1 to avoid https://gitlab.kitware.com/cmake/cmake/-/issues/21571 race
* [libudis86] Fix passing a bogus working directory which fails on CMake 3.20.x
* [dartsim] Disable unit tests, examples, and tutorials, some of which have CMake authoring errors rejected by 3.20.1.
* Add thrust to the cuda installees.
* [tensorflow] Put .bzl in CURRENT_BUILDTREES_DIR to avoid running out of disk space in CI and to respect --clean-after-build.
* [dimcli] Skip port broken by changes in VS2019 project system.
* [upb] Disable an additional warning.
* [libhv] Fix typo DISABLE_PARALLEL => DISABLE_PARALLEL_CONFIGURE
* Update pools
This reduces bootstrap cost for Windows customers, resolving the issue initially submitted as #12502 .
The `toolsrc` tree was extracted to https://github.com/microsoft/vcpkg-tool. `bootstrap.sh` was changed to download the right source tarball, extract, and build it. This was chosen over the previous attempt, a submodule, over concerns of accidentally destroying people's local modifications.
* [vcpkg] Miscellaneous internal improvements extracted from #15424
* [vcpkg] CR comments
* [armadillo] Use vcpkg_from_git() to workaround gitlab missing archive
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* Add meson from https://github.com/microsoft/vcpkg/pull/12860/
* Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/
* Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/
* Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/
* Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/
* Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774
* Remove no longer necessary apt-mark calls.
* Update nasm on Linux.
* Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share
* Delete no longer necessary azure storage firewall rules.
* Install the newer Windows SDK with the VS installer instead of manually.
* Install the VS2015 and VS2017 compilers.
* Update Powershell-Core to 7.1.0.
* Update source of WDK.
* Update pools.
* [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled.
Fixes:
C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001).
Fixes:
D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error]
try {
^
* [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX.
Note that this port sets /GS-, possibly because it may be for authoring shellcode.
* Bump storage API version to 2020-04-08.
* [vcpkg] Add `vcpkg export` to E2E tests. Enable E2E tests on MacOS.
* [vcpkg] Fix export --raw --output-dir=/path/ by changing directory to new export root
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* Added python script to generate all packages file list, added to azure pipeline to build at the end of the run
* fixed an issue that causes some leading slashes were being removed
* Header Database now includes hpp files for C++ header files
* Changed Header Database to include all files under \include\ folder
* Apply suggestions from code review, changing condition from eq to ne
Co-authored-by: Billy O'Neal <bion@microsoft.com>
* Update last condition from eq to ne
Co-authored-by: Billy O'Neal <bion@microsoft.com>
This changes our PR builds to treat 'fail' in the ci.baseline.txt as 'skip' instead of using tombstones.
We currently have large numbers of spurious failures that get enshrined in PRs through no fault of a PR author, removing the tombstones concept will fix those by allowing the user to retry. This does mean we accept some risk of not detecting when a port is 'fixed', but that failure is reasonable for us to handle after we see it in CI, but that seems worth it given that it lets us get rid of the tombstone concept.
This also helps out the binary caching feature, because we don't have to figure out how to productize tombstones.
* [vcpkg] Remove do-nothing Set-Content from Windows azure-pipelines.yml.
* [vcpkg] Fix OSX CI by ensuring the downloads directory exists in advance, and extract common command line parameters with powershell splatting.
* [tensorflow-cc] Prevent hang building tensorflow-cc asking to configure iOS.
* Skip ignition-msgs5:x64-osx
* [vcpkg] Use a crypto RNG to generate admin passwords
* Apply code review comments from Stephan and Casey.
* Extract functions into create-vmss-helpers.psm1.
* Put Wait-Shutdown back.
* [cmake] Update to 3.17.1
* update cmake directory name
* [cpuid] Fix install headers.
* [stormlib] Fix install headers
* [murmurhash] Fix install headers
* [metrohash] Fix install headers
* update baseline
* update baseline
* [otl] update hash
* update baseline
* [gts] Do not use ninja to avoid empty implib issue
* update baseline
* [dmlc] Re-trigger ci test
* [replxx] Re-trigger ci test
* update baseline
* [osg] Re-trigger ci test
* [replxx] Fix build failure
* Set cmake min version to 3.17.1
* update baseline
* [libnice] Re-trigger ci test
* [mlpack] Re-trigger ci test
* update to 3.17.2
* update cmake hash
* update baseline
* update baseline
* update baseline
* update baseline
* Update scripts/ci.baseline.txt
Co-authored-by: Billy O'Neal <billy.oneal@gmail.com>
* update baseline
* update baseline
* [magnum] Set magnum:arm64-windows to skip in baseline
* [nanogui] Set nanogui:arm64-windows to fail in baseline
* [nettle] Set nettle:x64-windows to fail, waiting for fix this issue in another PR
* re-trigger CI test
* update baseline
* Install unixODBC in Linux
* [nanodbc] Re-trigger CI test
* update baseline
* Remove space
* update baseline
`files.h/files.cpp`:
* Add end and else comments to all macros.
* Add "remove_all_inside" function which empties a directory without actually deleting the directory. This is necessary to handle the case where the directory is actually a directory symlink.
* Change remove_all to use std::remove when VCPKG_USE_STD_FILESYSTEM is set; this will engage POSIX delete support available on current Win10.
`commands.ci.cpp`: empty "installed".
`*/initialize_environment.*`: No longer clean the directories outside the tool.
`ci-step.ps1`: Remove unused console output tee-ing.
