* Decouple the storage account used for binary caching from the images.
We are being forced to change infrastructure somewhat more frequently than we would like, so change the binary caches to use a shared storage account like we already use for asset caching.
* blob blob blob
* Add quotes
* Avoid name stomping.
* update 2021-07-27
macOS -> 11.5.1, XCode -> 12.5.1
* update azure-pipelines
* update sha of macfuse
* change how macos-ci-base works
* fix build errors
* fix itpp:linux
* more fixes
* remove tab
* allow version changes in all the remove/rename-version patches
* fix libunistring for real
* robert CR
Now that vcpkg has artifact caching, meaning downloading artifacts usually doesn't have to leave the datacenter, there's no need to potentially dirty different runs with leftover downloads from each other.
This resulted in some "impossible" results in https://github.com/microsoft/vcpkg/pull/18687 where a previous PR run accidentially downloaded the Windows version of nodejs to /mnt/vcpkg-ci/downloads/node-v14.17.4-linux-x64.tar.xz
Remove no longer used "unstable" scaffolding.
Use ephemeral OS disks for better latency.
Add ability to change whether temp disks are used or explicitly provisioned disks are used; explicitly provisioned disks turned out to be more expensive than expected before.
Disable Windows Updates in the scale set because the machines are very short lived; working on a mechanism of patching the images instead.
* Cherry-pick https://github.com/microsoft/vcpkg/pull/15598
* Hook deploy-inteloneapi into create-vmss.ps1.
* Add script to resolve https://github.com/microsoft/vcpkg/issues/17521
* Move tls settings deployment to the front and respond to script triggering a reboot.
* Go back to provisioning an extra disk to workaround https://github.com/microsoft/vcpkg/issues/18379
* Disallow public access to blob storage and require TLS 1.2.
* Update Pools.
* Update tool to 2021-06-19
* [simage] Skip simage on uwp platforms as it appears broken by 16.10.
* [tensorflow-cc] Skip because changes in our MacOS hardware broke the port.
* [scripts-audit] vcpkg_acquire_msys (#17687)
* [vcpkg.cmake] Change find_package function -> macro in order to shorten trace (#17997)
switch from function to macro
* [scripts-audit] z_vcpkg_apply_patches (#18013)
* [scripts-audit] z_vcpkg_escape_regex_control_characters (#18013)
* [scripts-audit] vcpkg_fail_port_install (#18054)
Note that this change allows existing failures;
it changes MATCHES to STREQUAL, meaning that
vcpkg_fail_port_install(ON_ARCH "arm") now succeeds on arm64;
this change in behavior is fine (IMO) since it strictly succeeds
when it used to fail.
* [vcpkg] Update vcpkg tool version 2021-05-05
* [vcpkg] Hotfix bug in Microsoft/vcpkg-tool#34
* [vcpkg-ci] Ignore results from non-target triplets
* [protobuf] Fix regression introduced in #16904
* [vcpkg] Fix bug where 'staticcrt' was ignored
Co-authored-by: Robert Schumacher <ras0219@outlook.com>
* start 2021-04-16 process
Major new things:
* update to macOS Big Sur (11.*)
* switch from VirtualBox to Parallels due to ^ (and also ARM)
Minor new things:
* update from xcode CLT 12 to 12.4
This PR includes new stuff for creating the base box for parallels.
Still to do: using the new box type.
* update the vagrantfile stuff
* link the CI to the new version
* modify how macOS boxes are made
the Vagrantfile for the final VM will only download the azure pipeline stuff
this allows us to keep the versions of brew applications stable
* fix cpus and memory
* add vagrant plugins to installables
* Skip cppgraphqlgen ICE.
* [sdformat6] Remove unneeded include(FindBoost) which causes problems when the system cmake version doesn't match the one deployed by vcpkg.
* switch to dmg for installing osxfuse/sshfs
* Set cores to 11 (leaving 1 thread for host)
Co-authored-by: Billy Robert ONeal III <bion@microsoft.com>
* Update to CUDA 11.3 on Windows.
* Update PowerShell to 7.1.3.
* Update CUDA to 11.3 on Linux.
* "Explode" VM provisioning scripts for more consistent feedback during deploy. This resolves the deployment script often hanging with no feedback as to why.
* [libdatachannel] Fix use of deprecated result_type typedef.
Submitted upstream as https://github.com/paullouisageneau/libdatachannel/pull/413
* [libvpx] Get the libvpx outputs from the correct place.
(Perhaps VS2019 version 16.10 moved where these are placed? I've been defensive and left an attempt to find from the old location in place.)
* [chromaprint] Support implementations where lrintf is an intrinsic.
* Add provision-entire-image script.
* [cudnn] Disable download-on-the-fly due to licensing concerns.
* Add libnccl to Linux VMs.
* [wangle] Disable x64-windows due to an ICE.
* [cmake] Update cmake to 3.20.1 to avoid https://gitlab.kitware.com/cmake/cmake/-/issues/21571 race
* [libudis86] Fix passing a bogus working directory which fails on CMake 3.20.x
* [dartsim] Disable unit tests, examples, and tutorials, some of which have CMake authoring errors rejected by 3.20.1.
* Add thrust to the cuda installees.
* [tensorflow] Put .bzl in CURRENT_BUILDTREES_DIR to avoid running out of disk space in CI and to respect --clean-after-build.
* [dimcli] Skip port broken by changes in VS2019 project system.
* [upb] Disable an additional warning.
* [libhv] Fix typo DISABLE_PARALLEL => DISABLE_PARALLEL_CONFIGURE
* Update pools
This reduces bootstrap cost for Windows customers, resolving the issue initially submitted as #12502 .
The `toolsrc` tree was extracted to https://github.com/microsoft/vcpkg-tool. `bootstrap.sh` was changed to download the right source tarball, extract, and build it. This was chosen over the previous attempt, a submodule, over concerns of accidentally destroying people's local modifications.
* [vcpkg] fix checking out git registry ports
* fix the new issues in builtinregistryentry
* fix tests
* fix tests boogaloo
* [vcpkg] Fix issue where baseline is requested for overlay during version resolution
* split BuiltinRegistryEntry into two types
Co-authored-by: Robert Schumacher <ras0219@outlook.com>
* [vcpkg] Miscellaneous internal improvements extracted from #15424
* [vcpkg] CR comments
* [armadillo] Use vcpkg_from_git() to workaround gitlab missing archive
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* Add CI check using dummy version files
* Fix ci test
* Update maintainer's guideline
* Point CI check to actual version files
* Remove --git-verify-trees
* Fix typo
* Fix more typoes
* Even more typos
* Even more typos
* [vcpkg registries] Add git registries support
* Add git registries support to the registries module of vcpkg.
* add e2e tests for git registries
* fix vcpkg.cmake for registries
* fix CRs, remove a thing
* better error messages
* Billy CRs
* fix Robert's CR comment
* I learned about `-c` today
* format
* fix baseline.json
* failing to find baseline is technically not a bug
* [vcpkg] Add x-ci-verify-versions command
* Code cleanup
* Remove port version splitting from x-history
* Fix wrong message on success
* Parallelize versions file generator
* Use cpu_count()/2 to avoid crashes
* Check db SHA against local SHA
* Check baseline version with x-ci-verify-versions and make baseline generator much faster
* Implement x-add-version to update version db files
* Better checks for x-add-info and make x-ci-verify-versions silent on success
* Use find() instead of [] on maps
* Create version file if does not exist
* Allow redirection of ports/ and port_versions/
* add test ports
* WIP end-to-end tests
* Change pats in e2e tests
* Fix e2e args
* e2e once more
* Pass ersions feature flag to e2e
* Exit with code 1 if there are errors
* Files to test for failure cases
* Update test files
* Add test for x-add-version
* fix redirected ports in last test
* Add CI check (use dummy data)
* Add feature-flags=versions
* Ignore subdirectories inside ports
* Add --verify-git-trees switch
* [vcpkg] Fix build breaks
* [x-ci-verify-versions] PR comments
* [x-add-version] PR comments
* Fix merge conflicts
* Modify tests and pipeline
* Baselines should only have version-string
* Refactor x-add-version
* [vcpkg] Fix help message
* [vcpkg] Fix minor warnings
* `x-add-version --all` doesn't stop on first failure and reduced default verbosity
* [vcpkg] Fix default-baseline
* Load file instead of using paths provider
* Format
* Remove ci test
* Add fish port for testing
* Update version files
* Update fish port to cause SHA discrepancy
* Test for discrepancy between local SHA and declared SHA
* Missing = operator
* Check for error message since x-add-version exits with code 0
* Make x-add-version fail with non-zero exit code if not run with --all
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <ras0219@outlook.com>
* [vcpkg] Teach `vcpkg install` `--no-build-missing`
This switch causes failure to restore from cache to abort the build instead of performing a just-in-time build.
This is useful in complex CI systems where the build was expected to have been performed previously, such as via another pipeline.
* [vcpkg] Rename flag to require binary caching
* Add meson from https://github.com/microsoft/vcpkg/pull/12860/
* Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/
* Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/
* Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/
* Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/
* Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774
* Remove no longer necessary apt-mark calls.
* Update nasm on Linux.
* Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share
* Delete no longer necessary azure storage firewall rules.
* Install the newer Windows SDK with the VS installer instead of manually.
* Install the VS2015 and VS2017 compilers.
* Update Powershell-Core to 7.1.0.
* Update source of WDK.
* Update pools.
* [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled.
Fixes:
C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001).
Fixes:
D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error]
try {
^
* [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX.
Note that this port sets /GS-, possibly because it may be for authoring shellcode.
* Bump storage API version to 2020-04-08.
* [vcpkg registries] support versions
This PR merges the Registries changes and the versioning changes, so that one can use both at the same time.
There is one major difference between this PR and the RFC (#13590), which is that instead of version files looking like:
```json
[
...
]
```
version files look like:
```
{
"versions": [
...
]
}
```
this is to support interop between this PR and existing demos and the like;
fixing this, along with perhaps renaming `port_versions` to `port-versions` should be done after this is merged,
should be a trivial change.
* [vcpkg] Refactor end-to-end tests
* [vcpkg] Cherry-pick x-builtin-ports-root from #14999
* [vcpkg] Move create test from unit tests to e2e
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg docs] Change how documenting port functions works
Instead of using `##`, use comment blocks for documentation.
Also, add some minor docs and change RST -> MD
so we actually get docs generated.
* add CI stuff
* regenerate docs
* fix vcpkg_find_acquire_program to not use _execute_process
* [vcpkg] Add experimental x-azblob binary provider
* [vcpkg] Test azblob storage provider in CI
* [vcpkg] Address some CR comments from #13639
* [vcpkg] Fixup azure-pipelines
* [vcpkg] Fix regression where the downloaded package is purged before decompressing
* [vcpkg] Further refactor vcpkg::Downloads
* [vcpkg] Enable OSX for x-azblob testing
* [vcpkg] Reduce diff against master
* [vcpkg] Extract Downloads::details::split_uri_view
* [vcpkg] Address PR comments
* [vcpkg] Add testing and metrics for x-azblob
* [vcpkg] Add docs for x-azblob
This includes a note that it is currently experimental
* [vcpkg] Address CR comments
* [vcpkg] Revert pipeline changes except OSX to minimize disruption
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* [ampl-asl] Separate port ampl-mp module asl to ampl-asl
* [ampl-mp] Update version, separate port ampl-asl and fix arm build
* update baseline
* [VMSS] Pre-install ampl-mp:x86-windows
* Update ports/ampl-asl/portfile.cmake
* [vcpkg] Avoid computing triplet ABIs for editable packages
* [vcpkg] Address PR comments
* [vcpkg-end-to-end-tests] Add tests for --no-binarycaching, binarycaching by default, and --editable
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg osx ci] update to newer bits
these ones should actually work!
* update vagrantfile
* correct scripts, reformat, add docs
* Finish Get-InternalBaseBox, move archives
In order to avoid people maliciously deleting the boxes,
the archives share and file share are in different users now.
* baseline libqcow to be fixed later
* actually check for the mount point
* fix Install-Prerequisites again
* add +x to Get-InternalBaseBox.ps1
additionally, fix some errors
* [vcpkg ci:osx] Remove brew install
* add instructions for creating a new vagrant box
* fix the vagrant scripts for the new box
* finish fixing the setup
* [mecab jxrlib] fix ports for CI
mecab needed to use an actual ref that wasn't master,
and jxrlib needed a patch for xcode 12 CLTs.
Additionally, this fixes the mecab version to be a date, the date of the last commit,
since `1.0` is not the correct version (mecab doesn't have released versions)
* [many ports] fix compile with Xcode 12 CLTs
This mostly means fixing errors on implicit-function-declaration,
and removing some Werrors
* alac-decoder
* apr
* argtable2
* arrow
* hyperscan
* mcpp
* minizip
* mosquitto
* stormlib
* [many ports] even more Xcode 12 CLT fixes
* [jxrlib darknet] fix the last ports! (hopefully)
* CRs, plus minor wip changes to osx scripts
* [vcpkg docs] add docs for manifest files
These are just for the maintainer docs,
not user docs.
* [vcpkg] EBNF-ify platform expression parsing
this modifies nothing about what strings are accepted or rejected,
it just moves stuff around.
also adds tests.
* [vcpkg docs] add manifest mode example
* [wip] docs for augustin
also fix tabs
* [vcpkg manifest] switch to using maps for features
* Apply suggestions from code review
* un-experimentize format-manifest
* flesh out the user manifest mode docs
* CRs
* billy CRs
* final personal pass-thru
* [vcpkg] Add `vcpkg export` to E2E tests. Enable E2E tests on MacOS.
* [vcpkg] Fix export --raw --output-dir=/path/ by changing directory to new export root
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* [vcpkg format-manifest] initial convert-control attempt
TODO: manifest comments! we should keep $directives
* Finalize x-format-manifest
First, fix Json::parse -- "\c", for any c, was incorrectly parsed.
It would emit the escaped character, and then parse the character, so
that `\b` would give you { '\b', 'b' }.
Second, canonicalize source paragraphs as we're parsing them. This found
an error in qt5 -- The `declarative` feature was listed twice, and we
now catch it, so I removed the second paragraph.
Add PlatformExpression::complexity to allow ordering platform
expressions in a somewhat reasonable way.
Notes:
- We allow `all_modules` as a feature name for back-compat with
paraview
- In order to actually convert CONTROL to vcpkg.json, we'd need to
rename the qt5 `default` feature.
- We need to add support for $directives in x-format-manifest
* fix qt5 port
* format
* fix compile
* fix tests for canonicalization
* Clean up code
* add error message for nothing to format
* add extra_info field
* add `const X&` overloads for `Object::insert[_or_replace]`
* fix compile
* simple CRs
* add tests
* format
* Fix mosquitto port file
also unmerge a line
* fail the tests on malformed manifest
* fix format_all
* fix coroutine port-version
* format manifests
* [cmake] add port for qt testing
* only add cmcldeps on windows
* [cmake] cleanup and fix portfile for osx build
* [cmake] more cleanup
* move cmake to scripts/test_ports
* add the overlay scripts/test_ports to ci
* [vcpkg-ci] fix macos setup scripts
* the disk feature is still experimental
* fix Get-InstalledVirtualBoxExtensionPacks
* use new bash install script for brew
instead of old-style ruby
* [libb2] Add supports and install pkg-conifg on macOS VM
* Add blank line to the end of the json file
* vcpkg x-format-manifest --all
* Remove failed ports in ci baseline file
* [vcpkg] Move default binary cache from `$root/archives` to user-wide directory
Teach nuget/nugetconfig binary sources to use read/write.
* [vcpkg] Reformat
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
This PR also renames the VcpkgUserTriplet MSBuild variable to VcpkgTriplet to minimize user confusion compared to previous practice and documentation.
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* stage checking formatting
this means that the port CI won't run if the formatting failed
* fix invalid names
* add formatting of manifests
* fix dependsOn name
* fix a thing
* CRs
* oops, typo
* Added python script to generate all packages file list, added to azure pipeline to build at the end of the run
* fixed an issue that causes some leading slashes were being removed
* Header Database now includes hpp files for C++ header files
* Changed Header Database to include all files under \include\ folder
* Apply suggestions from code review, changing condition from eq to ne
Co-authored-by: Billy O'Neal <bion@microsoft.com>
* Update last condition from eq to ne
Co-authored-by: Billy O'Neal <bion@microsoft.com>
This changes our PR builds to treat 'fail' in the ci.baseline.txt as 'skip' instead of using tombstones.
We currently have large numbers of spurious failures that get enshrined in PRs through no fault of a PR author, removing the tombstones concept will fix those by allowing the user to retry. This does mean we accept some risk of not detecting when a port is 'fixed', but that failure is reasonable for us to handle after we see it in CI, but that seems worth it given that it lets us get rid of the tombstone concept.
This also helps out the binary caching feature, because we don't have to figure out how to productize tombstones.
* [vcpkg] Implement --x-write-nuget-packages-config= setting for `install` and `x-set-installed`.
* [vcpkg] Add end-to-end testing suite for install, remove, and binary caching
* [vcpkg] Define `$TestingRoot in end-to-end-tests.ps1
* [vcpkg] Address CR comments
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Add disk space report to PR/CI
Example output:
```
Disk Label Size Free Space
---- ----- ---- ----------
C: Sabrent 1907 GiB 1239 GiB
D: Dev 447 GiB 383 GiB
E: Samsung 960 Pro 1908 GiB 1084 GiB
H: Rocket 3815 GiB 863 GiB
R: 0 B 0 B
S: 0 B 0 B
```
* [vcpkg] Remove do-nothing Set-Content from Windows azure-pipelines.yml.
* [vcpkg] Fix OSX CI by ensuring the downloads directory exists in advance, and extract common command line parameters with powershell splatting.
* [tensorflow-cc] Prevent hang building tensorflow-cc asking to configure iOS.
* Skip ignition-msgs5:x64-osx
* [vcpkg] Use a crypto RNG to generate admin passwords
* Apply code review comments from Stephan and Casey.
* Extract functions into create-vmss-helpers.psm1.
* Put Wait-Shutdown back.
* [vcpkg] Remove powershell from the 'run vcpkg ci' path to reduce hangs from msys components.
* The BatchScript task uses filename rather than filePath
* cmd hates BOMs
* cmd hates forward slashes
* [cmake] Update to 3.17.1
* update cmake directory name
* [cpuid] Fix install headers.
* [stormlib] Fix install headers
* [murmurhash] Fix install headers
* [metrohash] Fix install headers
* update baseline
* update baseline
* [otl] update hash
* update baseline
* [gts] Do not use ninja to avoid empty implib issue
* update baseline
* [dmlc] Re-trigger ci test
* [replxx] Re-trigger ci test
* update baseline
* [osg] Re-trigger ci test
* [replxx] Fix build failure
* Set cmake min version to 3.17.1
* update baseline
* [libnice] Re-trigger ci test
* [mlpack] Re-trigger ci test
* update to 3.17.2
* update cmake hash
* update baseline
* update baseline
* update baseline
* update baseline
* Update scripts/ci.baseline.txt
Co-authored-by: Billy O'Neal <billy.oneal@gmail.com>
* update baseline
* update baseline
* [magnum] Set magnum:arm64-windows to skip in baseline
* [nanogui] Set nanogui:arm64-windows to fail in baseline
* [nettle] Set nettle:x64-windows to fail, waiting for fix this issue in another PR
* re-trigger CI test
* update baseline
* Install unixODBC in Linux
* [nanodbc] Re-trigger CI test
* update baseline
* Remove space
* update baseline
`files.h/files.cpp`:
* Add end and else comments to all macros.
* Add "remove_all_inside" function which empties a directory without actually deleting the directory. This is necessary to handle the case where the directory is actually a directory symlink.
* Change remove_all to use std::remove when VCPKG_USE_STD_FILESYSTEM is set; this will engage POSIX delete support available on current Win10.
`commands.ci.cpp`: empty "installed".
`*/initialize_environment.*`: No longer clean the directories outside the tool.
`ci-step.ps1`: Remove unused console output tee-ing.
* [vcpkg] Bump Linux VM memory size and do all operations on the temporary disk.
* [llvm] disable optional dependencies to libxml2 and zlib
* [vcpkg] Reduce quotas on the share to something reasonable.
* [mpir] Skip on Linux because it conflicts with libgmp.
Co-authored-by: yurybura <yurybura@gmail.com>
* Adds scripts to generate scale sets for testing Linux.
* Note workaround for https://github.com/microsoft/azure-pipelines-agent/pull/2929
* Switches Windows validation to 'Spot' VMs.
* Opens the git port 9418.
* Removes provisioning of the no longer used 'logs' file share.
* Changes Azure region to 'westus2', which is cheaper.
* Adds +x to all the scripts in scripts/azure-pipelines.
* Use 'xml-results' for all platforms instead of 'raw xml results' on Windows.
Included is a script that sets up all vcpkg's Azure infrastructure for Windows PR tests, and several updates to baselines. The baseline updates are generally caused by an updated copy of the MSVC++ compiler caused by updating the VMs, but some are caused by missed failures only detected now because this did a cleared out archives directory first.
Some of the build infrastructure isn't what I'd call 'pretty' (e.g. we're split into more scripts and such than I'd like) but this mirrors how our existing PR system works.
It is expected that the existing vcpkg Windows PR system will hate these baseline updates so we'll need to merge this, then remove that (duplicate) workflow immediately afterwards, then delete all the Windows VMs powering the old infrastructure.
