Commit Graph

82 Commits

Author SHA1 Message Date
Billy O'Neal
4e65ce52ab
Sort file lists and failure logs together. (#13067) 2020-08-22 22:23:59 -07:00
ras0219
481738beae
[vcpkg] Add vcpkg export to E2E tests. Enable E2E tests on all platforms. (#12198)
* [vcpkg] Add `vcpkg export` to E2E tests. Enable E2E tests on MacOS.

* [vcpkg] Fix export --raw --output-dir=/path/ by changing directory to new export root

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
2020-08-10 10:22:51 -07:00
nicole mazzuca
1c2af99415
[vcpkg format-manifest] Add convert-control flag (#12471)
* [vcpkg format-manifest] initial convert-control attempt

TODO: manifest comments! we should keep $directives

* Finalize x-format-manifest

First, fix Json::parse -- "\c", for any c, was incorrectly parsed.
It would emit the escaped character, and then parse the character, so
that `\b` would give you { '\b', 'b' }.

Second, canonicalize source paragraphs as we're parsing them. This found
an error in qt5 -- The `declarative` feature was listed twice, and we
now catch it, so I removed the second paragraph.

Add PlatformExpression::complexity to allow ordering platform
expressions in a somewhat reasonable way.

Notes:
  - We allow `all_modules` as a feature name for back-compat with
    paraview
  - In order to actually convert CONTROL to vcpkg.json, we'd need to
    rename the qt5 `default` feature.
  - We need to add support for $directives in x-format-manifest

* fix qt5 port

* format

* fix compile

* fix tests for canonicalization

* Clean up code

* add error message for nothing to format

* add extra_info field

* add `const X&` overloads for `Object::insert[_or_replace]`

* fix compile

* simple CRs

* add tests

* format

* Fix mosquitto port file

also unmerge a line

* fail the tests on malformed manifest

* fix format_all

* fix coroutine port-version

* format manifests
2020-08-02 10:08:07 -07:00
Jack·Boos·Yu
d367c4c0b8
[vcpkg] Improve format check failure message (#12460)
* [docs] Add format document

* improve the error message

* Check the error message

* Finish test

* restore file list
2020-07-29 21:12:06 -07:00
ras0219
bc07f4b052
[vcpkg-ci] Always publish file lists (#12384)
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
2020-07-13 12:14:51 -07:00
nicole mazzuca
9606917c81
[vcpkg ci] Update formatting CI (#12314)
* stage checking formatting

this means that the port CI won't run if the formatting failed

* fix invalid names

* add formatting of manifests

* fix dependsOn name

* fix a thing

* CRs

* oops, typo
2020-07-07 14:09:33 -07:00
nicole mazzuca
6a41626eaf
[vcpkg] Format the C++ in CI (#11655)
* [vcpkg] Format the C++ in the CI

* format the C++

* CR
2020-07-06 16:45:34 -07:00
Leo Yao
aa2b9ef32e
[vcpkg] Added python script to generate all packages file list, added to azur… (#12177)
* Added python script to generate all packages file list, added to azure pipeline to build at the end of the run

* fixed an issue that causes some leading slashes were being removed

* Header Database now includes hpp files for C++ header files

* Changed Header Database to include all files under \include\ folder

* Apply suggestions from code review, changing condition from eq to ne

Co-authored-by: Billy O'Neal <bion@microsoft.com>

* Update last condition from eq to ne

Co-authored-by: Billy O'Neal <bion@microsoft.com>
2020-07-06 13:38:22 -07:00
Billy O'Neal
a28bfe7674
[vcpkg] Remove the tombstones and 'ignore' baseline concepts. (#12197)
This changes our PR builds to treat 'fail' in the ci.baseline.txt as 'skip' instead of using tombstones.

We currently have large numbers of spurious failures that get enshrined in PRs through no fault of a PR author, removing the tombstones concept will fix those by allowing the user to retry. This does mean we accept some risk of not detecting when a port is 'fixed', but that failure is reasonable for us to handle after we see it in CI, but that seems worth it given that it lets us get rid of the tombstone concept.

This also helps out the binary caching feature, because we don't have to figure out how to productize tombstones.
2020-07-02 20:20:07 -07:00
ras0219
135f91de1e
[vcpkg] Implement --x-write-nuget-packages-config= setting for install and x-set-installed (#12138)
* [vcpkg] Implement --x-write-nuget-packages-config= setting for `install` and `x-set-installed`.

* [vcpkg] Add end-to-end testing suite for install, remove, and binary caching

* [vcpkg] Define `$TestingRoot in end-to-end-tests.ps1

* [vcpkg] Address CR comments

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
2020-07-01 11:36:09 -07:00
Billy O'Neal
532db1d8f7
[vcpkg] VM Update (#12176)
* [qt5] Add more dependent Linux packages for https://github.com/microsoft/vcpkg/pull/11596/

* [vcpkg] Add LLVM 10 to Windows CI machines for https://github.com/microsoft/vcpkg/pull/11655/

* [vcpkg] Add missing Param block to InstallLLVM function.

* [vcpkg] Add throw after all Write-Error calls for consistency with https://github.com/microsoft/vcpkg/pull/12172
2020-07-01 10:53:10 -07:00
Billy O'Neal
f10c49281a
[vcpkg] Add disk space report to Windows PR/CI (#12113)
* [vcpkg] Add disk space report to PR/CI

Example output:
```
Disk Label           Size     Free Space
---- -----           ----     ----------
C:   Sabrent         1907 GiB 1239 GiB
D:   Dev             447 GiB  383 GiB
E:   Samsung 960 Pro 1908 GiB 1084 GiB
H:   Rocket          3815 GiB 863 GiB
R:                   0 B      0 B
S:                   0 B      0 B
```
2020-06-25 16:38:13 -07:00
Billy O'Neal
0d37525d75
[vcpkg] Commonize pipelines build powershell and don't use binary caching in CI (rather than PR) builds (#12082) 2020-06-25 16:36:28 -07:00
Billy O'Neal
2bf1f30deb
[vcpkg] Use SSH keys instead of password authentication when minting Linux scale sets (#11999) 2020-06-21 19:13:13 -07:00
Billy O'Neal
6a8d2cb4a1
Attempt to mitigate Windows Defender detections by making provision-image.ps1 not a powershell script (#11978)
* Remove C:\ exclusion since we do all the interesting work on D: now.

* Rename provision-image.ps1 to remove executable extension.
2020-06-17 10:08:30 -07:00
Billy O'Neal
be298744f7
[vcpkg] Merge unit test pass into x86-windows. (#11858) 2020-06-11 17:49:22 -07:00
Billy O'Neal
55d6bd1b7d
[vcpkg] [cudnn] [msmpi] [openmpi] Update VMSS (#11365)
Co-authored-by: @JackBoosY 
Co-authored-by: Stefano Sinigardi <stesinigardi@hotmail.com>
Co-authored-by: Voskrese <10104740+Voskrese@users.noreply.github.com>
2020-06-11 11:13:09 -07:00
Billy O'Neal
1d3e985e4f
[vcpkg] Fix OSX CI by ensuring the downloads directory exists (#11839)
* [vcpkg] Remove do-nothing Set-Content from Windows azure-pipelines.yml.
* [vcpkg] Fix OSX CI by ensuring the downloads directory exists in advance, and extract common command line parameters with powershell splatting.
* [tensorflow-cc] Prevent hang building tensorflow-cc asking to configure iOS.
* Skip ignition-msgs5:x64-osx
2020-06-08 22:46:18 -07:00
Billy O'Neal
4fb2256085
[vcpkg] Allow CI to pass in all relevant directories and remove use of symbolic links (#11483) 2020-06-03 19:31:28 -07:00
Billy O'Neal
6d36e2a86b
[vcpkg] Use a crypto RNG to generate admin passwords (#11629)
* [vcpkg] Use a crypto RNG to generate admin passwords

* Apply code review comments from Stephan and Casey.

* Extract functions into create-vmss-helpers.psm1.

* Put Wait-Shutdown back.
2020-05-29 22:46:05 -07:00
Billy O'Neal
d9b4acf02d
[vcpkg] Move CI cleaning back out of the 'ci' command into a separate command to restore cross-compilation preinstalls. (#11545) 2020-05-27 18:40:23 -07:00
Billy O'Neal
f2c46e717d
[vcpkg] Remove powershell from the 'run vcpkg ci' path to reduce hangs from msys components. (#11512)
* [vcpkg] Remove powershell from the 'run vcpkg ci' path to reduce hangs from msys components.

* The BatchScript task uses filename rather than filePath

* cmd hates BOMs

* cmd hates forward slashes
2020-05-21 20:48:05 -07:00
Billy O'Neal
9b4535e7ee
Fix CMake PATH that fails Windows tests. (#11453) 2020-05-19 17:02:19 -07:00
Martin Vejdarski
ec2b59b93e
[harfbuzz,skia] Update and replace Skia dependencies with vcpkg (#11416) 2020-05-19 15:20:18 -07:00
Billy O'Neal
3b0080e3b0
[vcpkg] Harden file removals and clean directory contents in "CI" inside vcpkg itself. (#11432)
`files.h/files.cpp`:
* Add end and else comments to all macros.
* Add "remove_all_inside" function which empties a directory without actually deleting the directory. This is necessary to handle the case where the directory is actually a directory symlink.
* Change remove_all to use std::remove when VCPKG_USE_STD_FILESYSTEM is set; this will engage POSIX delete support available on current Win10.

`commands.ci.cpp`: empty "installed".

`*/initialize_environment.*`: No longer clean the directories outside the tool.

`ci-step.ps1`: Remove unused console output tee-ing.
2020-05-19 13:43:30 -07:00
nicole mazzuca
e806b2ed38
[vcpkg CI] Clean git directory before clone (#11450) 2020-05-19 12:23:39 -07:00
Billy O'Neal
5504dfa7da
[vcpkg] Turn on tests and PREfast in CI, and fix tests to pass. (#11239)
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
2020-05-14 12:49:31 -07:00
Billy O'Neal
6920b49047
[vcpkg] [llvm] [mpir] Bump Linux VM memory size and do all operations on the temporary disk. (#11174)
* [vcpkg] Bump Linux VM memory size and do all operations on the temporary disk.

* [llvm] disable optional dependencies to libxml2 and zlib

* [vcpkg] Reduce quotas on the share to something reasonable.

* [mpir] Skip on Linux because it conflicts with libgmp.

Co-authored-by: yurybura <yurybura@gmail.com>
2020-05-10 15:17:27 -07:00
Billy Robert O'Neal III
b8755728ab [vcpkg] Onboard Linux to VMSS, open 'git' port, and switch back to Azure Spot
* Adds scripts to generate scale sets for testing Linux.
    * Note workaround for https://github.com/microsoft/azure-pipelines-agent/pull/2929
* Switches Windows validation to 'Spot' VMs.
* Opens the git port 9418.
* Removes provisioning of the no longer used 'logs' file share.
* Changes Azure region to 'westus2', which is cheaper.
* Adds +x to all the scripts in scripts/azure-pipelines.
* Use 'xml-results' for all platforms instead of 'raw xml results' on Windows.
2020-04-30 21:51:31 -07:00
Billy O'Neal
7341c295b6
[vcpkg] Copy macos pipelines into azure-pipelines.yml 2020-04-28 13:02:39 -07:00
Billy Robert O'Neal III
9256735ad5 WIP 2020-04-22 02:47:03 -07:00
Billy Robert O'Neal III
070a18974b Change supporting infrastructure to use Azure Virtual Machine Scale Sets for vcpkg's PR builds, which should both improve our PR build times and reduce Azure spending by shutting down machines when they aren't being used.
Included is a script that sets up all vcpkg's Azure infrastructure for Windows PR tests, and several updates to baselines. The baseline updates are generally caused by an updated copy of the MSVC++ compiler caused by updating the VMs, but some are caused by missed failures only detected now because this did a cleared out archives directory first.

Some of the build infrastructure isn't what I'd call 'pretty' (e.g. we're split into more scripts and such than I'd like) but this mirrors how our existing PR system works.

It is expected that the existing vcpkg Windows PR system will hate these baseline updates so we'll need to merge this, then remove that (duplicate) workflow immediately afterwards, then delete all the Windows VMs powering the old infrastructure.
2020-04-21 17:12:21 -07:00