* [vcpkg registries] Add git registries support
* Add git registries support to the registries module of vcpkg.
* add e2e tests for git registries
* fix vcpkg.cmake for registries
* fix CRs, remove a thing
* better error messages
* Billy CRs
* fix Robert's CR comment
* I learned about `-c` today
* format
* fix baseline.json
* failing to find baseline is technically not a bug
* [vcpkg] Rewriting CmdLineBuilder (2/n)
I would like, and I think the team would like generally,
to switch to using stuff like `posix_spawn`, as opposed to the
existing use of `system` and `popen`.
This requires a pretty large change to how we use CmdLineBuilder.
The first change we have to make is that the execute functions
_cannot_ take a StringView anymore.
This PR makes that change.
* [vcpkg] Change version field in `baseline.json`
* Change name from `version-tag` to `baseline`
* [vcpkg] x-history serializes version scheme
* Update e2e tests
* Update baseline e2e test
* [vcpkg] Add stuff to the system.process API
* CmdLineBuilder - add rvalue overloads
* Add InWorkingDirectory support
* remove `&&` outside of system.process.cpp
* minor CR
* move the non-InWorkingDirectory functions into the header as inline
* Update toolsrc/include/vcpkg/base/system.process.h
Co-authored-by: Billy O'Neal <bion@microsoft.com>
* fix CmdLineBuilder name
* moar bugfixin
* fix native() location
Co-authored-by: Billy O'Neal <bion@microsoft.com>
* [vcpkg] Add x-ci-verify-versions command
* Code cleanup
* Remove port version splitting from x-history
* Fix wrong message on success
* Parallelize versions file generator
* Use cpu_count()/2 to avoid crashes
* Check db SHA against local SHA
* Check baseline version with x-ci-verify-versions and make baseline generator much faster
* Implement x-add-version to update version db files
* Better checks for x-add-info and make x-ci-verify-versions silent on success
* Use find() instead of [] on maps
* Create version file if does not exist
* Allow redirection of ports/ and port_versions/
* add test ports
* WIP end-to-end tests
* Change pats in e2e tests
* Fix e2e args
* e2e once more
* Pass ersions feature flag to e2e
* Exit with code 1 if there are errors
* Files to test for failure cases
* Update test files
* Add test for x-add-version
* fix redirected ports in last test
* Add CI check (use dummy data)
* Add feature-flags=versions
* Ignore subdirectories inside ports
* Add --verify-git-trees switch
* [vcpkg] Fix build breaks
* [x-ci-verify-versions] PR comments
* [x-add-version] PR comments
* Fix merge conflicts
* Modify tests and pipeline
* Baselines should only have version-string
* Refactor x-add-version
* [vcpkg] Fix help message
* [vcpkg] Fix minor warnings
* `x-add-version --all` doesn't stop on first failure and reduced default verbosity
* [vcpkg] Fix default-baseline
* Load file instead of using paths provider
* Format
* Remove ci test
* Add fish port for testing
* Update version files
* Update fish port to cause SHA discrepancy
* Test for discrepancy between local SHA and declared SHA
* Missing = operator
* Check for error message since x-add-version exits with code 0
* Make x-add-version fail with non-zero exit code if not run with --all
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <ras0219@outlook.com>
* [vcpkg] Teach `vcpkg install` `--no-build-missing`
This switch causes failure to restore from cache to abort the build instead of performing a just-in-time build.
This is useful in complex CI systems where the build was expected to have been performed previously, such as via another pipeline.
* [vcpkg] Rename flag to require binary caching
* [vcpkg registries] support versions
This PR merges the Registries changes and the versioning changes, so that one can use both at the same time.
There is one major difference between this PR and the RFC (#13590), which is that instead of version files looking like:
```json
[
...
]
```
version files look like:
```
{
"versions": [
...
]
}
```
this is to support interop between this PR and existing demos and the like;
fixing this, along with perhaps renaming `port_versions` to `port-versions` should be done after this is merged,
should be a trivial change.
Other changes:
* Changed Strings::replace_all to use more StringView
* Introduced Strings::inplace_replace_all for the common x = replace_all(move(x) pattern and tests
-> Also fixed bug if the search string was empty consuming infinite memory!
* Added many missing {}s in binarycaching.cpp
* [vcpkg] Remove extra indirection in IVersionedPortfileProvider/IBaselineProvider
* [vcpkg] Address CR comment
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Refactor end-to-end tests
* [vcpkg] Cherry-pick x-builtin-ports-root from #14999
* [vcpkg] Move create test from unit tests to e2e
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Improve error messages for incomplete Mono
* [vcpkg] Add mono upgrade notice for Ubuntu 18.04 users
* [vcpkg] Fix always failing to check CMake version due to preferring 'error' constructor in ExpectedS
* [vcpkg] Eagerly display the 'Ubuntu needs new mono' message to avoid users needing to install it twice
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Add semver versioning scheme
* Remove unnecessary code
* Fix SemVer comparison and add sorting test
* Add date scheme
* PR comments
* Use a different column for date and semver schemes.
* Use locale agnostic conversion to long
* Add tests for version scheme change
* Validate version strings before parsing
* Format
* Improve error messages
* PR comments
* PR comments pt. 2
* [vcpkg] Refactor deserializers to reduce duplicate functionality
* [vcpkg] Error on '#' in version strings to avoid confusion with port-version
* [vcpkg] Improve error message
* [vcpkg] Reorder field output
* [vcpkg] Fix tests
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Output versions during install plans
* [vcpkg] Display versions for already-installed packages
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* Check for the 'versions' feature flag while installing a manifest
* Implement two-level fallback for Baselines:
* First, fall back to current repo's baseline.json
* Second, fall back to the ports in ports/
* Optimize checking out baseline.json via `git show`
* Detect presence of '$x-default-baseline' in the manifest and use that as the baseline commit (to be replaced by Federation functionality)
* If port_versions/x-/xyz.json is not found, fall back to ports in ports/
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Add option cmake-args to pass additional args to the script execution
* apply clang format to install.cpp
* rename command to x-cmake-args
* imply editable if x-cmake-args are not empty.
* don't use bitwise or
* add a definition for CMAKE_SCRIPT_ARG
Co-authored-by: wangli28 <wangli28@beyondsoft.com>
* [vcpkg] Add experimental x-azblob binary provider
* [vcpkg] Test azblob storage provider in CI
* [vcpkg] Address some CR comments from #13639
* [vcpkg] Fixup azure-pipelines
* [vcpkg] Fix regression where the downloaded package is purged before decompressing
* [vcpkg] Further refactor vcpkg::Downloads
* [vcpkg] Enable OSX for x-azblob testing
* [vcpkg] Reduce diff against master
* [vcpkg] Extract Downloads::details::split_uri_view
* [vcpkg] Address PR comments
* [vcpkg] Add testing and metrics for x-azblob
* [vcpkg] Add docs for x-azblob
This includes a note that it is currently experimental
* [vcpkg] Address CR comments
* [vcpkg] Revert pipeline changes except OSX to minimize disruption
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* [llvm] install tools in tools/llvm
* remove the giant commented out block of code
* update port-version
* hopefully fix the port?
There are still some issues, namely that the python scripts and DLLs that are copied into `tools/llvm`
are not considered to be "installed by llvm", and thus are not removed when llvm is removed
* format!
* apparently REGEX REPLACE fails if a thing doesn't match
* fix LLVM_REMOVE_EXTENSION_REGEX on windows
* actually read the cmake regex docs...
* fix the name of the variable
* turns out CMAKE_MATCH_1 is the one I want
* need to update VERSION for new policy
* stop removing debug/bin
* fix faulty merge
* [vcpkg] Add disable-lock options
This should fix#14281 at least as a stopgap.
Additionally, adds better errors.
* billy CRs
* change from "disable-lock" to "allow-spurious-lock-failures"
* billy cr
* fix build issues occurring from default bazel config being used because customized bazel config is stored in wrong directory
* [tensorflow-cc] Update CONTROL and ci.baseline.txt
* fix also applies to windows static build
* fix debug and static builds as well as library naming for non-linux
- add patch to fix debug builds
- add patch to fix exports for static linking
- really build debug (instead of cloning release)
- override bazel build options for debug (work around bazel bug)
- bazel doesn't support static libraries: work around by building dynamic library and constructing static linkage commands from build log
- Windows .pdb file can't be >4GB even on x64: work around using reduced debug information
- Windows doesn't support .lib files >4GB even on x64, so split into multiple libs
- vcpkg requires equal amount of libs for debug and release: work around using handcrafted empty dummy libs
- fix naming of libs (.dll on Windows and .dylib on macOS)
* upgrade tensorflow from v1.14 to v2.3
- adapt patch files to tensorflow code changes
- update bazel from v0.25.2 to v3.1
- on Windows use python installed on the host instead of embedded python obtained via vcpkg because embedded python lacks pip, which we need to obtain numpy
- on Windows add MSYS2 to the PATH so that bazel tools can access MSYS2 GIT
- add support for custom CA certificates when using HTTPS_PROXY
* fix execute process macro
The existing implementation totally screwed up commands if the command's arguments contained semicolons (this is the case, e.g., in the FindPython modules of the cmake distribution).
* extend overriden execute_process to more than one COMMAND as there actually are use cases for this
* added another patch required for tensorflow v2.3, fixed path and working directory
* Revert "incorporate changes from microsoft:master"
* Revert "Revert "incorporate changes from microsoft:master""
* final fixes for static build + improving out messages
* enabling linux and osx in CI to see if it works now
* simplified code, fixed version numbers, fixed generated include cmake file
* fix failing postbuild check on handcrafted empty dummy library by spreading the last real libraries contents over the required number of libraries
* remove dead code commit by mistake again
* improvements from code review
* cleaner fix for debug code
* find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3)
* fix error in python helper script
* fix wrong libname in postbuild script
* fix python detection + switch to python on msys2 (instead of embedded python) for Windows as we need numpy
* fix order of arguments
* fix command (it may contain spaces such as C:\Program Files\...)
* revert last commit (root cause for CI failures is something different: there are line breaks in path)
* fix regex comparision
(value needs to be escaped as it may contains regex special characters such as brackets, eg C:/Program Files (x86)/...)
* fix linebreaks in generated file
* fix CRT linkage
(macOS doesn't support static CRT linkage; it's set to dynamic even static target triplets for macOS and linux)
* refactor implemenation to avoid as much code duplication as possible -- algorithmically identical
* fix version numbers in helper scripts
* enable work-around for Windows until bazel fix is available
* install missing python3-pip on linux
* fix linux build by patching
* apply timeout feature now available via merged master branch
* correct linux build patch
* improve debug build patches
(no functional difference because LOG(FATAL, ...) macro internally anyway calls abort(), which the compiler doesn't detect in debug mode...
* improve linux patch
* temporarily add debug to inspect what's going on on macOS CI
* remove temporary debug code and fix static linking scripts for linux and macOS
* fix regex escaping
* fix ambiguous match while grepping for the framework link command
* extend fix of ambiguous match while grepping for the framework link command
* fix what merge of master broke
* fix more what got broken by merging master
(all packages and their dependencies are now maintained manually instead of using pacman...)
* remove "unofficial" from filename
* added switch do distinct classic and manifest mode when generating config.cmake file
* create symlinks for libraries without version number
* fix linux postbuild script
* temporarily disable code making problems
* add note for linking on Linux and macOS
* forget to add README file in previous commit
* add file forgotton in macro fixing patch
* fix python library path
* fix macOS static link command
* update linkage instructions in README
* Update ports/tensorflow-cc/CONTROL
* Update ports/tensorflow-cc/portfile.cmake
* Update scripts/ci.baseline.txt
* use vcpkg_execute_required_process
* pass C_FLAGS and CXX_FLAGS to bazel
* fix INTERFACE_INCLUDE_DIRECTORIES
* fix optional c/cxx arguments
* also add linker opts
* update README
* merge static libs into one
to support force_load (cannot force_load both due to duplicate symbols)
* update README
* quote python path (it might contain spaces that don't get escaped inside outer quotes of bash command)
* fix python path also for static build
* add arm(64) as currently unsupported arch
* bazel 3.7 is available -> remove workaround
* update README, remove necessary c-ares from deps
* update msys package
* add uwp specific options, and minor general improvements
* fix string replace
* fix control file and windows path separator
* revert backslashes-fix -- the root cause was missing .exe extension
* upgrade to tf 2.3.1
* fix hard-coded version
* remove uwp work-in-progress code so that PR can be merged
* update README and print out usage info in portfile
* Update ports/tensorflow-cc/README-linux
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-linux
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-linux
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-linux
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-linux
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-windows
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/portfile.cmake
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/portfile.cmake
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/portfile.cmake
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/portfile.cmake
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/portfile.cmake
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* improve usage hints as discussed in review
* add comment
* apply changes from review
* make additional compiler / linker args space-proof
* Update ports/tensorflow-cc/README-macos
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/tensorflow-cc-config-shared.cmake.in
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/tensorflow-cc-config-shared.cmake.in
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-linux
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* Update ports/tensorflow-cc/README-macos
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* apply changes from code review
* maybe fix the config files
* rob.maynard CRs
* fix windows static lib naming for first part
* Update ports/tensorflow-cc/generate_static_link_cmd_windows.py
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* apply rob maynards 1st suggestion
* nicole CRs
* format/fix-compile
* fix missing string termination
* prefer IMPORTED_LOCATION over IMPORTED_LOCATION_RELEASE to have default fall-back
* hopefully fix the issue where no libraries are generated
* final stuff
Co-authored-by: Gehweiler <Joachim_Gehweiler@McAfee.com>
Co-authored-by: wangli28 <wangli28@beyondsoft.com>
Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com>
Co-authored-by: Joachim Gehweiler <joachim@Joachims-iMac.local>
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
* [vcpkg wip] start implementation of registries pt. 2
* Remove the `ports` field of `VcpkgPaths`
This is an implementation detail, and so we want to make sure that
it's obvious that this is an internal detail; thus, we add a new function
`builtin_ports_directory()`,
which returns the directory where the builtin ports backing store is.
* continue WIP
* wip
* It works!
* format
* fix some issues
* switch from function static to DelayedInit
* fix lexically_normal for experimental::filesystem
* format
* fix missing include
* add STL notice
* whee error handling
* moar error handling!
* ignore extra files in registries
* formatting
* Billy CRs
* Update toolsrc/include/vcpkg/versiont.h
* fix add_filename test
* fix tests
* remove unused function add_filename
* [vcpkg] Add script to generate ports versions history
* [vcpkg] Fix formatting
* Fetch port versions from commit ID
* Use global --x-json switch
* Use --no-checkout when cloning secondary instance
* Clone from local repository instead of from GitHub
* Use CmdLineBuilder to build git commands
* Use CmdLineBuilder and reduce repeated code
* Fetch version at baseline and code cleanup
* Guess version scheme from old CONTROL files
* Rename version db generator script
* Simplify x-history json output
* Use CONTROL/manifest parsers on x-history
* Use git-tree instaed of commit-id
* Remove 'ports' field from root object
* Clean up code
* More code cleanup
* Improve port version detection
* Improve generator logging
* Do not ignore parsing errors in CONTROL files
* PR review comments in Python script
* Fix subprocess.run() calls
* Make `canonicalize()` return error instead of terminating
* [vcpkg] Add tests for new test_parse_control_file paths
* Remove unnecessary std::move() calls
* Fix formatting
* Python formatting
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Add `versions` feature flag and version field manifest parsing
* Introduce FeatureFlagSettings struct to more easily access feature flags throughout the program
* To avoid users accidentally starting to write "version" instead of "version-string" in their manifests, vcpkg explicitly detects and prevents usage of ports with schemes other than "String"
* Drive-by fix of copiable SourceControlFileLocation and an exposed use-after-move bug
This code is largely extracted from PR #13777
Co-authored-by: Victor Romero <romerosanchezv@gmail.com>
* [vcpkg] Address CR Comments. Fix test crash on Linux.
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Victor Romero <romerosanchezv@gmail.com>
* [vcpkg] Split vcpkg/base/json.h into vcpkg/base/jsonreader.h
* [vcpkg] Extract definitions of Configuration-Deserializer (& friends)
These types are only used by VcpkgPaths during the initial parse.
* [vcpkg] Introduce levenshtein-distance suggestions for json errors
* [vcpkg] Fix regression in supports handling
* [vcpkg] Fix signed/unsigned mismatch
* [vcpkg] Address CR comments
* [vcpkg] Address CR comments
* Fix compiler error from merge conflict.
* [vcpkg] Change parameters of Reader::check_for_unexpected_fields to better match declaration
* [vcpkg] Improve errors from features set
* [vcpkg] Fix includes
* [vcpkg] Reuse code
* [vcpkg] Check the "name" field always to maximize error information
* [docs] Improve english phrasing in manifests.md
* [vcpkg] Correct docs link for manifests
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* [vcpkg] Avoid computing triplet ABIs for editable packages
* [vcpkg] Address PR comments
* [vcpkg-end-to-end-tests] Add tests for --no-binarycaching, binarycaching by default, and --editable
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] Enable recursive vcpkg calls
Via envvars VCPKG_COMMAND and VCPKG_X_RECURSIVE_DATA. Child processes can call vcpkg via "$VCPKG_COMMAND <args>" in limited internal circumstances.
* [vcpkg] Address CR comments
* [vcpkg] Do not move through Optional<&> into Optional<T>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* [vcpkg] add env var VCPKG_OVERLAY_TRIPLETS
* shorten way to complicated doc
* run clang format
* Apply CR
Co-authored-by: ras0219 <533828+ras0219@users.noreply.github.com>
* Apply CR2
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
Co-authored-by: ras0219 <533828+ras0219@users.noreply.github.com>
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
* Add compiler info to nuspec description
* Run clang-format on some files
* Fix the unit tests
* [vcpkg] Clarify NuGet description to note 'CXX compiler'
* [vcpkg] Fix tests
Co-authored-by: frivard <frivard@coveo.com>
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
