vcpkg/ports/tensorflow-common/fix-linux-build.patch

78 lines
3.1 KiB
Diff
Raw Normal View History

[tensorflow-cc] More or less completely rewritten port, update to 2.3.1 (#13028) * fix build issues occurring from default bazel config being used because customized bazel config is stored in wrong directory * [tensorflow-cc] Update CONTROL and ci.baseline.txt * fix also applies to windows static build * fix debug and static builds as well as library naming for non-linux - add patch to fix debug builds - add patch to fix exports for static linking - really build debug (instead of cloning release) - override bazel build options for debug (work around bazel bug) - bazel doesn't support static libraries: work around by building dynamic library and constructing static linkage commands from build log - Windows .pdb file can't be >4GB even on x64: work around using reduced debug information - Windows doesn't support .lib files >4GB even on x64, so split into multiple libs - vcpkg requires equal amount of libs for debug and release: work around using handcrafted empty dummy libs - fix naming of libs (.dll on Windows and .dylib on macOS) * upgrade tensorflow from v1.14 to v2.3 - adapt patch files to tensorflow code changes - update bazel from v0.25.2 to v3.1 - on Windows use python installed on the host instead of embedded python obtained via vcpkg because embedded python lacks pip, which we need to obtain numpy - on Windows add MSYS2 to the PATH so that bazel tools can access MSYS2 GIT - add support for custom CA certificates when using HTTPS_PROXY * fix execute process macro The existing implementation totally screwed up commands if the command's arguments contained semicolons (this is the case, e.g., in the FindPython modules of the cmake distribution). * extend overriden execute_process to more than one COMMAND as there actually are use cases for this * added another patch required for tensorflow v2.3, fixed path and working directory * Revert "incorporate changes from microsoft:master" * Revert "Revert "incorporate changes from microsoft:master"" * final fixes for static build + improving out messages * enabling linux and osx in CI to see if it works now * simplified code, fixed version numbers, fixed generated include cmake file * fix failing postbuild check on handcrafted empty dummy library by spreading the last real libraries contents over the required number of libraries * remove dead code commit by mistake again * improvements from code review * cleaner fix for debug code * find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3) * fix error in python helper script * fix wrong libname in postbuild script * fix python detection + switch to python on msys2 (instead of embedded python) for Windows as we need numpy * fix order of arguments * fix command (it may contain spaces such as C:\Program Files\...) * revert last commit (root cause for CI failures is something different: there are line breaks in path) * fix regex comparision (value needs to be escaped as it may contains regex special characters such as brackets, eg C:/Program Files (x86)/...) * fix linebreaks in generated file * fix CRT linkage (macOS doesn't support static CRT linkage; it's set to dynamic even static target triplets for macOS and linux) * refactor implemenation to avoid as much code duplication as possible -- algorithmically identical * fix version numbers in helper scripts * enable work-around for Windows until bazel fix is available * install missing python3-pip on linux * fix linux build by patching * apply timeout feature now available via merged master branch * correct linux build patch * improve debug build patches (no functional difference because LOG(FATAL, ...) macro internally anyway calls abort(), which the compiler doesn't detect in debug mode... * improve linux patch * temporarily add debug to inspect what's going on on macOS CI * remove temporary debug code and fix static linking scripts for linux and macOS * fix regex escaping * fix ambiguous match while grepping for the framework link command * extend fix of ambiguous match while grepping for the framework link command * fix what merge of master broke * fix more what got broken by merging master (all packages and their dependencies are now maintained manually instead of using pacman...) * remove "unofficial" from filename * added switch do distinct classic and manifest mode when generating config.cmake file * create symlinks for libraries without version number * fix linux postbuild script * temporarily disable code making problems * add note for linking on Linux and macOS * forget to add README file in previous commit * add file forgotton in macro fixing patch * fix python library path * fix macOS static link command * update linkage instructions in README * Update ports/tensorflow-cc/CONTROL * Update ports/tensorflow-cc/portfile.cmake * Update scripts/ci.baseline.txt * use vcpkg_execute_required_process * pass C_FLAGS and CXX_FLAGS to bazel * fix INTERFACE_INCLUDE_DIRECTORIES * fix optional c/cxx arguments * also add linker opts * update README * merge static libs into one to support force_load (cannot force_load both due to duplicate symbols) * update README * quote python path (it might contain spaces that don't get escaped inside outer quotes of bash command) * fix python path also for static build * add arm(64) as currently unsupported arch * bazel 3.7 is available -> remove workaround * update README, remove necessary c-ares from deps * update msys package * add uwp specific options, and minor general improvements * fix string replace * fix control file and windows path separator * revert backslashes-fix -- the root cause was missing .exe extension * upgrade to tf 2.3.1 * fix hard-coded version * remove uwp work-in-progress code so that PR can be merged * update README and print out usage info in portfile * Update ports/tensorflow-cc/README-linux Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-linux Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-linux Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-linux Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-linux Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-windows Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/portfile.cmake Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/portfile.cmake Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/portfile.cmake Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/portfile.cmake Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/portfile.cmake Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * improve usage hints as discussed in review * add comment * apply changes from review * make additional compiler / linker args space-proof * Update ports/tensorflow-cc/README-macos Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/tensorflow-cc-config-shared.cmake.in Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/tensorflow-cc-config-shared.cmake.in Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-linux Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * Update ports/tensorflow-cc/README-macos Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * apply changes from code review * maybe fix the config files * rob.maynard CRs * fix windows static lib naming for first part * Update ports/tensorflow-cc/generate_static_link_cmd_windows.py Co-authored-by: nicole mazzuca <mazzucan@outlook.com> * apply rob maynards 1st suggestion * nicole CRs * format/fix-compile * fix missing string termination * prefer IMPORTED_LOCATION over IMPORTED_LOCATION_RELEASE to have default fall-back * hopefully fix the issue where no libraries are generated * final stuff Co-authored-by: Gehweiler <Joachim_Gehweiler@McAfee.com> Co-authored-by: wangli28 <wangli28@beyondsoft.com> Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com> Co-authored-by: Joachim Gehweiler <joachim@Joachims-iMac.local> Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
2020-11-13 02:15:48 +08:00
diff --git a/third_party/aws/aws-checksums.bazel b/third_party/aws/aws-checksums.bazel
--- a/third_party/aws/aws-checksums.bazel
+++ b/third_party/aws/aws-checksums.bazel
@@ -7,6 +7,13 @@
exports_files(["LICENSE"])
+load("@bazel_skylib//lib:selects.bzl", "selects")
+
+selects.config_setting_group(
+ name = "is_linux_debug",
+ match_all = ["@org_tensorflow//tensorflow:linux_x86_64", "@org_tensorflow//tensorflow:debug"],
+)
+
cc_library(
name = "aws-checksums",
srcs = select({
@@ -27,4 +34,10 @@
deps = [
"@aws-c-common",
],
+ defines = select({
+ ":is_linux_debug": [
+ "DEBUG_BUILD"
+ ],
+ "//conditions:default": [],
+ }),
)
diff --git a/tensorflow/core/kernels/data/experimental/io_ops.cc b/tensorflow/core/kernels/data/experimental/io_ops.cc
--- a/tensorflow/core/kernels/data/experimental/io_ops.cc
+++ b/tensorflow/core/kernels/data/experimental/io_ops.cc
@@ -28,6 +28,12 @@
namespace data {
namespace experimental {
+/*static*/ constexpr const char* const LoadDatasetOp::kCompression;
+/*static*/ constexpr const char* const LoadDatasetOp::kReaderFunc;
+/*static*/ constexpr const char* const LoadDatasetOp::kReaderFuncTarguments;
+
+/*static*/ constexpr const int SaveDatasetOp::kFileFormatVersion;
+
SaveDatasetOp::SaveDatasetOp(OpKernelConstruction* ctx)
: HybridAsyncOpKernel(ctx, "tf_data_save_dataset") {
OP_REQUIRES_OK(ctx, ctx->GetAttr(kCompression, &compression_));
diff --git a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc b/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc
--- a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc
+++ b/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc
@@ -63,6 +63,14 @@
namespace data {
namespace experimental {
+/*static*/ constexpr const char* const SnapshotDatasetV2Op::kCompression;
+/*static*/ constexpr const char* const SnapshotDatasetV2Op::kReaderFunc;
+/*static*/ constexpr const char* const SnapshotDatasetV2Op::kShardFunc;
+/*static*/ constexpr const char* const SnapshotDatasetV2Op::kReaderFuncTarguments;
+/*static*/ constexpr const char* const SnapshotDatasetV2Op::kShardFuncTarguments;
+
+/*static*/ constexpr const int SnapshotDatasetV2Op::kFileFormatVersion;
+
// ==== Snapshot Implementation ====
/* The current snapshot on-disk layout is as follows:
[tensorflow] Add C port (#14392) * fix build issues occurring from default bazel config being used because customized bazel config is stored in wrong directory * [tensorflow-cc] Update CONTROL and ci.baseline.txt * fix also applies to windows static build * fix debug and static builds as well as library naming for non-linux - add patch to fix debug builds - add patch to fix exports for static linking - really build debug (instead of cloning release) - override bazel build options for debug (work around bazel bug) - bazel doesn't support static libraries: work around by building dynamic library and constructing static linkage commands from build log - Windows .pdb file can't be >4GB even on x64: work around using reduced debug information - Windows doesn't support .lib files >4GB even on x64, so split into multiple libs - vcpkg requires equal amount of libs for debug and release: work around using handcrafted empty dummy libs - fix naming of libs (.dll on Windows and .dylib on macOS) * upgrade tensorflow from v1.14 to v2.3 - adapt patch files to tensorflow code changes - update bazel from v0.25.2 to v3.1 - on Windows use python installed on the host instead of embedded python obtained via vcpkg because embedded python lacks pip, which we need to obtain numpy - on Windows add MSYS2 to the PATH so that bazel tools can access MSYS2 GIT - add support for custom CA certificates when using HTTPS_PROXY * fix execute process macro The existing implementation totally screwed up commands if the command's arguments contained semicolons (this is the case, e.g., in the FindPython modules of the cmake distribution). * extend overriden execute_process to more than one COMMAND as there actually are use cases for this * added another patch required for tensorflow v2.3, fixed path and working directory * Revert "incorporate changes from microsoft:master" * Revert "Revert "incorporate changes from microsoft:master"" * final fixes for static build + improving out messages * enabling linux and osx in CI to see if it works now * simplified code, fixed version numbers, fixed generated include cmake file * fix failing postbuild check on handcrafted empty dummy library by spreading the last real libraries contents over the required number of libraries * remove dead code commit by mistake again * improvements from code review * cleaner fix for debug code * find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3) * fix error in python helper script * fix wrong libname in postbuild script * fix python detection + switch to python on msys2 (instead of embedded python) for Windows as we need numpy * fix order of arguments * fix command (it may contain spaces such as C:\Program Files\...) * revert last commit (root cause for CI failures is something different: there are line breaks in path) * fix regex comparision (value needs to be escaped as it may contains regex special characters such as brackets, eg C:/Program Files (x86)/...) * fix linebreaks in generated file * fix CRT linkage (macOS doesn't support static CRT linkage; it's set to dynamic even static target triplets for macOS and linux) * refactor implemenation to avoid as much code duplication as possible -- algorithmically identical * fix version numbers in helper scripts * enable work-around for Windows until bazel fix is available * install missing python3-pip on linux * fix linux build by patching * apply timeout feature now available via merged master branch * correct linux build patch * improve debug build patches (no functional difference because LOG(FATAL, ...) macro internally anyway calls abort(), which the compiler doesn't detect in debug mode... * improve linux patch * temporarily add debug to inspect what's going on on macOS CI * remove temporary debug code and fix static linking scripts for linux and macOS * fix regex escaping * fix ambiguous match while grepping for the framework link command * extend fix of ambiguous match while grepping for the framework link command * fix what merge of master broke * fix more what got broken by merging master (all packages and their dependencies are now maintained manually instead of using pacman...) * remove "unofficial" from filename * added switch do distinct classic and manifest mode when generating config.cmake file * create symlinks for libraries without version number * fix linux postbuild script * temporarily disable code making problems * add note for linking on Linux and macOS * forget to add README file in previous commit * add file forgotton in macro fixing patch * fix python library path * fix macOS static link command * update linkage instructions in README * Update ports/tensorflow-cc/CONTROL * Update ports/tensorflow-cc/portfile.cmake * Update scripts/ci.baseline.txt * use vcpkg_execute_required_process * pass C_FLAGS and CXX_FLAGS to bazel * fix INTERFACE_INCLUDE_DIRECTORIES * fix optional c/cxx arguments * also add linker opts * update README * merge static libs into one to support force_load (cannot force_load both due to duplicate symbols) * update README * quote python path (it might contain spaces that don't get escaped inside outer quotes of bash command) * fix python path also for static build * add arm(64) as currently unsupported arch * bazel 3.7 is available -> remove workaround * update README, remove necessary c-ares from deps * update msys package * add uwp specific options, and minor general improvements * fix string replace * fix control file and windows path separator * revert backslashes-fix -- the root cause was missing .exe extension * upgrade to tf 2.3.1 * fix hard-coded version * remove uwp work-in-progress code so that PR can be merged * add [tensorflow] C API port * missing in previous commit * fix include file * remove unnecessary suffix * update README and print out usage info in portfile * Update ports/tensorflow-cc/CONTROL Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com> * Update ports/tensorflow/portfile.cmake Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com> * follow-up to code review * fix suffix parameter * fix quoting * extend linux patch * another try to fix quoting of possibly empty string parameter * different approach to fix empty string arguments * update list of headers for tensorflow 2.3.1 (was still 2.3.0) * Update ports/tensorflow/CONTROL Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com> * move common stuff into tensorflow-common as discussed in review * Apply suggestions from code review Co-authored-by: nicole mazzuca <mazzucan@outlook.com> Co-authored-by: Gehweiler <Joachim_Gehweiler@McAfee.com> Co-authored-by: wangli28 <wangli28@beyondsoft.com> Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com> Co-authored-by: Joachim Gehweiler <joachim@Joachims-iMac.local> Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com> Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
2020-11-26 03:18:44 +08:00
diff --git a/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc b/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc
--- a/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc
+++ b/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc
@@ -52,6 +52,8 @@
/* static */ constexpr const char* const
DataServiceDatasetOp::kMaxOutstandingRequests;
/* static */ constexpr const char* const
+ DataServiceDatasetOp::kTaskRefreshIntervalHintMs;
+/* static */ constexpr const char* const
DataServiceDatasetOp::kIterationCounter;
/* static */ constexpr const char* const DataServiceDatasetOp::kOutputTypes;
/* static */ constexpr const char* const DataServiceDatasetOp::kOutputShapes;