server
server copied to clipboard
triton-inference-server
build.py fails during onnxruntime backend installation
Description
I am trying to build a custom triton server image, but it fails during the onnxruntime backend installation. Leaving it out, the build runs successfully.
15:35:03 #15 4.827 CMake Error at /workspace/build/Release/_deps/google_nsync-src/CMakeLists.txt:1 (cmake_minimum_required):
15:35:03 #15 4.827 Compatibility with CMake < 3.5 has been removed from CMake.
15:35:03 #15 4.827
15:35:03 #15 4.827 Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
15:35:03 #15 4.827 to tell CMake that the project requires at least <min> but has been updated
15:35:03 #15 4.827 to work with policies introduced by <max> or earlier.
15:35:03 #15 4.827
15:35:03 #15 4.827 Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
15:35:03 #15 4.827
15:35:03 #15 4.827
15:35:03 #15 4.828 CMake Warning in CMakeLists.txt:
15:35:03 #15 4.828 CMAKE_SKIP_INSTALL_RULES was enabled even though installation rules have
15:35:03 #15 4.828 been specified
15:35:03 #15 4.828
15:35:03 #15 4.828
15:35:03 #15 4.829 -- Configuring incomplete, errors occurred!
15:35:03 #15 4.838 Namespace(build_dir='/workspace/build', config=['Release'], update=True, build=True, clean=False, parallel=0, nvcc_threads=-1, test=False, skip_tests=False, compile_no_warning_as_error=True, enable_nvtx_profile=False, enable_memory_profile=False, enable_training=False, enable_training_apis=False, enable_training_ops=False, enable_nccl=False, mpi_home=None, nccl_home=None, use_mpi=False, enable_onnx_tests=False, path_to_protoc_exe=None, fuzz_testing=False, enable_symbolic_shape_infer_tests=False, gen_doc=None, gen_api_doc=False, use_cuda=True, cuda_version=None, cuda_home='/usr/local/cuda', cudnn_home='/usr', enable_cuda_line_info=False, enable_cuda_nhwc_ops=False, enable_pybind=False, build_wheel=False, wheel_name_suffix=None, skip_keras_test=False, build_csharp=False, build_nuget=False, msbuild_extra_options=None, build_java=False, build_nodejs=False, build_objc=False, build_shared_lib=True, build_apple_framework=False, cmake_extra_defines=[["CMAKE_CUDA_ARCHITECTURES='75;80;86;89;90;100;120'"]], target=None, x86=False, rv64=False, arm=False, arm64=False, arm64ec=False, buildasx=False, riscv_toolchain_root='', riscv_qemu_path='', msvc_toolset=None, windows_sdk_version=None, android=False, android_abi='arm64-v8a', android_api=27, android_sdk_path='', android_ndk_path='', android_cpp_shared=False, android_run_emulator=False, use_gdk=False, gdk_edition='.', gdk_platform='Scarlett', ios=False, visionos=False, macos=None, apple_sysroot='', ios_toolchain_file='', visionos_toolchain_file='', xcode_code_signing_team_id='', xcode_code_signing_identity='', cmake_generator=None, osx_arch='x86_64', apple_deploy_target=None, enable_address_sanitizer=False, use_binskim_compliant_compile_flags=False, disable_memleak_checker=False, use_vcpkg=False, build_wasm=False, build_wasm_static_lib=False, emsdk_version='3.1.59', enable_wasm_simd=False, enable_wasm_threads=False, disable_wasm_exception_catching=False, enable_wasm_api_exception_catching=False, enable_wasm_exception_throwing_override=True, wasm_run_tests_in_browser=False, enable_wasm_profiling=False, enable_wasm_debug_info=False, wasm_malloc=None, emscripten_settings=None, use_extensions=False, extensions_overridden_path=None, cmake_path='cmake', ctest_path='ctest', skip_submodule_sync=True, use_mimalloc=False, use_dnnl=False, dnnl_gpu_runtime='', dnnl_opencl_root='', use_openvino='CPU', dnnl_aarch64_runtime='', dnnl_acl_root='', use_coreml=False, use_webnn=False, use_snpe=False, snpe_root=None, use_nnapi=False, use_vsinpu=False, nnapi_min_api=None, use_jsep=False, use_webgpu=False, use_qnn=False, qnn_home=None, use_rknpu=False, use_preinstalled_eigen=False, eigen_path=None, enable_msinternal=False, llvm_path=None, use_vitisai=False, use_tvm=False, tvm_cuda_runtime=False, use_tvm_hash=False, use_tensorrt=True, use_tensorrt_builtin_parser=True, use_tensorrt_oss_parser=False, tensorrt_home='/usr/src/tensorrt', test_all_timeout='10800', use_migraphx=False, migraphx_home=None, use_full_protobuf=False, llvm_config='', skip_onnx_tests=False, skip_winml_tests=False, skip_nodejs_tests=False, enable_msvc_static_runtime=False, use_dml=False, dml_path='', use_winml=False, winml_root_namespace_override=None, dml_external_project=False, use_telemetry=False, enable_wcos=False, enable_lto=False, enable_transformers_tool_test=False, use_acl=False, acl_home=None, acl_libs=None, use_armnn=False, armnn_relu=False, armnn_bn=False, armnn_home=None, armnn_libs=None, build_micro_benchmarks=False, minimal_build=None, include_ops_by_config=None, enable_reduced_operator_type_support=False, disable_contrib_ops=False, disable_ml_ops=False, disable_rtti=False, disable_types=[], disable_exceptions=False, rocm_version=None, use_rocm=False, rocm_home=None, code_coverage=False, enable_lazy_tensor=False, ms_experimental=False, enable_external_custom_op_schemas=False, external_graph_transformer_path=None, enable_cuda_profiling=False, use_cann=False, cann_home=None, enable_rocm_profiling=False, use_xnnpack=False, use_avx512=False, use_azure=False, use_cache=False, use_triton_kernel=False, use_lock_free_queue=False, allow_running_as_root=True)
15:35:03 #15 4.838 Failed to import psutil. Please `pip install psutil` for better estimation of nvcc threads. Use nvcc_threads=1
15:35:03 #15 4.838 Traceback (most recent call last):
15:35:03 #15 4.838 File "/workspace/onnxruntime/tools/ci_build/build.py", line 2998, in <module>
15:35:03 #15 4.838 sys.exit(main())
15:35:03 #15 4.838 ^^^^^^
15:35:03 #15 4.838 File "/workspace/onnxruntime/tools/ci_build/build.py", line 2853, in main
15:35:03 #15 4.839 generate_build_tree(
15:35:03 #15 4.839 File "/workspace/onnxruntime/tools/ci_build/build.py", line 1674, in generate_build_tree
15:35:03 #15 4.839 run_subprocess(
15:35:03 #15 4.839 File "/workspace/onnxruntime/tools/ci_build/build.py", line 867, in run_subprocess
15:35:03 #15 4.840 return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
15:35:03 #15 4.840 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:35:03 #15 4.840 File "/workspace/onnxruntime/tools/python/util/run.py", line 49, in run
15:35:03 #15 4.840 completed_process = subprocess.run(
15:35:03 #15 4.840 ^^^^^^^^^^^^^^^
15:35:03 #15 4.840 File "/usr/lib/python3.12/subprocess.py", line 571, in run
15:35:03 #15 4.840 raise CalledProcessError(retcode, process.args,
15:35:03 #15 4.840 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '/workspace/onnxruntime/cmake', '--compile-no-warning-as-error', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-DPython_EXECUTABLE=/usr/bin/python3', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-Donnxruntime_USE_VCPKG=OFF', '-Donnxruntime_USE_MIMALLOC=OFF', '-Donnxruntime_ENABLE_PYTHON=OFF', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_OBJC=OFF', '-Donnxruntime_BUILD_SHARED_LIB=ON', '-Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_VSINPU=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_TENSORRT=ON', '-Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_TVM_CUDA_RUNTIME=OFF', '-Donnxruntime_TVM_USE_HASH=OFF', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_USE_JSEP=OFF', '-Donnxruntime_USE_WEBGPU=OFF', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_TRAINING_APIS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=OFF', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-Donnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=OFF', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_USE_CUDA_NHWC_OPS=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON', '-Donnxruntime_ENABLE_WEBASSEMBLY_API_EXCEPTION_CATCHING=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=ON', '-Donnxruntime_WEBASSEMBLY_RUN_TESTS_IN_BROWSER=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF', '-Donnxruntime_ENABLE_LAZY_TENSOR=OFF', '-Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF', '-Donnxruntime_ENABLE_CUDA_PROFILING=OFF', '-Donnxruntime_ENABLE_ROCM_PROFILING=OFF', '-Donnxruntime_USE_XNNPACK=OFF', '-Donnxruntime_USE_WEBNN=OFF', '-Donnxruntime_USE_CANN=OFF', '-Donnxruntime_USE_TRITON_KERNEL=OFF', '-Donnxruntime_DISABLE_FLOAT8_TYPES=OFF', '-Donnxruntime_DISABLE_SPARSE_TENSORS=OFF', '-Donnxruntime_DISABLE_OPTIONAL_TYPE=OFF', '-Donnxruntime_NVCC_THREADS=1', '-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc', '-Donnxruntime_TENSORRT_HOME=/usr/src/tensorrt', '-Donnxruntime_USE_OPENVINO=ON', '-Donnxruntime_NPU_NO_FALLBACK=OFF', '-Donnxruntime_USE_OPENVINO_GPU=OFF', '-Donnxruntime_USE_OPENVINO_CPU=ON', '-Donnxruntime_USE_OPENVINO_NPU=OFF', '-Donnxruntime_USE_OPENVINO_GPU_NP=OFF', '-Donnxruntime_USE_OPENVINO_CPU_NP=OFF', '-Donnxruntime_USE_OPENVINO_NPU_NP=OFF', '-Donnxruntime_USE_OPENVINO_HETERO=OFF', '-Donnxruntime_USE_OPENVINO_DEVICE=CPU', '-Donnxruntime_USE_OPENVINO_MULTI=OFF', '-Donnxruntime_USE_OPENVINO_AUTO=OFF', '-Donnxruntime_USE_FULL_PROTOBUF=ON', '-DProtobuf_USE_STATIC_LIBS=ON', '-DCUDA_CUDA_LIBRARY=/usr/local/cuda/lib64/stubs', "-DCMAKE_CUDA_ARCHITECTURES='75;80;86;89;90;100;120'", '-DCMAKE_TLS_VERIFY=ON', '-DFETCHCONTENT_QUIET=OFF', '-Donnxruntime_USE_CUDA=ON', '-Donnxruntime_CUDA_HOME=/usr/local/cuda', '-Donnxruntime_CUDNN_HOME=/usr', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/workspace/build/Release/installed']' returned non-zero exit status 1.
15:35:03 #15 ERROR: process "/bin/sh -c ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home \"/usr/local/cuda\" --cudnn_home \"/usr\" --use_tensorrt --use_tensorrt_builtin_parser --tensorrt_home \"/usr/src/tensorrt\" --allow_running_as_root --use_openvino CPU" did not complete successfully: exit code: 1
15:35:03 ------
15:35:03 > importing cache manifest from tritonserver_onnxruntime:
15:35:03 ------
15:35:03 ------
15:35:03 > importing cache manifest from tritonserver_onnxruntime_cache0:
15:35:03 ------
15:35:03 ------
15:35:03 > importing cache manifest from tritonserver_onnxruntime_cache1:
15:35:03 ------
15:35:03 ------
15:35:03 > [ 9/24] RUN ./build.sh --config Release --skip_submodule_sync --parallel --build_shared_lib --compile_no_warning_as_error --build_dir /workspace/build --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES='75;80;86;89;90;100;120' --update --build --use_cuda --cuda_home "/usr/local/cuda" --cudnn_home "/usr" --use_tensorrt --use_tensorrt_builtin_parser --tensorrt_home "/usr/src/tensorrt" --allow_running_as_root --use_openvino CPU:
15:35:03 4.839 run_subprocess(
15:35:03 4.839 File "/workspace/onnxruntime/tools/ci_build/build.py", line 867, in run_subprocess
15:35:03 4.840 return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
15:35:03 4.840 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15:35:03 4.840 File "/workspace/onnxruntime/tools/python/util/run.py", line 49, in run
15:35:03 4.840 completed_process = subprocess.run(
15:35:03 4.840 ^^^^^^^^^^^^^^^
15:35:03 4.840 File "/usr/lib/python3.12/subprocess.py", line 571, in run
15:35:03 4.840 raise CalledProcessError(retcode, process.args,
15:35:03 4.840 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '/workspace/onnxruntime/cmake', '--compile-no-warning-as-error', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-DPython_EXECUTABLE=/usr/bin/python3', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-Donnxruntime_USE_VCPKG=OFF', '-Donnxruntime_USE_MIMALLOC=OFF', '-Donnxruntime_ENABLE_PYTHON=OFF', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_OBJC=OFF', '-Donnxruntime_BUILD_SHARED_LIB=ON', '-Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_VSINPU=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_TENSORRT=ON', '-Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_TVM_CUDA_RUNTIME=OFF', '-Donnxruntime_TVM_USE_HASH=OFF', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_USE_JSEP=OFF', '-Donnxruntime_USE_WEBGPU=OFF', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_TRAINING_APIS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=OFF', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-Donnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=OFF', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_USE_CUDA_NHWC_OPS=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON', '-Donnxruntime_ENABLE_WEBASSEMBLY_API_EXCEPTION_CATCHING=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=ON', '-Donnxruntime_WEBASSEMBLY_RUN_TESTS_IN_BROWSER=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF', '-Donnxruntime_ENABLE_LAZY_TENSOR=OFF', '-Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF', '-Donnxruntime_ENABLE_CUDA_PROFILING=OFF', '-Donnxruntime_ENABLE_ROCM_PROFILING=OFF', '-Donnxruntime_USE_XNNPACK=OFF', '-Donnxruntime_USE_WEBNN=OFF', '-Donnxruntime_USE_CANN=OFF', '-Donnxruntime_USE_TRITON_KERNEL=OFF', '-Donnxruntime_DISABLE_FLOAT8_TYPES=OFF', '-Donnxruntime_DISABLE_SPARSE_TENSORS=OFF', '-Donnxruntime_DISABLE_OPTIONAL_TYPE=OFF', '-Donnxruntime_NVCC_THREADS=1', '-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc', '-Donnxruntime_TENSORRT_HOME=/usr/src/tensorrt', '-Donnxruntime_USE_OPENVINO=ON', '-Donnxruntime_NPU_NO_FALLBACK=OFF', '-Donnxruntime_USE_OPENVINO_GPU=OFF', '-Donnxruntime_USE_OPENVINO_CPU=ON', '-Donnxruntime_USE_OPENVINO_NPU=OFF', '-Donnxruntime_USE_OPENVINO_GPU_NP=OFF', '-Donnxruntime_USE_OPENVINO_CPU_NP=OFF', '-Donnxruntime_USE_OPENVINO_NPU_NP=OFF', '-Donnxruntime_USE_OPENVINO_HETERO=OFF', '-Donnxruntime_USE_OPENVINO_DEVICE=CPU', '-Donnxruntime_USE_OPENVINO_MULTI=OFF', '-Donnxruntime_USE_OPENVINO_AUTO=OFF', '-Donnxruntime_USE_FULL_PROTOBUF=ON', '-DProtobuf_USE_STATIC_LIBS=ON', '-DCUDA_CUDA_LIBRARY=/usr/local/cuda/lib64/stubs', "-DCMAKE_CUDA_ARCHITECTURES='75;80;86;89;90;100;120'", '-DCMAKE_TLS_VERIFY=ON', '-DFETCHCONTENT_QUIET=OFF', '-Donnxruntime_USE_CUDA=ON', '-Donnxruntime_CUDA_HOME=/usr/local/cuda', '-Donnxruntime_CUDNN_HOME=/usr', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/workspace/build/Release/installed']' returned non-zero exit status 1.
15:35:03 ------
15:35:03 Dockerfile.ort:67
15:35:03 --------------------
15:35:03 65 | ARG COMMON_BUILD_ARGS="--config ${ONNXRUNTIME_BUILD_CONFIG} --skip_submodule_sync --parallel --build_shared_lib --compile_no_warning_as_error --build_dir /workspace/build --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES='75;80;86;89;90;100;120' "
15:35:03 66 |
15:35:03 67 | >>> RUN ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home "/usr/local/cuda" --cudnn_home "/usr" --use_tensorrt --use_tensorrt_builtin_parser --tensorrt_home "/usr/src/tensorrt" --allow_running_as_root --use_openvino CPU
15:35:03 68 |
15:35:03 69 | #
15:35:03 --------------------
15:35:03 ERROR: failed to solve: process "/bin/sh -c ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home \"/usr/local/cuda\" --cudnn_home \"/usr\" --use_tensorrt --use_tensorrt_builtin_parser --tensorrt_home \"/usr/src/tensorrt\" --allow_running_as_root --use_openvino CPU" did not complete successfully: exit code: 1
15:35:03 gmake[2]: *** [CMakeFiles/ort_target.dir/build.make:74: onnxruntime/lib/libonnxruntime.so] Error 1
15:35:03 gmake[1]: *** [CMakeFiles/Makefile2:278: CMakeFiles/ort_target.dir/all] Error 2
15:35:03 gmake: *** [Makefile:136: all] Error 2
15:35:04 error: build failed
Triton Information
I am using the latest r25.02 release branches, but I also get the same result with r25.01.
To Reproduce
Run this command locally in the server repository:
python3 ./build.py --backend=onnxruntime --enable-gpu
Expected behavior
Run the build successfully.
Also tried adding this argument to the build command, but it is still failing:
--extra-backend-cmake-arg=onnxruntime:CMAKE_POLICY_VERSION_MINIMUM=3.5
I have the same error too
4.260 [ 33%] No update step for 'google_nsync-populate'
4.278 [ 44%] Performing patch step for 'google_nsync-populate'
4.290 patching file public/nsync_atomic.h
4.314 [ 55%] No configure step for 'google_nsync-populate'
4.351 [ 66%] No build step for 'google_nsync-populate'
4.387 [ 77%] No install step for 'google_nsync-populate'
4.424 [ 88%] No test step for 'google_nsync-populate'
4.460 [100%] Completed 'google_nsync-populate'
4.480 [100%] Built target google_nsync-populate
4.487 CMake Error at /workspace/build/Release/_deps/google_nsync-src/CMakeLists.txt:1 (cmake_minimum_required):
4.487 Compatibility with CMake < 3.5 has been removed from CMake.
4.487
4.487 Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
4.487 to tell CMake that the project requires at least <min> but has been updated
4.487 to work with policies introduced by <max> or earlier.
4.487
4.487 Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
4.487
4.487
4.487 CMake Warning in CMakeLists.txt:
4.487 CMAKE_SKIP_INSTALL_RULES was enabled even though installation rules have
4.487 been specified
4.487
4.487
4.489 -- Configuring incomplete, errors occurred!
Also tried adding this argument to the build command, but it is still failing:
--extra-backend-cmake-arg=onnxruntime:CMAKE_POLICY_VERSION_MINIMUM=3.5
That cmake flag is effective, but without the onnxruntime: prefix AND without -D:
--extra-core-cmake-arg=CMAKE_POLICY_VERSION_MINIMUM=3.5
Still it is not a solution, as it merely moves the process on to other compilation problems... in fact the entire issue could be renamed to a more generic one specifying cmake version (mine is 4.0.0, how about you?), because the problem seems to be related to unsupported multiple versions of cmake - both the genuinely obsolete ones and the latest 4.0.0 that seems to trigger that deprecation error as a false ;positive.
And the deprecation error occurs even without compiling the ONNX backend: in my case it happens for any backend, e.g. the Python one, so I'd suggest verifying it and renaming the issue.
I attempted to build all the backends using my initial setup (CMake version 3.30.2 on my PC and version 3.28.3 specified by build.py). I successfully built all the backends except for the ONNX and Python ones.
However, with CMake version 4.0.0, the process already fails during the third-party installation, whether i add the extra flags or not.
I also encountered this issue, the root cause is that:
- cmake with specific version 3.28.3 is installed from kitware repository (the latest version is 4.0.0): https://github.com/triton-inference-server/onnxruntime_backend/blob/fd2a4e45305d6d6b60fec2fb8e499fa47702784a/tools/gen_ort_dockerfile.py#L158
- when openvino is enabled and being installed, the installation script upgrades cmake to the latest 4.0.0 version: https://github.com/triton-inference-server/onnxruntime_backend/blob/fd2a4e45305d6d6b60fec2fb8e499fa47702784a/tools/gen_ort_dockerfile.py#L190
- when onnxruntime is being built, it builds dependency google_nsync which is not compatible with cmake 4.0.0 version, so failure happens
For me, I am not using OpenVINO, so I mitigated by following patch:
--- a/build.py
+++ b/build.py
@@ -717,7 +717,7 @@ def onnxruntime_cmake_args(images, library_paths):
if (
(target_machine() != "aarch64")
and (target_platform() != "rhel")
- and (FLAGS.ort_openvino_version is not None)
+ and (FLAGS.ort_openvino_version)
):
cargs.append(
cmake_backend_enable(
And build with OpenVINO disabled:
python3 ./build.py --backend=onnxruntime --ort-openvino-version=''
If you don't need OpenVINO neither, you can work around like this, otherwise a patch on the OpenVINO install_openvino_dependencies.sh script to specify cmake version should work.
Facing the same , did anyone able to compile and build with onnx backend with or without docker , which branch and what command options ?
As mentioned by @lunwang-ttd, the root cause was because CMake was updated to version 4.0.0. However, as I was using an earlier version of Triton (i.e. r24.01), the fix proposed by him did not work for me. Instead, I made the following change that should be easy enough to be manually patched in any version:
--- a/build.py
+++ b/build.py
@@ -74,7 +74,7 @@ TRITON_VERSION_MAP = {
"24.01", # triton container
"24.01", # upstream container
"1.16.3", # ORT
- "2023.0.0", # ORT OpenVINO
+ None, # ORT OpenVINO
"2023.0.0", # Standalone OpenVINO
"3.2.6", # DCGM version
"py310_23.1.0-1", # Conda version
Then, I was able to successfully compile with:
python build.py --enable-gpu --backend onnxruntime --backend tensorflow (I don't need OpenVino)
Unlike the method proposed by @lunwang-ttd, there is no need to specify --ort-openvino-version=''
Sorry for the late reppy @l0rem1psum. It seems the above mentioned error is solved with your suggestion. But later ran into a strange error.
https://github.com/triton-inference-server/server/issues/8389
