envoyproxy
dynamic_modules: scaffolds config API & HTTP Filter
Commit Message: dynamic_modules: scaffolds config API & HTTP Filter Additional Description:
This scaffolds the configuration API marked as work-in-progress, and the skeleton HTTP filter implementation based on the configuration.
The real implementations will follow after this commit.
Risk Level: low Testing: done Docs Changes: n/a Release Notes: n/a (not enabled yet) Platform Specific Features: [Optional Runtime guard:] [Optional Fixes #Issue] [Optional Fixes commit #PR or SHA] [Optional Deprecated:] [Optional API Considerations:]
As a reminder, PRs marked as draft will not be automatically assigned reviewers, or be handled by maintainer-oncall triage.
Please mark your PR as ready when you want it to be reviewed!
:cat:
Caused by: https://github.com/envoyproxy/envoy/pull/36448 was opened by mathetake.
![repokitteh-read-only[bot] avatar](https://avatars.githubusercontent.com/in/16729?v=4 "Published by a pub.dev verified publisher"){kind=small}
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).
:cat:
Caused by: https://github.com/envoyproxy/envoy/pull/36448 was opened by mathetake.
cc @marc-barry
ps the flaky CI failure is nothing to do with this PR
ERROR: /source/test/extensions/load_balancing_policies/client_side_weighted_round_robin/BUILD:40:24: Compiling test/extensions/load_balancing_policies/client_side_weighted_round_robin/integration_test.cc failed: (Killed): clang-14 failed: error executing command (from target //test/extensions/load_balancing_policies/client_side_weighted_round_robin:integration_test)
no idea why suddenly gcc is getting killed ... (merged main again and hope all good)
~hmm looks legit and~ failed to reproduce locally either..
found the exact same failure on main: https://github.com/envoyproxy/envoy/actions/runs/11463146676/job/31896388874
looks like pending RBE jobs are piling up and that's why it's getting killed - I will retry later
@phlax i am getting flaky gcc: fatal error: Killed signal terminated program cc1plus (flaky in the sense that the killed build target varies each time) and I am seeing this on the main branch as well (https://github.com/envoyproxy/envoy/actions/runs/11463146676/job/31896388874). Retries so far failed to work. Do you have any idea what's going on?
hi anything that has killed in it implies OOM - this is due to our new RBE backend using smaller workers by default
to whatever build or test is failing add:
rbe_pool = "6gig"
hmm - the test target is really small and if they need the 6gig then almost entire test target in this repo needs one. (meanwhile I pushed the main merge again)
FYI https://github.com/envoyproxy/envoy/blob/551982e5f2d950b51ee718ab60b320dbb98fb733/test/extensions/dynamic_modules/http/factory_test.cc this is one of two cases killed during build - I would be really surprised if this one needs such setting
I removed the unnecessary test build deps https://github.com/envoyproxy/envoy/pull/36448/commits/ae9756fc9a2b8c529804238e219ba35bbacefe1e - hope this resolves this
hmm still happening. @phlax this hasn't happened before I did a main merge yesterday. I am inclined to believe that build target dependency added recently into the test framework (//test/mock etc?) or whatever is causing this. The current failing target is https://github.com/envoyproxy/envoy/blob/ae9756fc9a2b8c529804238e219ba35bbacefe1e/test/extensions/dynamic_modules/http/BUILD#L11-L22 and quite frankly it's insane if this one needs such option.
From the log, it's failing to link crazy amount of files for this really small build target.
/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/test/extensions/dynamic_modules/http/_objs/factory_test/factory_test.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/test/extensions/dynamic_modules/http/_objs/factory_test/factory_test.pic.o' -fPIC -DENVOY_ADMIN_FUNCTIONALITY -DENVOY_ENABLE_QUIC -DENVOY_ENABLE_FULL_PROTOS -DENVOY_ENABLE_YAML -DENVOY_ENABLE_HTTP_DATAGRAMS -DENVOY_MOBILE_ENABLE_LISTENER -DENVOY_GOOGLE_GRPC -DFMT_HEADER_ONLY -DSPDLOG_FMT_EXTERNAL '-DGTEST_HAS_ABSL=1' -DNGHTTP2_STATICLIB '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/com_github_gabime_spdlog -iquote bazel-out/k8-fastbuild/bin/external/com_github_gabime_spdlog -iquote external/com_github_fmtlib_fmt -iquote bazel-out/k8-fastbuild/bin/external/com_github_fmtlib_fmt -iquote external/com_google_absl -iquote bazel-out/k8-fastbuild/bin/external/com_google_absl -iquote external/com_google_protobuf -iquote bazel-out/k8-fastbuild/bin/external/com_google_protobuf -iquote external/com_github_cyan4973_xxhash -iquote bazel-out/k8-fastbuild/bin/external/com_github_cyan4973_xxhash -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-fastbuild/bin/external/com_googlesource_code_re2 -iquote external/envoy_api -iquote bazel-out/k8-fastbuild/bin/external/envoy_api -iquote external/com_google_googleapis -iquote bazel-out/k8-fastbuild/bin/external/com_google_googleapis -iquote external/com_envoyproxy_protoc_gen_validate -iquote bazel-out/k8-fastbuild/bin/external/com_envoyproxy_protoc_gen_validate -iquote external/com_github_cncf_xds -iquote bazel-out/k8-fastbuild/bin/external/com_github_cncf_xds -iquote external/dev_cel -iquote bazel-out/k8-fastbuild/bin/external/dev_cel -iquote external/opencensus_proto -iquote bazel-out/k8-fastbuild/bin/external/opencensus_proto -iquote external/com_github_jbeder_yaml_cpp -iquote bazel-out/k8-fastbuild/bin/external/com_github_jbeder_yaml_cpp -iquote external/utf8_range -iquote bazel-out/k8-fastbuild/bin/external/utf8_range -iquote external/boringssl -iquote bazel-out/k8-fastbuild/bin/external/boringssl -iquote external/com_github_google_tcmalloc -iquote bazel-out/k8-fastbuild/bin/external/com_github_google_tcmalloc -iquote external/com_google_googletest -iquote bazel-out/k8-fastbuild/bin/external/com_google_googletest -iquote external/com_github_google_quiche -iquote bazel-out/k8-fastbuild/bin/external/com_github_google_quiche -iquote external/com_googlesource_googleurl -iquote bazel-out/k8-fastbuild/bin/external/com_googlesource_googleurl -iquote external/com_github_nlohmann_json -iquote bazel-out/k8-fastbuild/bin/external/com_github_nlohmann_json -iquote external/com_github_openhistogram_libcircllhist -iquote bazel-out/k8-fastbuild/bin/external/com_github_openhistogram_libcircllhist -iquote external/com_github_mirror_tclap -iquote bazel-out/k8-fastbuild/bin/external/com_github_mirror_tclap -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools -iquote external/com_github_grpc_grpc -iquote bazel-out/k8-fastbuild/bin/external/com_github_grpc_grpc -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/internal_visibility -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/port -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_align -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/lite -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_allocation_policy -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_cleanup -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/string_block -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/varint_shuffle -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io_win32 -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/third_party/utf8_range/_virtual_includes/utf8_validity -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/third_party/utf8_range/_virtual_includes/utf8_range -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/gzip_stream -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/stubs -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/printer -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/zero_copy_sink -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/tokenizer -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/any_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/empty_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/struct_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/wrappers_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_layering_check_legacy -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/api_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/source_context_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/type_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/duration_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/field_mask_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/timestamp_proto -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/compiler/_virtual_includes/importer -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/delimited_message_util -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/differencer -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/field_mask_util -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/json_util -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/json -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/parser -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/descriptor_traits -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/lexer -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/message_path -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/zero_copy_buffered_stream -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/untyped_message -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/type_resolver -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_legacy -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/unparser -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/writer -Ibazel-out/k8-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/time_util -Ibazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/logger_impl_lib_standard -Ibazel-out/k8-fastbuild/bin/external/dev_cel/proto/cel/expr/_virtual_includes/checked_proto -Ibazel-out/k8-fastbuild/bin/external/dev_cel/proto/cel/expr/_virtual_includes/syntax_proto -Ibazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/thread_impl_lib_posix -Ibazel-out/k8-fastbuild/bin/source/common/filesystem/_virtual_includes/filesystem_impl_lib_posix -Ibazel-out/k8-fastbuild/bin/source/common/event/_virtual_includes/dispatcher_lib -Ibazel-out/k8-fastbuild/bin/source/common/event/_virtual_includes/signal_impl_lib_posix -Ibazel-out/k8-fastbuild/bin/source/common/filesystem/_virtual_includes/watcher_lib -Ibazel-out/k8-fastbuild/bin/source/common/api/_virtual_includes/os_sys_calls_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_export_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_logging_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_server_stats_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_stack_trace_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_platform_iovec_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_prefetch_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_command_line_flags_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_flag_utils_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_reference_counted_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_testvalue_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quic_base_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_flags_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_mem_slice_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_time_utils_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_client_stats_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/filesystem/_virtual_includes/directory_iterator_impl_lib_posix -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_googleurl_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_mutex_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_containers_impl_lib -Ibazel-out/k8-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_lower_case_string_impl_lib -Ibazel-out/k8-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_header_policy_impl_lib -isystem external/com_github_gabime_spdlog/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_gabime_spdlog/include -isystem external/com_github_fmtlib_fmt/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_fmtlib_fmt/include -isystem bazel-out/k8-fastbuild/bin/bazel/foreign_cc/zlib/include -isystem external/com_github_jbeder_yaml_cpp/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_jbeder_yaml_cpp/include -isystem external/boringssl/include -isystem bazel-out/k8-fastbuild/bin/external/boringssl/include -isystem external/com_google_googletest/googlemock -isystem bazel-out/k8-fastbuild/bin/external/com_google_googletest/googlemock -isystem external/com_google_googletest/googlemock/include -isystem bazel-out/k8-fastbuild/bin/external/com_google_googletest/googlemock/include -isystem external/com_google_googletest/googletest -isystem bazel-out/k8-fastbuild/bin/external/com_google_googletest/googletest -isystem external/com_google_googletest/googletest/include -isystem bazel-out/k8-fastbuild/bin/external/com_google_googletest/googletest/include -isystem bazel-out/k8-fastbuild/bin/bazel/foreign_cc/event/include -isystem bazel-out/k8-fastbuild/bin/bazel/foreign_cc/liburing/include -isystem bazel/external/http_parser -isystem bazel-out/k8-fastbuild/bin/bazel/external/http_parser -isystem bazel-out/k8-fastbuild/bin/external/envoy/bazel/foreign_cc/nghttp2/include -isystem external/com_github_nlohmann_json/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_nlohmann_json/include -isystem external/com_github_openhistogram_libcircllhist/src -isystem bazel-out/k8-fastbuild/bin/external/com_github_openhistogram_libcircllhist/src -isystem external/com_github_mirror_tclap/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_mirror_tclap/include '-DABSL_MIN_LOG_LEVEL=4' -fdebug-types-section -fPIC -Wno-deprecated-declarations '-std=c++20' -fsized-deallocation -Wall -Wextra -Werror -Wnon-virtual-dtor -Woverloaded-virtual -Wold-style-cast -Wformat -Wformat-security -Wvla -Wno-deprecated-declarations -Wreturn-type -Wno-maybe-uninitialized -DTCMALLOC -DENVOY_OBJECT_TRACE_ON_DUMP -DENVOY_HOT_RESTART -DENVOY_NGHTTP2 -DENVOY_ADMIN_HTML -DENVOY_STATIC_EXTENSION_REGISTRATION -DENVOY_GOOGLE_GRPC -DENVOY_HANDLE_SIGNALS -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c test/extensions/dynamic_modules/http/factory_test.cc -o bazel-out/k8-fastbuild/bin/test/extensions/dynamic_modules/http/_objs/factory_test/factory_test.pic.o
ok now I see rbe_pool = "6gig", is indeed added to factory_context_mocks ....
problem/solution is as i said above
ci was changed yesterday - altho tbf its been changing over the last few days/week - so not sure about previous runs
yeah anyways added the option - https://github.com/envoyproxy/envoy/pull/36448/commits/f46d3d72f9bba653328bed1430a4e7d1945f4350 thanks for the advice! (I think we need to audit the build dependency graph especially around the common build test targets)
to remove extraneous build/test deps? that would be amazing - did a bit of this previously - but thinning out the tree would make things much faster/use less resources
to remove extraneous build/test deps?
exactly. I think the problem i am having now is essentially because of that (judging by the fact that this has started happening after a certain main merge + happening at the linking phase where linker tried to put every dependencies in-memory regardless an object file is used or not)