Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wasm_filter_integration_test is flaky #37651

Open
antoniovleonti opened this issue Dec 13, 2024 · 0 comments
Open

wasm_filter_integration_test is flaky #37651

antoniovleonti opened this issue Dec 13, 2024 · 0 comments

Comments

@antoniovleonti
Copy link
Contributor

antoniovleonti commented Dec 13, 2024

test failure link: https://github.com/envoyproxy/envoy/actions/runs/12304301149/job/34341428315

INFO: From Action test/extensions/common/wasm/test_data/test_rust.wasm:
FAIL: //test/extensions/filters/http/wasm:wasm_filter_integration_test (shard 7 of 16) (see /build/bazel_root/base/execroot/envoy/bazel-out/k8-dbg/testlogs/test/extensions/filters/http/wasm/wasm_filter_integration_test/shard_7_of_16/test.log)
INFO: From Testing //test/extensions/filters/http/wasm:wasm_filter_integration_test (shard 7 of 16):
==================== Test output for //test/extensions/filters/http/wasm:wasm_filter_integration_test (shard 7 of 16):
Note: This is test shard 7 of 16.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from Runtimes/WasmFilterIntegrationTest
[ RUN      ] Runtimes/WasmFilterIntegrationTest.BodyManipulation/downstream_v8_cpp_http2
[2024-12-12 21:50:07.578][7352][critical][assert] [test/integration/base_integration_test.cc:479] assert failure: 0. Details: Timed out waiting for listeners.
[2024-12-12 21:50:07.580][7352][critical][backtrace] [./source/server/backtrace.h:127] Caught Aborted, suspect faulting address 0x6c00001cb8
[2024-12-12 21:50:07.580][7352][critical][backtrace] [./source/server/backtrace.h:111] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-12-12 21:50:07.580][7352][critical][backtrace] [./source/server/backtrace.h:112] Envoy version: 0/1.33.0-dev/test/DEBUG/BoringSSL
/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x25
==================
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=7352)
    #0 malloc ??:? (wasm_filter_integration_test+0x51b29f1)
    #1 operator new(unsigned long) ??:? (wasm_filter_integration_test+0x12b1c477)
    #2 Envoy::BackwardsTrace::addrMapping(bool) ??:? (wasm_filter_integration_test+0x109c5e3c)
    #3 Envoy::BackwardsTrace::logTrace() ??:? (wasm_filter_integration_test+0x109bd185)
    #4 Envoy::SignalAction::sigHandler(int, siginfo_t*, void*) ??:? (wasm_filter_integration_test+0x109b9886)
    #5 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) tsan_interceptors_posix.cpp:? (wasm_filter_integration_test+0x51baa81)
    #6 Envoy::BaseIntegrationTest::createGeneratedApiTestServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, Envoy::Server::FieldValidationConfig, bool, std::__1::unique_ptr<Envoy::IntegrationTestServer, std::__1::default_delete<Envoy::IntegrationTestServer> >&) ??:? (wasm_filter_integration_test+0xb454111)
    #7 Envoy::BaseIntegrationTest::createGeneratedApiTestServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, Envoy::Server::FieldValidationConfig, bool) ??:? (wasm_filter_integration_test+0xb44fb99)
    #8 Envoy::BaseIntegrationTest::createEnvoy() ??:? (wasm_filter_integration_test+0xb44fa2c)
    #9 Envoy::BaseIntegrationTest::initialize() ??:? (wasm_filter_integration_test+0xb44c3bf)
    #10 Envoy::HttpIntegrationTest::initialize() ??:? (wasm_filter_integration_test+0xaca7d9c)
    #11 Envoy::Extensions::Wasm::(anonymous namespace)::WasmFilterIntegrationTest_BodyManipulation_Test::TestBody() wasm_filter_integration_test.cc:? (wasm_filter_integration_test+0x52401d8)
    #12 non-virtual thunk to Envoy::Extensions::Wasm::(anonymous namespace)::WasmFilterIntegrationTest_BodyManipulation_Test::TestBody() wasm_filter_integration_test.cc:? (wasm_filter_integration_test+0x5240e8c)
    #13 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ??:? (wasm_filter_integration_test+0x129ff347)
    #14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ??:? (wasm_filter_integration_test+0x129e48ea)
    #15 testing::Test::Run() ??:? (wasm_filter_integration_test+0x129c8178)
    #16 testing::TestInfo::Run() ??:? (wasm_filter_integration_test+0x129c8fb1)
    #17 testing::TestSuite::Run() ??:? (wasm_filter_integration_test+0x129c9b50)
    #18 testing::internal::UnitTestImpl::RunAllTests() ??:? (wasm_filter_integration_test+0x129d9b8a)
    #19 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ??:? (wasm_filter_integration_test+0x12a051f7)
    #20 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ??:? (wasm_filter_integration_test+0x129e80da)
    #21 testing::UnitTest::Run() ??:? (wasm_filter_integration_test+0x129d944d)
    #22 RUN_ALL_TESTS() ??:? (wasm_filter_integration_test+0x10057a67)
    #23 Envoy::TestRunner::runTests(int, char**) ??:? (wasm_filter_integration_test+0x10056b43)
    #24 main ??:? (wasm_filter_integration_test+0x10053db6)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal ??:? in __interceptor_malloc
==================
[2024-12-12 21:50:08.867][7352][critical][backtrace] [./source/server/backtrace.h:119] #0: Envoy::SignalAction::sigHandler() [0x109b985b]
[2024-12-12 21:50:08.965][7352][critical][backtrace] [./source/server/backtrace.h:119] #1: __tsan::CallUserSignalHandler() [0x51baa82]
Execution result: https://mordenite.cluster.engflow.com/actions/executions/ChCgHfMQ0UNCiqLjEC0JFA-PEgdkZWZhdWx0GiUKIEBCX1ca_iaPl0na2EbPkOdJs6XKh-mK6DzrBsULqOpfEKkD
================================================================================
INFO: Elapsed time: 5038.646s, Critical Path: 2416.43s
INFO: 27377 processes: 7908 remote cache hit, 12171 internal, 1 local, 3 processwrapper-sandbox, 7294 remote.
//test/extensions/filters/http/wasm:wasm_filter_integration_test         FAILED in 1 out of 16 in 303.2s
  Stats over 16 runs: max = 303.2s, min = 80.8s, avg = 117.9s, dev = 63.7s
  /build/bazel_root/base/execroot/envoy/bazel-out/k8-dbg/testlogs/test/extensions/filters/http/wasm/wasm_filter_integration_test/shard_7_of_16/test.log
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant