diff options
| author | Dmitry Vyukov <dvyukov@google.com> | 2024-10-21 11:53:44 +0200 |
|---|---|---|
| committer | Dmitry Vyukov <dvyukov@google.com> | 2024-10-24 09:34:38 +0000 |
| commit | 9fc8fe026baab9959459256f2d47f4bbf21d405a (patch) | |
| tree | 6d97a7ac2b8e69f5fa7a92a4b3824b1ad9e571c7 /executor/executor.cc | |
| parent | a85e9d5032fdf305457a6400bd3af4a8df6c45c4 (diff) | |
executor: better handling for hanged test processes
Currently we kill hanged processes and consider the corresponding test finished.
We don't kill/wait for the actual test subprocess (we don't know its pid to kill,
and waiting will presumably hang). This has 2 problems:
1. If the hanged process causes "task hung" report, we can't reproduce it,
since the test finished too long ago (manager thinks its finished and
discards the request).
2. The test process still consumed per-pid resources.
Explicitly detect and handle such cases:
Manager keeps these hanged tests forever,
and we assign a new proc id for future processes
(don't reuse the hanged one).
Diffstat (limited to 'executor/executor.cc')
| -rw-r--r-- | executor/executor.cc | 7 |
1 files changed, 4 insertions, 3 deletions
diff --git a/executor/executor.cc b/executor/executor.cc index 305675bce..5b5e06422 100644 --- a/executor/executor.cc +++ b/executor/executor.cc @@ -470,7 +470,8 @@ static void setup_control_pipes(); static bool coverage_filter(uint64 pc); static rpc::ComparisonRaw convert(const kcov_comparison_t& cmp); static flatbuffers::span<uint8_t> finish_output(OutputData* output, int proc_id, uint64 req_id, uint32 num_calls, - uint64 elapsed, uint64 freshness, uint32 status, const std::vector<uint8_t>* process_output); + uint64 elapsed, uint64 freshness, uint32 status, bool hanged, + const std::vector<uint8_t>* process_output); static void parse_execute(const execute_req& req); static void parse_handshake(const handshake_req& req); @@ -1343,7 +1344,7 @@ void write_extra_output() } flatbuffers::span<uint8_t> finish_output(OutputData* output, int proc_id, uint64 req_id, uint32 num_calls, uint64 elapsed, - uint64 freshness, uint32 status, const std::vector<uint8_t>* process_output) + uint64 freshness, uint32 status, bool hanged, const std::vector<uint8_t>* process_output) { // In snapshot mode the output size is fixed and output_size is always initialized, so use it. int out_size = flag_snapshot ? output_size : output->size.load(std::memory_order_relaxed) ? @@ -1376,7 +1377,7 @@ flatbuffers::span<uint8_t> finish_output(OutputData* output, int proc_id, uint64 flatbuffers::Offset<flatbuffers::Vector<uint8_t>> output_off = 0; if (process_output) output_off = fbb.CreateVector(*process_output); - auto exec_off = rpc::CreateExecResultRaw(fbb, req_id, proc_id, output_off, error_off, prog_info_off); + auto exec_off = rpc::CreateExecResultRaw(fbb, req_id, proc_id, output_off, hanged, error_off, prog_info_off); auto msg_off = rpc::CreateExecutorMessageRaw(fbb, rpc::ExecutorMessagesRaw::ExecResult, flatbuffers::Offset<void>(exec_off.o)); fbb.FinishSizePrefixed(msg_off); |