[BugFix] Fix a bug that will lead no writable spill storage directories issue

[duanyyyyyyy]

Why I'm doing:

Fix a bug for be that will lead no writable spill storage directories issue
From the stack and code we can see that
W20241212 15:34:15.303994 139676325242624 pipeline_driver_executor.cpp:168] [Driver] Process error, query_id=7a9a93e1-b85b-11ef-99e1-d61caef69f96, instance_id=7a9a93e1-b85b-11ef-99e1-d61caef69f9a, status=Capaticy limit exceeded: no writable spill storage directories
be/src/exec/spill/log_block_manager.cpp:251 value_or_err_L251
be/src/exec/spill/data_stream.cpp:67 value_or_err_L67
be/src/exec/spill/data_stream.cpp:82 _prepare_block(state, total_write_size)
be/src/exec/spill/serde.cpp:159 output->append(state, {Slice(serialize_buffer.data(), written_bytes)}, written_bytes, chunk->num_rows())
be/src/exec/spill/mem_table.cpp:91 serde->serialize(_runtime_state, serde_ctx, chunk, output, need_aligned)
be/src/exec/spill/spill_components.cpp:478 mem_table->finalize(yield_ctx, output)
be/src/exec/spill/spill_components.cpp:503 spill_partition(yield_ctx, context, partition)
be/src/exec/spill/spill_components.cpp:686 _spill_input_partitions(ctx, spill_ctx, spilling_partitions)
be/src/exec/spill/spiller.hpp:40 task_status()
be/src/exec/hash_joiner.cpp:185 spiller()->spill(state, chunk, spill::ResourceMemTrackerGuard(tls_mem_tracker, state->query_ctx()->weak_from_this(), spiller()->weak_from_this()))
be/src/exec/pipeline/hashjoin/spillable_hash_join_build_operator.cpp:210 _join_builder->append_chunk_to_spill_buffer(state, spill_chunk)
in the acquire_writable_dir in dir_manager.cpp we will increase the size first

and in the log_block_manager.cpp we will decrease the size when LogBlockContainer is closed

And also in the log_block_manager.cpp when we append data we will also increase the size


So the bug comes.
And I added some log to provide the bug
in log_block_manger.cpp

in dir_manager.cpp

here is the log


The bug produce

What I'm doing:

Remove the RETURN_IF_ERROR(_writable_file->pre_allocate(total_size)); in the append_data method
Fixes #issue

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.4
  • 3.3
  • 3.2
  • 3.1
  • 3.0

[github-actions]

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[stdpain]

This is not a proper fix. see detail in #53925

[duanyyyyyyy]

This is not a proper fix. see detail in #53925

The reason for the bug is the size increase twice and decrease only once I don't think the patch solve it....

[stdpain]

This is not a proper fix. see detail in #53925

The reason for the bug is the size increase twice and decrease only once I don't think the patch solve it....

here was the PoxisWriteableFile implements not Container::preallocate

    Status pre_allocate(uint64_t size) override {
        TEST_ERROR_POINT("PosixFileSystem::pre_allocate");
        uint64_t offset = std::max(_filesize, _pre_allocated_size);
        int ret;
        RETRY_ON_EINTR(ret, fallocate(_fd, 0, offset, size));
        if (ret != 0) {
            if (errno == EOPNOTSUPP) {
                LOG(WARNING) << "The filesystem does not support fallocate().";
            } else if (errno == ENOSYS) {
                LOG(WARNING) << "The kernel does not implement fallocate().";
            } else {
                return io_error(_filename, errno);
            }
        }
        _pre_allocated_size = offset + size;
        return Status::OK();
    }

[duanyyyyyyy]

This is not a proper fix. see detail in #53925

The reason for the bug is the size increase twice and decrease only once I don't think the patch solve it....

here was the PoxisWriteableFile implements not Container::preallocate

    Status pre_allocate(uint64_t size) override {
        TEST_ERROR_POINT("PosixFileSystem::pre_allocate");
        uint64_t offset = std::max(_filesize, _pre_allocated_size);
        int ret;
        RETRY_ON_EINTR(ret, fallocate(_fd, 0, offset, size));
        if (ret != 0) {
            if (errno == EOPNOTSUPP) {
                LOG(WARNING) << "The filesystem does not support fallocate().";
            } else if (errno == ENOSYS) {
                LOG(WARNING) << "The kernel does not implement fallocate().";
            } else {
                return io_error(_filename, errno);
            }
        }
        _pre_allocated_size = offset + size;
        return Status::OK();
    }

I got it, U are right

[stdpain]

closed via #53925