Why I'm doing:

When performing large data ingestions, memory limitations during the ingestion process can result in the final Rowset containing a large number of segment files. This negatively impacts query performance and increases resource consumption (CPU and memory) during subsequent compaction operations.

To address this, we propose a new large data ingestion strategy that spills data during the ingestion process and assembles it into final segment files in the final stage of the ingestion. This approach reduces the number of segment files in the Rowset.

This PR represents the first phase, implementing the management of SpillBlockManager within each Delta writer.

What I'm doing:

Fix #53954

Three classes are introduced here:

1. LoadSpillBlockContainer: Responsible for managing the blocks continuously generated during the import process.
2. LoadSpillBlockManager: Responsible for managing both the BlockManager and the LoadSpillBlockContainer. The BlockManager handles block allocation and release. Each Delta writer has its own LoadSpillBlockManager, which will be released along with the Delta writer at the end of the import process, reclaiming all intermediate files.
3. SpillMemTableSink: Responsible for implementing the spill logic of the memtable (TODO).

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.4
  • 3.3
  • 3.2
  • 3.1
  • 3.0