Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Parameter to Ignore Delete Messages in Flink StarRocks Connector (Default: false) #392

Open
bugbubug opened this issue Nov 4, 2024 · 1 comment · May be fixed by #405
Open

Add Parameter to Ignore Delete Messages in Flink StarRocks Connector (Default: false) #392

bugbubug opened this issue Nov 4, 2024 · 1 comment · May be fixed by #405

Comments

@bugbubug
Copy link

bugbubug commented Nov 4, 2024

Issue Description:

Background:
The Flink StarRocks Connector currently processes delete messages, which may not be desired in certain use cases. To enhance flexibility and usability, we propose adding a parameter that allows users to ignore delete messages during data ingestion.

Proposed Feature:
We suggest introducing a new parameter, ignoreDeleteMessages, which will be included in both Flink SQL and the Flink API. This parameter will default to false, meaning delete messages will be processed unless explicitly set to true.

Benefits:

  • Provides users with greater control over data ingestion.
  • Enhances performance by reducing unnecessary processing of delete messages in scenarios where they are not needed.

Implementation Considerations:

  • Ensure backward compatibility by maintaining the default behavior of processing delete messages.
  • Update relevant documentation to reflect the new parameter.

We believe this enhancement will significantly improve the user experience and make the connector more versatile.


Issue 描述:

背景:
目前,Flink StarRocks Connector 会处理删除消息,而在某些用例中,这可能并不是所期望的。为了提高灵活性和可用性,我们建议增加一个参数,使用户能够在数据摄取过程中忽略删除消息。

建议功能:
我们建议在 Flink SQL 和 Flink API 中引入一个新参数 ignoreDeleteMessages。该参数的默认值为 false,这意味着删除消息将被处理,除非显式设置为 true

好处:

  • 提供用户更大的数据摄取控制。
  • 通过减少在不需要的情况下对删除消息的处理,提高性能。

实施考虑事项:

  • 确保通过维护处理删除消息的默认行为来实现向后兼容性。
  • 更新相关文档以反映新参数。

我们相信这一增强功能将显著改善用户体验,使连接器更具多样性。

@bugbubug
Copy link
Author

bugbubug commented Nov 4, 2024

I would like to take responsibility for implementing this feature. I believe it will greatly enhance the usability of the Flink StarRocks Connector. I will start working on it and keep you updated on my progress. If there are any specific requirements or considerations to keep in mind, please let me know. Thank you!

@bugbubug bugbubug linked a pull request Dec 12, 2024 that will close this issue
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant