Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: support for collection-based windowed aggregations #486

Open
daniil-quix opened this issue Sep 5, 2024 · 0 comments
Open

Feature: support for collection-based windowed aggregations #486

daniil-quix opened this issue Sep 5, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@daniil-quix
Copy link
Collaborator

daniil-quix commented Sep 5, 2024

Add support for windowed aggregations that return collections (aka "aggregate()") rather than a single reduced value ("reduce()").
This type of API is helpful for aggregations requiring full datasets (e.g., medians), data buffering for ML models, etc.

Currently, aggregating collections is technically possible with reduce(), but it's not optimal:

  • Every time a new record comes to the window, the whole window state needs to be deserialized, updated, and serialized back.
  • Due to the above, more data will be sent to the changelog topic on every update, which may overflow the max message size in Kafka.

The API examples will be added later to this issue as research continues.

@daniil-quix daniil-quix converted this from a draft issue Sep 5, 2024
@daniil-quix daniil-quix added this to the Q4 2024 milestone Sep 5, 2024
@daniil-quix daniil-quix added the enhancement New feature or request label Sep 5, 2024
@gwaramadze gwaramadze self-assigned this Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: In Research
Development

No branches or pull requests

2 participants