Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka cloud events #125

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Kafka cloud events #125

wants to merge 9 commits into from
+2,156 −498

Conversation

jekka001
Copy link

@jekka001 jekka001 commented Oct 22, 2024 •
edited
Loading

Pull Request description

Use Kafka for cloud events in case KAFKA queue

  • Implemented the ability to use Apache Kafka for processing event and telemetry queues.
  • Maintained compatibility with existing mechanisms, allowing the transition to Kafka use config parameter queue.type: "kafka".

General checklist

  • You have reviewed the guidelines document.
  • Labels that classify your pull request have been added.
  • The milestone is specified and corresponds to fix version.
  • Description references specific issue.
  • Description contains human-readable scope of changes.
  • Description contains brief notes about what needs to be added to the documentation.
  • No merge conflicts, commented blocks of code, code formatting issues.
  • Changes are backward compatible or upgrade script is provided.
  • Similar PR is opened for PE version to simplify merge. Crosslinks between PRs added. Required for internal contributors only.

Back-End feature checklist

  • Added corresponding unit and/or integration test(s). Provide written explanation in the PR description if you have failed to add tests.
  • If new dependency was added: the dependency tree is checked for conflicts.
  • If new service was added: the service is marked with corresponding @TbCoreComponent, @TbRuleEngineComponent, @TbTransportComponent, etc.
  • If new REST API was added: the RestClient.java was updated, issue for Python REST client is created.
  • If new yml property was added: make sure a description is added (above or near the property).

@AndriiLandiak AndriiLandiak requested review from AndriiLandiak and volodymyr-babak and removed request for AndriiLandiak October 22, 2024 11:31
@AndriiLandiak AndriiLandiak added the enhancement New feature or request label Oct 22, 2024
@AndriiLandiak AndriiLandiak self-requested a review October 22, 2024 11:31
@jekka001 jekka001 changed the title [WIP] Kafka cloud events Kafka cloud events Oct 22, 2024
AndriiLandiak
AndriiLandiak reviewed Oct 25, 2024
View reviewed changes
application/src/main/java/org/thingsboard/server/controller/BaseController.java Show resolved Hide resolved
dao/src/main/java/org/thingsboard/server/dao/cloud/MessageConstants.java Outdated Show resolved Hide resolved
dao/src/main/java/org/thingsboard/server/dao/cloud/DefaultEdgeSettingsService.java Outdated Show resolved Hide resolved
common/proto/src/main/java/org/thingsboard/server/common/util/ProtoUtils.java Outdated Show resolved Hide resolved
application/src/main/java/org/thingsboard/server/service/cloud/QueueConstants.java Outdated Show resolved Hide resolved
...ication/src/main/java/org/thingsboard/server/service/cloud/PostgresUplinkMessageService.java Show resolved Hide resolved
...ication/src/main/java/org/thingsboard/server/service/cloud/PostgresUplinkMessageService.java Outdated Show resolved Hide resolved
application/src/main/java/org/thingsboard/server/service/cloud/KafkaUplinkMessageService.java Show resolved Hide resolved
application/src/main/java/org/thingsboard/server/service/cloud/BaseUplinkMessageService.java Outdated Show resolved Hide resolved
application/src/main/java/org/thingsboard/server/service/cloud/BaseUplinkMessageService.java Outdated Show resolved Hide resolved
@jekka001
Use Kafka for cloud events in case KAFKA queue
a442daa 
- Renamed BaseCloudEventService to PostgresCloudEventService and refactored it.
- Created KafkaCloudEventService.
- Created queues: tb_cloud_event and tb_cloud_event_ts.
- Refactored BaseUplinkMessageService.
- Created KafkaUplinkMessageService.
- Created TbCoreCloudEventProvider.

- Refactored all components that used these services.
@jekka001
Resolve Merge
efaea08
@jekka001
Added tests for Kafka events
1d51899 
- added KafkaContainerTestSuite for start tests in kafka queue type mode
- added service second service edge, kafka and zookeeper service to docker-compose
- refactored tests
@jekka001
Added Conversation Changes
8ecc452
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AndriiLandiak AndriiLandiak AndriiLandiak left review comments

@volodymyr-babak volodymyr-babak Awaiting requested review from volodymyr-babak

Assignees

@jekka001 jekka001

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jekka001 @AndriiLandiak