Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with batch reading of messages from SQS queue (Trigger) #536

Open
surya9teja opened this issue Sep 16, 2024 · 0 comments
Open

Problem with batch reading of messages from SQS queue (Trigger) #536

surya9teja opened this issue Sep 16, 2024 · 0 comments
Labels
area/plugin Plugin-related issue or feature request good first issue Great issue for new contributors

Comments

@surya9teja
Copy link

Issue description

Hi,
Does anyone knows how to parse the multiple SQS messages into JSON format. Currently I have something like this

id: sqs-testing
namespace: microform.boa

triggers:
  - id: sqs_trigger
    type: "io.kestra.plugin.aws.sqs.Trigger"
    accessKeyId: "{{kv(key='aws_access_key', errorOnMissing=true)}}"
    secretKeyId: "{{kv(key='aws_secret_key', errorOnMissing=true)}}"
    region: "eu-west-2"
    serdeType: JSON
    queueUrl: "queue_url"
    maxRecords: 10
    maxDuration: PT10S


tasks:
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{trigger}}"

The output logs loos like this

{"uri":"kestra:///microform/boa/sqs-testing/executions/2KUnGBcpzCvltFiRWcgMk0/trigger/sqs-trigger/15337633167485749557.ion","count":2}

Each message in SQS

{
   "document_ids":[
      2341,
      4565623,
      23561
   ]
}

When I do trigger with batch of more than 1 it seems likes it doesn't actually parse it into a json format, the output of the .ion file

{document_ids:[2341,456,23561]}
{document_ids:[231,5623,23561]}
{document_ids:[2341,4565623,23561]}

Not sure, whether I am reading it wrong or I messed up something. 

@kestrabot kestrabot bot added this to Issues Sep 16, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Issues Sep 16, 2024
@Ben8t Ben8t added the area/plugin Plugin-related issue or feature request label Sep 16, 2024
@tchiotludo tchiotludo added good first issue Great issue for new contributors and removed kind/good-first-issue labels Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/plugin Plugin-related issue or feature request good first issue Great issue for new contributors
Projects
Status: Backlog
Development

No branches or pull requests

4 participants