Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grok TransformItems task requires Ion based file #7

Open
shrutimantri opened this issue Aug 18, 2024 · 0 comments
Open

Grok TransformItems task requires Ion based file #7

shrutimantri opened this issue Aug 18, 2024 · 0 comments
Labels
bug Something isn't working kind/cooldown Great candidate for the cooldown period

Comments

@shrutimantri
Copy link
Contributor

Expected Behavior

The log file generally is as follows:

2020-12-14T12:18:51.397Z INFO Hi from Kestra team!
2020-12-14T12:18:51.397Z INFO Your output is: 12345
2020-12-14T12:18:51.397Z WARN Warning message!!
2020-12-14T12:18:51.397Z ERROR Somethiong went wrong..
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
  File "/tmp/teste.py", line 9, in <module>
    run_my_stuff()
NameError: name 'run_my_stuff' is not defined
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!

The expected behavior will be for the Grok TransformItems to be able to transform the file.

Actual Behaviour

Grok TransformItems is expecting an Ion file.

Using this type of log file (which is how log file generally is), results in an error:

2020-12-14T12:18:51.397Z INFO Hi from Kestra team!
2020-12-14T12:18:51.397Z INFO Your output is: 12345
2020-12-14T12:18:51.397Z WARN Warning message!!
2020-12-14T12:18:51.397Z ERROR Somethiong went wrong..
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
  File "/tmp/teste.py", line 9, in <module>
    run_my_stuff()
NameError: name 'run_my_stuff' is not defined
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!

Error:

2024-08-16 19:06:48.705com.fasterxml.jackson.core.JsonParseException: Corrupt content to decode; underlying IonReader problem: (com.amazon.ion.impl.IonReaderTextRawX$IonReaderTextParsingException) Syntax error at line 1 offset 50: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR]
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
2024-08-16 19:06:48.705Corrupt content to decode; underlying IonReader problem: (com.amazon.ion.impl.IonReaderTextRawX$IonReaderTextParsingException) Syntax error at line 1 offset 50: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR]
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
 Syntax error at line 1 offset 50: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR]

The task expects an Ion file like:

"2020-12-14T12:18:51.397Z INFO Hi from Kestra team!"
"2020-12-14T12:18:51.397Z INFO Your output is: 12345"
"2020-12-14T12:18:51.397Z WARN Warning message!!"
"2020-12-14T12:18:51.397Z ERROR Somethiong went wrong.."
"DEBUG:root:This message should go to the log file"
"ERROR:root:Got exception on main handler"Traceback
"Traceback (most recent call last):"
"  File \"/tmp/teste.py\", line 9, in <module>"
"    run_my_stuff()"
"NameError: name 'run_my_stuff' is not defined"
"2020-12-14T12:18:51.397Z INFO Hi from Kestra team!"

We do not have any task that will transform the given file in the Ion format. So, I am not sure how we can leverage this task.

Steps To Reproduce

  1. Use the flow given in the example.
  2. The input file can be log_file.txt with the following contents:
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!
2020-12-14T12:18:51.397Z INFO Your output is: 12345
2020-12-14T12:18:51.397Z WARN Warning message!!
2020-12-14T12:18:51.397Z ERROR Somethiong went wrong..
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
  File "/tmp/teste.py", line 9, in <module>
    run_my_stuff()
NameError: name 'run_my_stuff' is not defined
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!

Environment Information

  • Kestra Version: 0.18.1
  • Plugin version: 0.18.1
  • Operating System (OS / Docker / Kubernetes): Docker
  • Java Version (If not docker): N/A

Example flow

id: grok-flow
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: grok
    type: io.kestra.plugin.transform.grok.TransformItems
    pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
    from: "{{ inputs.file }}"
@shrutimantri shrutimantri added the bug Something isn't working label Aug 18, 2024
@kestrabot kestrabot bot added this to Issues Aug 18, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Issues Aug 18, 2024
@anna-geller anna-geller added the kind/cooldown Great candidate for the cooldown period label Aug 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working kind/cooldown Great candidate for the cooldown period
Projects
Status: Backlog
Development

No branches or pull requests

2 participants