Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the CloudQuery configs dynamic #27

Open
anna-geller opened this issue Jan 22, 2024 · 0 comments
Open

Make the CloudQuery configs dynamic #27

anna-geller opened this issue Jan 22, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@anna-geller
Copy link
Member

Feature description

You'll get an error:

Error: failed to sync v3 source hackernews: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: failed to validate spec: could not parse start_time: parsing time "{{ render(vars.start_date) }}" as "2006-01-02T15:04:05Z07:00": cannot parse "{{ render(vars.start_date) }}" as "2006"

when trying to pass a start_date as dynamic variable coming from kestra's trigger.date

reproducer giving that error:

id: cloudquery_sync
namespace: dev

variables: 
  start_date: |
    {{ trigger.date ?? execution.startDate | dateAdd(-1, "DAYS") | date("yyyy-MM-dd'T'HH:mm:ssXXX", timeZone="UTC") }}

tasks:
  - id: hn_to_duckdb
    type: io.kestra.plugin.cloudquery.Sync
    incremental: false
    outputFiles:
      - "*.csv"
    configs:
      - kind: source
        spec:
          name: hackernews
          path: cloudquery/hackernews
          registry: cloudquery
          version: v3.0.21
          tables: ["*"]
          destinations:
            - file
          spec:
            item_concurrency: 100
            start_time: "{{ render(vars.start_date) }}"
            # start_time: "2024-01-22T00:00:00Z"
      - kind: destination
        spec:
          name: file
          path: cloudquery/file
          version: v3.4.17
          spec:
            path: "{% raw %}{{TABLE}}/{{UUID}}.{{FORMAT}}{% endraw %}"
            format: csv
triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "@daily"
    timezone: US/Eastern

Hardcoding the same string is working fine:

id: cloudquery_sync
namespace: dev

tasks:
  - id: hn_to_duckdb
    type: io.kestra.plugin.cloudquery.Sync
    incremental: false
    outputFiles:
      - "*.csv"
    configs:
      - kind: source
        spec:
          name: hackernews
          path: cloudquery/hackernews
          registry: cloudquery
          version: v3.0.21
          tables: ["*"]
          destinations:
            - file
          spec:
            item_concurrency: 100
            start_time: "2024-01-22T00:00:00Z"
      - kind: destination
        spec:
          name: file
          path: cloudquery/file
          version: v3.4.17
          spec:
            path: "{% raw %}{{TABLE}}/{{UUID}}.{{FORMAT}}{% endraw %}"
            format: csv
triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "@daily"
    timezone: US/Eastern
@anna-geller anna-geller added the enhancement New feature or request label Jan 22, 2024
@anna-geller anna-geller added this to the v0.17.0 milestone Jan 22, 2024
@anna-geller anna-geller changed the title Make the configs dynamic Make the CloudQuery configs dynamic Jan 23, 2024
@anna-geller anna-geller modified the milestones: v0.17.0, v0.20.0 Apr 10, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jun 10, 2024
@anna-geller anna-geller removed this from the v0.20.0 milestone Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

1 participant