Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for uv in Python script plugins (python, dbt, singer) #109

Open
anna-geller opened this issue Feb 17, 2024 · 0 comments
Open
Labels
area/plugin Plugin-related issue or feature request enhancement New feature or request good first issue Great issue for new contributors kind/pending-feedback Idea waiting for user feedback

Comments

@anna-geller
Copy link
Member

anna-geller commented Feb 17, 2024

Add support for uv:

Working example that would be great to support in a less verbose way:

id: script_in_venv
namespace: dev
tasks:
  - id: bash
    type: io.kestra.plugin.scripts.python.Commands
    runner: DOCKER
    docker: 
      image: python:3.12-slim
    inputFiles:
      main.py: |
        import requests
        from kestra import Kestra

        response = requests.get('https://google.com')
        print(response.status_code)
        Kestra.outputs({'status': response.status_code, 'text': response.text})
    beforeCommands:
      - pip install uv
      - uv venv --quiet
      - . .venv/bin/activate --quiet
      - uv pip install --quiet requests kestra
    commands:
      - python main.py

This should work:

id: dbtGitDockerDuckDB
namespace: blueprint

tasks:
  - id: dbt
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
    - id: cloneRepository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/kestra-io/dbt-demo
      branch: main

    - id: dbt-build
      type: io.kestra.plugin.dbt.cli.DbtCLI
      runner: PROCESS
      beforeCommands:
        - curl -LsSf https://astral.sh/uv/install.sh | sh
        - source $HOME/.cargo/env
        - uv venv
        - source .venv/bin/activate
        - uv pip install dbt-duckdb
      commands:
        - dbt deps
        - dbt build

Currently, there are some issues to using uv as /bin/sh is forced as an interpreter. It makes sense to create a virtual environment with uv by default in all Python-based plugins when PROCESS runner is used. Alternatively, if easier, we can add UV runner that will create and activate uv venv by default

@anna-geller anna-geller added the enhancement New feature or request label Feb 17, 2024
@anna-geller anna-geller added this to the v0.16.0 milestone Feb 17, 2024
@anna-geller anna-geller modified the milestones: v0.16.0, v0.20.0 Feb 29, 2024
@anna-geller anna-geller added kind/pending-feedback Idea waiting for user feedback and removed enhancement New feature or request labels Mar 25, 2024
@anna-geller anna-geller removed this from the v0.20.0 milestone Mar 25, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jun 10, 2024
@anna-geller anna-geller changed the title Add support for uv in all script plugins Add support for uv in Python script plugins (python, dbt, singer) Aug 20, 2024
@anna-geller anna-geller added enhancement New feature or request area/plugin Plugin-related issue or feature request labels Aug 20, 2024
@tchiotludo tchiotludo added good first issue Great issue for new contributors and removed kind/good-first-issue labels Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/plugin Plugin-related issue or feature request enhancement New feature or request good first issue Great issue for new contributors kind/pending-feedback Idea waiting for user feedback
Projects
Status: Backlog
Development

No branches or pull requests

2 participants