-
Notifications
You must be signed in to change notification settings - Fork 155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[QUESTION] How to deploy a Pathway AirByte streaming ETL microservice to Google Cloud run? #53
Comments
Hi @vikas-velora apologies for the slow turnaround on your question - our team is verifying if this is the case. As a general rule, we advocate deployment from source (in this spirit: https://cloud.google.com/run/docs/deploying-source-code), and will provide the easiest recipe that works in this direction. The intended experience is something like this one with Render: https://pathway.com/developers/user-guide/deployment/render-deploy/. |
Got it, thanks. Will await your response.
…On Tue, 21 May 2024 at 7:29 AM, Adrian Kosowski ***@***.***> wrote:
Hi @vikas-velora <https://github.com/vikas-velora> apologies for the slow
turnaround on your question - our team is verifying if this is the case.
As a general rule, we advocate deployment from source (in this spirit:
https://cloud.google.com/run/docs/deploying-source-code), and will
provide the easiest recipe that works in this direction. The intended
experience is something like this one with Render:
https://pathway.com/developers/user-guide/deployment/render-deploy/.
—
Reply to this email directly, view it on GitHub
<#53 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BHP3QYFAAUEZ6CX56ZSZ4MTZDKS7RAVCNFSM6AAAAABH4CO73CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRRGU3DGNBUGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Adrian,
One question - if we want to dynamically deploy a service based on user
inputs, would it be better to deploy on Google cloud functions? Right now,
pathway works using yaml config, we want to parameterize it and take inputs
like OAuth token and GitHub repo name etc from users.
Thanks,
Vikas
…On Tue, 21 May 2024 at 7:57 AM, Vikas Singhvi ***@***.***> wrote:
Got it, thanks. Will await your response.
On Tue, 21 May 2024 at 7:29 AM, Adrian Kosowski ***@***.***>
wrote:
> Hi @vikas-velora <https://github.com/vikas-velora> apologies for the
> slow turnaround on your question - our team is verifying if this is the
> case.
>
> As a general rule, we advocate deployment from source (in this spirit:
> https://cloud.google.com/run/docs/deploying-source-code), and will
> provide the easiest recipe that works in this direction. The intended
> experience is something like this one with Render:
> https://pathway.com/developers/user-guide/deployment/render-deploy/.
>
> —
> Reply to this email directly, view it on GitHub
> <#53 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BHP3QYFAAUEZ6CX56ZSZ4MTZDKS7RAVCNFSM6AAAAABH4CO73CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRRGU3DGNBUGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Hi Vikas, To give a small technical heads-up, there is a way to dockerize the airbyte connector code for the local tests. Precisely, you would need to install Docker in your Dockerfile as follows:
And then run with mounting two volumes, as follows:
The first volume is required to enable DinD, while the second one is needed because the While I've also tried to use the So, to wrap it up, the way to go will be to run the airbyte connector in the GCP - a feature that must be added to Pathway. I am currently checking this possibility and will be back to you today or in a few days. |
Thanks so much @zxqfd555-pw . We tried multiple ways, and were unable to deploy - at least this confirms that it was not something to do with our knowledge 😊. Will wait for your update. |
Hi Vikas! A quick heads-up: we can eliminate the need for the DinD technique for airbyte connectors by introducing a mode where they run as GCP jobs. I am in the process of implementing it, and we can release the corresponding update next week. |
Thanks for the update.
…On Fri, 24 May 2024 at 3:42 PM, Sergey Kulik ***@***.***> wrote:
Hi Vikas!
A quick heads-up: we can eliminate the need for the DinD technique for
airbyte connectors by introducing a mode where they run as GCP jobs. I am
in the process of implementing it, and we can release the corresponding
update next week.
—
Reply to this email directly, view it on GitHub
<#53 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BHP3QYEM6DKIL3URLDYNFKTZD4G7JAVCNFSM6AAAAABH4CO73CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRZGE3DINZRGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi Vikas! Please note that now you can run airbyte data extraction jobs as Google Cloud Runs, which eliminates the need for DinD. Please refer to the Airbyte connector docs for the details. |
Thanks Sergey.
…On Mon, 10 Jun 2024 at 6:39 PM, Sergey Kulik ***@***.***> wrote:
Hi Vikas!
Please note that now you can run airbyte data extraction jobs as Google
Cloud Runs, which eliminates the need for DinD. Please refer to the Airbyte
connector docs
<https://pathway.com/developers/api-docs/pathway-io/airbyte> for the
details.
—
Reply to this email directly, view it on GitHub
<#53 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BHP3QYCXIDXJLEWUBWL6KILZGWQQDAVCNFSM6AAAAABH4CO73CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYGMZDQMJQGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi,
Is there a way to deploy pathway airbyte streaming ETL microservice to Google cloud run? If yes, how to go about it?
thanks.
The text was updated successfully, but these errors were encountered: