Skip to content

Commit

Permalink
feat(docs): replace deprecated Echo with the Log task (#89)
Browse files Browse the repository at this point in the history
Using a deprecated task in examples is confusing.

Also some MD fixes...
  • Loading branch information
yuri1969 authored Oct 4, 2023
1 parent 3e5e66d commit 9b1915c
Show file tree
Hide file tree
Showing 14 changed files with 67 additions and 57 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/index.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
{ "index" : { "_index" : "kestra_users", "_id" : "john" } }
{"id":"john","username":"[email protected]","auths":[{"type":"io.kestra.ee.models.auths.BasicAuth","salt":"0ghOqlkf41KpHl0D5Hf1Qhu77uiQ4ez3","password":"a74f765210a72aeb690824c1805bc364b71769362afbf7cfd9b1fb7b283902b41f630aee940e3b38058a43552c964293e802a9276e1d501455b39616802953b5","uid":"BasicAuth"}],"groups":["admin"],"deleted":false}
{ "index" : { "_index" : "kestra_flows", "_id" : "io.kestra.terraform.data_simple" } }
{"id":"simple","namespace":"io.kestra.terraform.data","revision":1,"tasks":[{"id":"t1","type":"io.kestra.core.tasks.debugs.Echo","format":"first {{task.id}}","level":"TRACE"},{"id":"t2","type":"io.kestra.core.tasks.debugs.Echo","disabled":true,"format":"second {{task.type}}","level":"WARN"},{"id":"t3","type":"io.kestra.core.tasks.debugs.Echo","format":"third {{flow.id}}","level":"ERROR"}],"deleted":false}
{"id":"simple","namespace":"io.kestra.terraform.data","revision":1,"tasks":[{"id":"t1","type":"io.kestra.core.tasks.log.Log","message":"first {{task.id}}","level":"TRACE"},{"id":"t2","type":"io.kestra.core.tasks.log.Log","disabled":true,"message":"second {{task.type}}","level":"WARN"},{"id":"t3","type":"io.kestra.core.tasks.log.Log","message":"third {{flow.id}}","level":"ERROR"}],"deleted":false}
{ "index" : { "_index" : "kestra_namespaces", "_id" : "io.kestra.terraform.data" } }
{"id":"io.kestra.terraform.data","description":"My Kestra Namespace data","deleted":false}
{ "index" : { "_index" : "kestra_templates", "_id" : "io.kestra.terraform.data_simple" } }
{"id":"simple","namespace":"io.kestra.terraform.data","tasks":[{"id":"t1","type":"io.kestra.core.tasks.debugs.Echo","format":"first {{task.id}}","level":"TRACE"},{"id":"t2","type":"io.kestra.core.tasks.debugs.Echo","disabled":true,"format":"second {{task.type}}","level":"WARN"},{"id":"t3","type":"io.kestra.core.tasks.debugs.Echo","format":"third {{flow.id}}","level":"ERROR"}],"deleted":false}
{"id":"simple","namespace":"io.kestra.terraform.data","tasks":[{"id":"t1","type":"io.kestra.core.tasks.log.Log","message":"first {{task.id}}","level":"TRACE"},{"id":"t2","type":"io.kestra.core.tasks.log.Log","disabled":true,"message":"second {{task.type}}","level":"WARN"},{"id":"t3","type":"io.kestra.core.tasks.log.Log","message":"third {{flow.id}}","level":"ERROR"}],"deleted":false}
{ "index" : { "_index" : "kestra_bindings", "_id" : "john" } }
{"id":"john","type":"USER","externalId":"john","roleId":"admin","deleted":false}
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@
![Kestra orchestrator](https://kestra.io/ui.gif)


## Documentation

* The official Kestra documentation can be found under: [kestra.io](https://kestra.io)
* This Terraform provider documentation can be found [here](https://kestra.io/docs/terraform/)

## License
Apache 2.0 © [Kestra Technologies](https://kestra.io)

Apache 2.0 © [Kestra Technologies](https://kestra.io)
44 changes: 27 additions & 17 deletions docs/guides/working-with-yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,21 @@ page_title: "Working with Yaml"

# Working with Yaml

Most of kestra ressource need to be described as Yaml like [kestra_flow](../resources/flow.md) & [kestra_template](../resources/template.md).
Most of Kestra resource need to be described as Yaml like [kestra_flow](../resources/flow.md) & [kestra_template](../resources/template.md).

We have chosen to use a full yaml in terraform definition since the structure is recursive and dynamic, so it can't be described using terraform internal schema.

There is 2 ways (for flow) to handle yaml:
* use `keep_original_source = true` method: the default one, the raw yaml will be send and save in Kestra.
* use `keep_original_source = false` method: the yaml will be encoded in json before behind to the server, so comment and indent will be handle by the server
There is 2 ways (for flow) to handle yaml:

!> Take care with `keep_original_source = false` that this terraform provider is not aware of task & plugins. It can't know default values of properties, and most of convertion logic done by Kestra Server. If you see diff that **is always present** (even just after apply), your flow on terraform must have a minor difference return from the server. In this case, **copy the source from Kestra UI** in your terraform files to avoid these difference.
* use `keep_original_source = true` method: the default one, the raw yaml will be send and save in Kestra.
* use `keep_original_source = false` method: the yaml will be encoded in json before behind to the server, so comment and indent will be handle by the server

!> Take care with `keep_original_source = false` that this terraform provider is not aware of task & plugins. It can't know default values of properties, and most of conversion logic done by Kestra Server. If you see diff that **is always present** (even just after apply), your flow on terraform must have a minor difference return from the server. In this case, **copy the source from Kestra UI** in your terraform files to avoid these difference.

There is in terraform a lot of function that allow to work properly with this yaml content :

## Simple multiline string example

You can use simple terraform multiline string with [Heredoc String](https://www.terraform.io/docs/language/expressions/strings.html#heredoc-strings) :

```terraform
Expand All @@ -33,14 +34,15 @@ inputs:
tasks:
- id: t2
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
EOT
}
```

## External files

Better will be to use a [file function](https://www.terraform.io/docs/language/functions/file.html). Just create a file `.yml` near your terraform `.tf` and include the whole file in your resource:

```yaml
Expand All @@ -51,8 +53,8 @@ inputs:

tasks:
- id: t2
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
EOT
```
Expand All @@ -66,20 +68,24 @@ resource "kestra_flow" "example" {
```

## External files with template

Even better will be to use a [templatefile function](https://www.terraform.io/docs/language/functions/templatefile.html) that will allow more complex flows to be more readable. You can include some external external and this one can also include other file.

!> Take care about the indent functon that need to fit your actual flow ident. Terraform don't know anything about your yaml (it's a simple string), so you need to handle properly the indent count by yourself using the [indent function](https://www.terraform.io/docs/language/functions/indent.html)

### Dealing with included yaml string

Imagine a flow that will query an external database. Embedding the full query can lead to very long flow definition. In the case you can use `templatefile` to allow inclusion of an external files from the yaml.

Create a sql file:

```sql
SELECT *
SELECT *
FROM ....
```

Create the yaml file for the flow:

```yaml
tasks:
- id: "query"
Expand All @@ -92,7 +98,8 @@ tasks:
fetchOne: true
```
And finaly create the resource invoking the `templatefile`:
And finally create the resource invoking the `templatefile`:

```terraform
resource "kestra_flow" "example" {
namespace = "io.kestra.mynamespace"
Expand All @@ -111,30 +118,33 @@ tasks:
username: postgres
password: pg_passwd
sql: |
SELECT *
SELECT *
FROM ....
fetchOne: true
```

### Include full yaml part
By the same way, you can also include a full yaml specs inside anothers one.

By the same way, you can also include a full yaml specs inside another one.

Create 2 yaml files:

```yaml
id: t1
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
```

```yaml
id: t2
type: io.kestra.core.tasks.debugs.Echo
format: second {{task.id}}
type: io.kestra.core.tasks.log.Log
message: second {{task.id}}
level: TRACE
```

Create the yaml file for the flow:

```yaml
tasks:
- ${indent(4, file("t1.yml"))}
Expand Down
8 changes: 4 additions & 4 deletions docs/resources/flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@ variables:
tasks:
- id: t2
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
taskDefaults:
- type: io.kestra.core.tasks.debugs.Echo
- type: io.kestra.core.tasks.log.Log
values:
format: third {{flow.id}}
message: third {{flow.id}}
EOT
}
```
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/namespace.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ k2:
v1: 1
EOT
task_defaults = <<EOT
- type: io.kestra.core.tasks.debugs.Echo
- type: io.kestra.core.tasks.log.Log
values:
format: first {{flow.id}}
message: first {{flow.id}}
- type: io.kestra.core.tasks.debugs.Return
values:
format: first {{flow.id}}
Expand Down
4 changes: 2 additions & 2 deletions docs/resources/template.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ resource "kestra_template" "example" {
content = <<EOT
tasks:
- id: t2
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
EOT
}
Expand Down
8 changes: 4 additions & 4 deletions examples/resources/kestra_flow/resource.tf
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ variables:
tasks:
- id: t2
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
taskDefaults:
- type: io.kestra.core.tasks.debugs.Echo
- type: io.kestra.core.tasks.log.Log
values:
format: third {{flow.id}}
message: third {{flow.id}}
EOT
}
4 changes: 2 additions & 2 deletions examples/resources/kestra_namespace/resource.tf
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ k2:
v1: 1
EOT
task_defaults = <<EOT
- type: io.kestra.core.tasks.debugs.Echo
- type: io.kestra.core.tasks.log.Log
values:
format: first {{flow.id}}
message: first {{flow.id}}
- type: io.kestra.core.tasks.debugs.Return
values:
format: first {{flow.id}}
Expand Down
4 changes: 2 additions & 2 deletions examples/resources/kestra_template/resource.tf
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ resource "kestra_template" "example" {
content = <<EOT
tasks:
- id: t2
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE
EOT
}
16 changes: 8 additions & 8 deletions internal/provider/resource_flow_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@ func TestAccResourceFlow(t *testing.T) {
"tasks:",
" - ${indent(4, file(\"/tmp/unit-test/t1.yml\"))}",
"taskDefaults:",
" - type: io.kestra.core.tasks.debugs.Echo",
" - type: io.kestra.core.tasks.log.Log",
" values:",
" format: third {{flow.id}}",
" message: third {{flow.id}}",
"inputs:",
" - name: my-value",
" type: STRING",
Expand Down Expand Up @@ -73,13 +73,13 @@ func TestAccResourceFlow(t *testing.T) {
concat(
"tasks:",
" - id: t2",
" type: io.kestra.core.tasks.debugs.Echo",
" format: first {{task.id}}",
" type: io.kestra.core.tasks.log.Log",
" message: first {{task.id}}",
" level: TRACE",
"taskDefaults:",
" - type: io.kestra.core.tasks.debugs.Echo",
" - type: io.kestra.core.tasks.log.Log",
" values:",
" format: third {{flow.id}}",
" message: third {{flow.id}}",
"inputs:",
" - name: my-value",
" type: STRING",
Expand Down Expand Up @@ -183,8 +183,8 @@ func TestAccIncohrenceResourceFlow(t *testing.T) {
"id: \"wrong-id\"",
"tasks:",
" - id: t2",
" type: io.kestra.core.tasks.debugs.Echo",
" format: first {{task.id}}",
" type: io.kestra.core.tasks.log.Log",
" message: first {{task.id}}",
" level: TRACE",
),
false,
Expand Down
8 changes: 4 additions & 4 deletions internal/provider/resource_namespace_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ func TestAccResourceNamespace(t *testing.T) {
" v1: 1",
),
concat(
"- type: io.kestra.core.tasks.debugs.Echo",
"- type: io.kestra.core.tasks.log.Log",
" values:",
" format: first {{flow.id}}",
" message: first {{flow.id}}",
"- type: io.kestra.core.tasks.debugs.Return",
" values:",
" format: first {{flow.id}}",
Expand All @@ -50,9 +50,9 @@ func TestAccResourceNamespace(t *testing.T) {
"k1: 1",
),
concat(
"- type: io.kestra.core.tasks.debugs.Echo",
"- type: io.kestra.core.tasks.log.Log",
" values:",
" format: first {{flow.id}}",
" message: first {{flow.id}}",
"- type: io.kestra.core.tasks.debugs.Return",
" values:",
" format: second {{flow.id}}",
Expand Down
8 changes: 4 additions & 4 deletions internal/provider/resource_template_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ func TestAccTemplate(t *testing.T) {
concat(
"tasks:",
" - id: t1",
" type: io.kestra.core.tasks.debugs.Echo",
" format: first {{task.id}}",
" type: io.kestra.core.tasks.log.Log",
" message: first {{task.id}}",
" level: TRACE",
),
),
Expand All @@ -44,8 +44,8 @@ func TestAccTemplate(t *testing.T) {
concat(
"tasks:",
" - id: t2",
" type: io.kestra.core.tasks.debugs.Echo",
" format: first {{task.id}}",
" type: io.kestra.core.tasks.log.Log",
" message: first {{task.id}}",
" level: TRACE",
),
),
Expand Down
4 changes: 2 additions & 2 deletions internal/resources/bigint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ tasks:
format: second {{ execution.id }}

- id: t3
type: io.kestra.core.tasks.debugs.Echo
format: third all optional args {{ outputs.t2.value }}
type: io.kestra.core.tasks.log.Log
message: third all optional args {{ outputs.t2.value }}
timeout: PT1H
retry:
maxAttempt: 3600000
Expand Down
4 changes: 2 additions & 2 deletions internal/resources/t1.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
id: t1
type: io.kestra.core.tasks.debugs.Echo
format: first {{task.id}}
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE

0 comments on commit 9b1915c

Please sign in to comment.