Skip to content

Commit

Permalink
docs: move all expressions to a single reference doc (#1955)
Browse files Browse the repository at this point in the history
* docs: move all expressions to a single reference and improve the docs text

* Update index.md

* docs: move task run info

* fix: wording + rm handlebars

* Update index.md

* Update index.md

* Update index.md

* Update index.md

* avoid broken links

* docs: avoid broken links
  • Loading branch information
anna-geller authored Nov 20, 2024
1 parent c6a56c8 commit 60947e1
Show file tree
Hide file tree
Showing 26 changed files with 2,215 additions and 4,221 deletions.
2 changes: 0 additions & 2 deletions content/docs/04.workflow-components/01.flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,6 @@ Flows have a number of variable expressions giving you information about them dy
| `{{ flow.revision }}` | The revision of the flow. |


You can get the full list of variable expressions [here](../expressions/02.expression-types.md#flow-and-execution-expressions).

## Listeners (deprecated)

Listeners are special tasks that can listen to the current flow, and launch tasks *outside the flow*, meaning launch tasks that are not part of the flow.
Expand Down
116 changes: 89 additions & 27 deletions content/docs/04.workflow-components/01.tasks/02.taskruns.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ namespace: company.team

tasks:
- id: return
type: io.kestra.plugin.core.debug.Return
type: io.kestra.plugin.core.debug.Return
format: "{{ taskrun }}"
```
Expand All @@ -62,6 +62,8 @@ The logs show the following:
}
```

## Task Run Values

Some [Flowable Tasks](./00.flowable-tasks.md), such as [ForEach](./00.flowable-tasks.md) and [ForEachItem](./00.flowable-tasks.md#foreachitem), group tasks together. You can use the expression `{{ taskrun.value }}` to access the value for that task run.

In the example below, `foreach` will iterate twice over the values `[1, 2]`:
Expand All @@ -77,11 +79,19 @@ tasks:
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ taskrun.value }}"
message:
- "{{ taskrun }}"
- "{{ taskrun.value }}"
- "{{ taskrun.id }}"
- "{{ taskrun.startDate }}"
- "{{ taskrun.attemptsCount }}"
- "{{ taskrun.parentId }}"
```
This outputs two separate log tasks, one with `1` and the other with `2`.

You can also use the `{{ parents }}` expression to access a task run value from a parent task. Here's an example of it with `ForEach`:
### Parent Task Run Values

You can also use the `{{ parent.taskrun.value }}` expression to access a task run value from a parent task within nested flowable child tasks:

```yaml
id: loop
Expand All @@ -101,22 +111,75 @@ tasks:
then:
- id: log_parent
type: io.kestra.plugin.core.log.Log
message: "{{ parents }}"
message: "{{ parent.taskrun.value }}"
```
This will iterate through the `log` and `if` tasks twice as there are two items in `values` property.

The `log_parent` task outputs the task run value produced by `foreach` on the first iteration:
This will iterate through the `log` and `if` tasks twice as there are two items in `values` property. The `log_parent` task will log the parent task run value as `1` and then `2`.

```json
[
{
"taskrun": {
"value": "1"
}
}
]
### Parent vs. Parents in Nested Flowable Tasks

When using nested [Flowable tasks](./00.flowable-tasks.md), only the direct parent task is accessible via `taskrun.value`. To access a parent task higher up the tree, you can use the `parent` and the `parents` expressions.

The following flow shows a more complex example with nested flowable parent tasks:

```yaml
id: each_switch
namespace: company.team
tasks:
- id: simple
type: io.kestra.plugin.core.log.Log
message:
- "{{ task.id }}"
- "{{ taskrun.startDate }}"
- id: hierarchy_1
type: io.kestra.plugin.core.flow.ForEach
values: ["caseA", "caseB"]
tasks:
- id: hierarchy_2
type: io.kestra.plugin.core.flow.Switch
value: "{{ taskrun.value }}"
cases:
caseA:
- id: hierarchy_2_a
type: io.kestra.plugin.core.debug.Return
format: "{{ task.id }}"
caseB:
- id: hierarchy_2_b_first
type: io.kestra.plugin.core.debug.Return
format: "{{ task.id }}"
- id: hierarchy_2_b_second
type: io.kestra.plugin.core.flow.ForEach
values: ["case1", "case2"]
tasks:
- id: switch
type: io.kestra.plugin.core.flow.Switch
value: "{{ taskrun.value }}"
cases:
case1:
- id: switch_1
type: io.kestra.plugin.core.log.Log
message:
- "{{ parents[0].taskrun.value }}"
- "{{ parents[1].taskrun.value }}"
case2:
- id: switch_2
type: io.kestra.plugin.core.log.Log
message:
- "{{ parents[0].taskrun.value }}"
- "{{ parents[1].taskrun.value }}"
- id: simple_again
type: io.kestra.plugin.core.log.Log
message:
- "{{ task.id }}"
- "{{ taskrun.startDate }}"
```

The `parent` variable gives direct access to the first parent, while the `parents[INDEX]` gives you access to the parent higher up the tree.


::collapse{title="Task Run JSON Object Example"}
```json
{
Expand Down Expand Up @@ -152,51 +215,50 @@ The `log_parent` task outputs the task run value produced by `foreach` on the fi
"histories": [
{
"state": "CREATED",
"date": "2021-05-04T12:02:54.121836Z"
"date": "2025-05-04T12:02:54.121836Z"
},
{
"state": "RUNNING",
"date": "2021-05-04T12:02:54.121841Z"
"date": "2025-05-04T12:02:54.121841Z"
},
{
"state": "SUCCESS",
"date": "2021-05-04T12:02:54.131892Z"
"date": "2025-05-04T12:02:54.131892Z"
}
],
"duration": "PT0.010056S",
"endDate": "2021-05-04T12:02:54.131892Z",
"startDate": "2021-05-04T12:02:54.121836Z"
"endDate": "2025-05-04T12:02:54.131892Z",
"startDate": "2025-05-04T12:02:54.121836Z"
}
}
],
"outputs": {
"value": "1-1_return > s1 ⬅ 2021-05-04T12:02:53.938333Z"
"value": "1-1_return > s1 ⬅ 2025-05-04T12:02:53.938333Z"
},
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2021-05-04T12:02:53.938333Z"
"date": "2025-05-04T12:02:53.938333Z"
},
{
"state": "RUNNING",
"date": "2021-05-04T12:02:54.116336Z"
"date": "2025-05-04T12:02:54.116336Z"
},
{
"state": "SUCCESS",
"date": "2021-05-04T12:02:54.144135Z"
"date": "2025-05-04T12:02:54.144135Z"
}
],
"duration": "PT0.205802S",
"endDate": "2021-05-04T12:02:54.144135Z",
"startDate": "2021-05-04T12:02:53.938333Z"
"endDate": "2025-05-04T12:02:54.144135Z",
"startDate": "2025-05-04T12:02:53.938333Z"
}
}
```
::

Read more about it on the [Expression Usage](../../expressions/02.expression-usage.md#parent-tasks-with-flowable-tasks) page.

## Task Runs Page (EE)

Expand All @@ -208,4 +270,4 @@ If you have Kestra setup using the [Kafka and Elasticsearch backend](../../07.ar

![taskrun_view](/docs/workflow-components/taskrun_view.png)

It's similar to the [Execution View](../../08.ui/02.executions.md) but only shows Task Runs.
It's similar to the [Execution View](../../08.ui/02.executions.md) but only shows Task Runs.
1 change: 0 additions & 1 deletion content/docs/04.workflow-components/03.execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ There are a number of execution expressions which you can use inside of your flo
| `{{ execution.startDate }}` | The start date of the current execution, can be formatted with `{{ execution.startDate \| date("yyyy-MM-dd HH:mm:ss.SSSSSS") }}`. |
| `{{ execution.originalId }}` | The original execution ID, this id will never change even in case of replay and keep the first execution ID. |

You can read more about them on the [expressions page](../expressions/02.expression-types.md).

## Execute a flow from the UI

Expand Down
4 changes: 2 additions & 2 deletions content/docs/04.workflow-components/04.variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Since 0.14, Variables are no longer rendered recursively. You can read more abou

## Dynamic Variables

If you want to have an expression inside of your variable, you will need to wrap it in `render` when you use it in a task.
If you want to have an expression inside of your variable, you will need to wrap it in `render` when you use it in a task.

For example, this variable will only display the current time in the log message when wrapped in `render`. Otherwise, the log message will just contain the expression as a string:

Expand All @@ -68,7 +68,7 @@ You will need to wrap the variable expression in `render` every time you want to

### How do I escape a block in Pebble syntax to ensure that it won't be parsed?

To ensure that a block of code won't be parsed by Pebble, you can use the `{% raw %}` and `{% endraw %}` [Pebble tags](../expressions/06.tag.md#raw). For example, the following Pebble expression will return the string `{{ myvar }}` instead of the value of the `myvar` variable:
To ensure that a block of code won't be parsed by Pebble, you can use the `{% raw %}` and `{% endraw %}` [Pebble tags](../expressions/index.md#tag). For example, the following Pebble expression will return the string `{{ myvar }}` instead of the value of the `myvar` variable:

```yaml
{% raw %}{{ myvar }}{% endraw %}
Expand Down
8 changes: 4 additions & 4 deletions content/docs/04.workflow-components/06.outputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ tasks:
In the example above, the first task produces an output based on the task property `format`. This output attribute is then used in the second task `message` property.

The expression `{{ outputs.produce_output.value }}` references the previous task output attribute. You can read more about the expression syntax on the [Using Expressions](../expressions/02.expression-usage.md) page.
The expression `{{ outputs.produce_output.value }}` references the previous task output attribute.

::alert{type="info"}
In the example above, the **Return** task produces an output attribute `value`. Every task produces different output attributes. You can look at each task outputs documentation or use the **Outputs** tab of the **Executions** page to find out about specific task output attributes.
Expand Down Expand Up @@ -133,7 +133,7 @@ The **Outputs** tab would contain the output for each of the inner task.

### Loop over a list of JSON objects

On loop, the `value` is always a JSON string, so the `{{ taskrun.value }}` is the current element as JSON string. If you want to access properties, you need to use the [json function](../expressions/04.function.md#fromjson) to have a proper object and to access each property easily.
Within the loop, the `value` is always a JSON string, so the `{{ taskrun.value }}` is the current element as JSON string. To access properties, you need to wrap it in the `fromJson()` function to have a JSON object allowing to access each property easily.

```yaml
id: loop_sequentially_over_list
Expand All @@ -148,7 +148,7 @@ tasks:
tasks:
- id: inner
type: io.kestra.plugin.core.debug.Return
format: "{{ json(taskrun.value).key }} > {{ json(taskrun.value).value }}"
format: "{{ fromJson(taskrun.value).key }} > {{ fromJson(taskrun.value).value }}"
```


Expand Down Expand Up @@ -238,7 +238,7 @@ tasks:

When there are multiple levels of [EachSequential](/plugins/core/tasks/flows/io.kestra.plugin.core.flow.EachSequential) tasks, you can use the `parents` variable to access the `taskrun.value` of the parent of the current EachSequential. For example, for two levels of EachSequential you can use `outputs.sibling[parents[0].taskrun.value][taskrun.value].value`.

The latter can become very complex when parents exist (multiple imbricated EachSequential). For this, you can use the special [currentEachOutput](../expressions/04.function.md#currenteachoutput) function. No matter the number of parents, the following example will retrieve the correct output attribute: `currentEachOutput(outputs.sibling).value` thanks to this function.
The latter can become very complex when parents exist (multiple imbricated EachSequential). For this, you can use the `currentEachOutput()` function. No matter the number of parents, the following example will retrieve the correct output attribute: `currentEachOutput(outputs.sibling).value` thanks to this function.

::alert{type="warning"}
Accessing sibling task outputs is impossible on [Parallel](/plugins/core/tasks/flows/io.kestra.plugin.core.flow.Parallel) or [EachParallel](/plugins/core/tasks/flows/io.kestra.plugin.core.flow.EachParallel) as they run tasks in parallel.
Expand Down
2 changes: 1 addition & 1 deletion content/docs/05.concepts/04.secret.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To retrieve secrets in a flow, use the `secret()` function, e.g. `"{{ secret('AP

Your flows often need to interact with external systems. To do that, they need to programmatically authenticate using passwords or API keys. Secrets help you securely store such variables and avoid hard-coding sensitive information within your workflow code.

You can leverage the `secret()` function, described in the [function section](../expressions/04.function.md#secret), to retrieve sensitive variables within your flow code.
You can leverage the `secret()` function to retrieve sensitive variables within your flow code.


## Secrets in the Enterprise Edition
Expand Down
2 changes: 0 additions & 2 deletions content/docs/05.concepts/06.pebble.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,6 @@ Pebble can be very useful to make small transformation on the fly - without the

For instance, we can use the `date` filter to format date values: `'{{ inputs.my_date | date("yyyyMMdd") }}'`

You can find more documentation on the `date` filter on the [Expressions page](../expressions/03.filter/temporal.md#date)

## Coalesce operator to conditionally use trigger or execution date

Most of the time, a flow will be triggered automatically. Either on schedule or based on external events. It’s common to use the date of the execution to process the corresponding data and make the flow dependent on time.
Expand Down
10 changes: 5 additions & 5 deletions content/docs/05.concepts/11.storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ tasks:
to: "/upload/file.ion"
```

If you need to access data from the internal storage, you can use the Pebble [read](../expressions/04.function.md#read) function to read the file's content as a string.
If you need to access data from the internal storage, you can use the `read()` function to read the file's content as a string.

Dedicated tasks allow managing the files stored inside the internal storage:
- [Concat](/plugins/core/tasks/storages/io.kestra.plugin.core.storage.Concat): concat multiple files.
Expand Down Expand Up @@ -161,9 +161,9 @@ As we can see, when we `Set` a new value for `user_name`, we have to use another

## Processing data

You can make basic data processing thanks to variables processing offered by the Pebble templating engine, see [variables basic usage](../expressions/index.md).
For basic data processing, you can leverage Kestra's [Pebble templating engine](../expressions/index.md).

But these are limited, and you may need more powerful data processing tools; for this, Kestra offers various data processing tasks like file transformations or scripts.
For more complex data transformations, Kestra offers various data processing plugins incl. transform tasks or custom scripts.

### Converting files

Expand Down Expand Up @@ -440,9 +440,9 @@ The same syntax applies to SQL queries, custom scripts, and many more. Check the

#### How to read a file from the internal storage as a JSON object?

There is a [Pebble function](../expressions/04.function.md#fromjson) called `{{ json(myvar) }}` and a [Pebble transformation filter](../expressions/03.filter/json.md) that you can apply using `{{ myvar | json }}`.
You can use the Pebble function `{{ fromJson(myvar) }}` and a `{{ myvar | toJson }}` filter to process JSON data.

::collapse{title="The json() function"}
::collapse{title="The fromJson() function"}

The function is used to convert a string to a JSON object. For example, the following Pebble expression will convert the string `{"foo": [666, 1, 2]}` to a JSON object and then return the first value of the `foo` key, which is `42`:

Expand Down
2 changes: 0 additions & 2 deletions content/docs/11.migration-guide/0.14.0/recursive-rendering.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,6 @@ triggers:
cron: "*/1 * * * *"
```
Check the [`render()` function's documentation](../../expressions/04.function.md#render) for more details about how to use that function and when to use it.

---
## Migrating a 0.13.0 flow to the new rendering behavior in 0.14.0
Expand Down
Loading

0 comments on commit 60947e1

Please sign in to comment.