Skip to content

Commit

Permalink
feat(docs): improve datasource configuration documentation (#546)
Browse files Browse the repository at this point in the history
Fixes #485
  • Loading branch information
loicmathieu authored Sep 28, 2023
1 parent 3f7c073 commit c5b930d
Showing 1 changed file with 38 additions and 32 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@
title: Database configuration
---

First, you need to configure Kestra to use a database for its Queue and Repository, for example for MySQL:
First, you need to configure Kestra to use a database for its Queue and Repository, for example for PostgreSQL:

```yaml
kestra:
queue:
type: mysql
type: postgres
repository:
type: mysql
type: postgres
```
Currently, Kestra supports three databases: H2, MySQL, and PostgreSQL. H2 can be convenient for local development, but MySQL or PostgreSQL must be used in production.
Expand All @@ -19,48 +19,46 @@ The minimum version for PostgreSQL is 14.
::
::alert{type="info"}
If you experience performance issues when using PostgreSQL, you can tune the cost optimizer parameter `random_page_cost=1.1`, which should make PostgreSQL use the right index for the queues table. You can also configure `kestra.queue.postgres.disable-seq-scan=true` so that Kestra turns off sequential scans on the queue polling query forcing Postgres to use the index.
If you experience performance issues when using PostgreSQL, you can tune the cost optimizer parameter `random_page_cost=1.1`, which should make PostgreSQL use the right index for the queues table. You can also configure `kestra.queue.postgres.disable-seq-scan=true` so that Kestra turns off sequential scans on the queue polling query forcing PostgreSQL to use the index.
::

## Minimal configuration
The most important thing is to configure the way Kestra connects to a database, what is called a datasource.

The most important thing is to configure the way Kestra connects to a database, what is called a datasource. Under the cover, Kestra uses The [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool ; each configuration options described inside the HikariCP documentation can be configured.

Here is a minimal configuration for MySQL:
Here is a minimal configuration for PostgreSQL:
```yaml
kestra:
queue:
type: mysql
type: postgres
repository:
type: mysql
type: postgres
datasources:
mysql:
url: jdbc:mysql://localhost:3306/kestra
driverClassName: com.mysql.cj.jdbc.Driver
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
dialect: MYSQL
```


Here is a minimal configuration for PostgreSQL:
Here is a minimal configuration for MySQL:
```yaml
kestra:
queue:
type: postgres
type: mysql
repository:
type: postgres
type: mysql
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driverClassName: org.postgresql.Driver
mysql:
url: jdbc:mysql://localhost:3306/kestra
driverClassName: com.mysql.cj.jdbc.Driver
username: kestra
password: k3str4
dialect: MYSQL
```


Here is a minimal configuration for H2:
```yaml
kestra:
Expand All @@ -77,15 +75,25 @@ datasources:
driverClassName: org.h2.Driver
```

## Connection pool size

The total number of connections opened to the database will depend on the [deployment architecture](../../08.architecture.md#deployment-architecture) used. Each Kestra instance will open a pool of up to the `maximum-pool-size` setting of 10 by default, with a minimum size of the `minimum-idle` setting of 10 by default.

For example:
- If you deploy Kestra as a standalone server, it will open 10 connections to the database.
- If you deploy each Kestra component separately, it will open 40 connections to the database (10 per component).
- If you deploy each Kestra component separately with 3 replicas, it will open 120 connections to the database.

Usually, the default connection pool sizing is enough, as HikariCP is optimized to use a low number of connections.

## Datasource properties

Since Kestra is built with [Micronaut](https://micronaut.io) and [HikariCP](https://github.com/brettwooldridge/HikariCP), many configuration options are available to configure the datasource and the connection pool:
Here are the datasource configuration properties, for more information read the [HikariCP configuration](https://github.com/brettwooldridge/HikariCP#gear-configuration-knobs-baby) documentation:


| Properties | Type | Description |
|-------------------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| `url` | String | The jdbc connection string. |
| `url` | String | The JDBC connection string. |
| `catalog` | String | The default schema name to be set on connections. |
| `schema` | String | The default catalog name to be set on connections. |
| `username` | String | The default username used. |
Expand All @@ -96,33 +104,31 @@ Since Kestra is built with [Micronaut](https://micronaut.io) and [HikariCP](http
| `connection-test-query` | String | The SQL query to be executed to test the validity of connections. |
| `connection-timeout` | Long | The maximum number of milliseconds that a client will wait for a connection from the pool. |
| `idle-timeout` | Long | The maximum amount of time (in milliseconds) that a connection is allowed to sit idle in the pool. |
| `minimum-idle` | Long | The minimum number of idle connections that HikariCP tries to maintain in the pool, including both idle and in-use connections. |
| `minimum-idle` | Long | The minimum number of idle connections that HikariCP tries to maintain in the pool, including both idle and in-use connections. Defaults to the value of `maximum-pool-size` |
| `initialization-fail-timeout` | Long | The pool initialization failure timeout. |
| `leak-detection-threshold` | Long | The amount of time that a connection can be out of the pool before a message is logged indicating a possible connection leak. |
| `maximum-pool-size` | Int | The maximum size that the pool is allowed to reach, including both idle and in-use connections. |
| `maximum-pool-size` | Int | The maximum size that the pool is allowed to reach, including both idle and in-use connections. Defaults to 10. |
| `max-lifetime` | Long | The maximum lifetime of a connection in the pool. |
| `validation-timeout` | Long | The maximum number of milliseconds that the pool will wait for a connection to be validated as alive. |


Here is the default HikariCP configuration:

```yaml
transaction-isolation: default
transaction-isolation: default # driver default
pool-name: HikariPool-<Generated>
connection-init-sql: null
connection-test-query: null
connection-timeout: 30000
idle-timeout: 600000
minimum-idle: 10
connection-timeout: 30000 # 30 seconds
idle-timeout: 600000 # 10 minutes
minimum-idle: 10 # same as maximum-pool-size
initialization-fail-timeout: 1
leak-detection-threshold: 0
maximum-pool-size: 10 # Generates 40 connections
max-lifetime: 1800000
maximum-pool-size: 10
max-lifetime: 1800000 # 30 minutes
validation-timeout: 5000
```

Implementation details are accessible [here](https://github.com/brettwooldridge/HikariCP/blob/master/src/main/java/com/zaxxer/hikari/HikariConfig.java).

## Queues configuration

### `kestra.jdbc.queues`
Expand Down

1 comment on commit c5b930d

@vercel
Copy link

@vercel vercel bot commented on c5b930d Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.