Skip to content

Commit

Permalink
Review comments and fix some tests
Browse files Browse the repository at this point in the history
  • Loading branch information
jedcunningham committed Dec 13, 2024
1 parent 5b263b7 commit 9e253a0
Show file tree
Hide file tree
Showing 4 changed files with 30 additions and 4 deletions.
21 changes: 19 additions & 2 deletions airflow/config_templates/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2657,11 +2657,28 @@ usage_data_collection:
dag_bundles:
description: |
Configuration for the DAG bundles. This allows Airflow to load DAGs from different sources.
Airflow will consume all options added to this section. Below you will see only the default,
``dags_folder``. The option name is the bundle name and the value is a json object with the following
keys:
- classpath: The classpath of the bundle class
- kwargs: The keyword arguments to pass to the bundle class
- refresh_interval: The interval in seconds to refresh the bundle from its source.
For example, to add a new bundle named ``hello`` to my Airflow instance, add the following to your
airflow.cfg (this is just an example, the classpath and kwargs are not real):
.. code-block:: ini
[dag_bundles]
hello: {classpath: "airflow.some.classpath", kwargs: {"hello": "world"}, refresh_interval: 60}
options:
dags_folder:
description: |
This is the default DAG bundle that loads DAGs from the traditional `[core] dags_folder`.
It can be disabled by setting it to an empty string.
This is the default DAG bundle that loads DAGs from the traditional ``[core] dags_folder``.
By default, ``refresh_interval`` it set to ``[scheduler] dag_dir_list_interval``, but that can be
overridden here if desired.
Parsing DAGs from the DAG folder can be disabled by setting this option to an empty string.
version_added: ~
type: string
example: ~
Expand Down
2 changes: 1 addition & 1 deletion airflow/dag_processing/bundles/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@


class DagBundlesManager(LoggingMixin):
"""Manager for Dag Bundles."""
"""Manager for DAG bundles."""

@property
def bundle_configs(self) -> dict[str, dict]:
Expand Down
10 changes: 9 additions & 1 deletion airflow/models/dagbundle.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,15 @@


class DagBundleModel(Base):
"""A table for DAG Bundle information."""
"""
A table for storing DAG bundle metadata.
We track the following information about each bundle, as it can be useful for
informational purposes and for debugging:
- enabled: Is the bundle currently found in configuration?
- latest_version: The latest version Airflow has seen for the bundle.
- last_refreshed: When the bundle was last refreshed.
"""

__tablename__ = "dag_bundle"
name = Column(StringID(), primary_key=True)
Expand Down
1 change: 1 addition & 0 deletions tests/dag_processing/bundles/test_dag_bundle_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ def clear_db():
clear_db_dag_bundles()


@pytest.mark.db_test
def test_sync_bundles_to_db(clear_db):
bundle_manager = DagBundlesManager()

Expand Down

0 comments on commit 9e253a0

Please sign in to comment.