Eagerly materialize a chain / DAG of assets #26255
Unanswered
christeefy
asked this question in
Q&A
Replies: 1 comment
-
I could be wrong, but is it an option to set on_cron on asset3, and add deps=asset2 to it, and set asset2 with deps=asset1? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Assuming I have a chain of assets (
asset1
->asset2
->asset3
) with the following intentions:asset1
should be triggered via cronasset3
(and transitivelyasset2
) should be triggered eagerly wheneverasset1
materializedUsing the Declarative Automation API, the implementation would like this:
In the current iteration of the API (v1.9.3),
asset2
'sAutomationCondition.eager()
needs to be specified. Without it,asset3
wouldn't eagerly materialize.The issue is that this doesn't scale with many assets — data practitioners would have to review the Global Asset Graph (for complicated DAGs) to identify non-root assets and update their automation condition accordingly. Non-root assets added going forward also have to be remembered to be added with the automation condition, lest they break the "chain of eager conditions".
Is there a clean, maintainable way to set the automation condition for various assets within a group? For a target (leaf) asset, is there a way to specify eager automation conditions for it and all upstream dependencies somehow?
Beta Was this translation helpful? Give feedback.
All reactions