Emitting MaterializeResult whilst returning a value
#26198
-
ContextContextHistorically Dagster allows returning assets directly in its definition: @asset
def my_asset():
return "Hello, world!"although in the past year the above pattern is less recommended, in-lieu of each asset responsible for serialization & deserialization: @asset
def table():
df = ...
df.to_csv("...")
@asset(deps=[table])
def downstream1():
df = pd.read_csv("...")
...If I were to guess, this is to prevent anti-patterns like loading a massive table from the warehouse and passing it between assets. That said, it feels redundant to have each dependent asset worry about (de)serialization and knowledge of that upstream asset's internals. In any case, I see that ProblemIn the process of onboarding team members onto Dagster, we ran into the following use case (simplified): @asset
def latest_data_load_date():
latest_date = "..."
yield MaterializeResult(metadata={"latest_data_load_date": MetadataValue.timestamp(latest_date)})
return latest_date
@asset
def downstream(latest_data_load_date: str):
...We have two intentions with
The Error stack traceFurther investigation show that the plumbing between assets are broken with the use of And so, we have to choose between fulfilling either of our requirements for the asset, but not both. So the asks are:
Other users also have the same question, so it would be nice to gain clarification on proper usage / composition of these APIs. Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
You can switch from will follow up on the right place to put this in the docs |
Beta Was this translation helpful? Give feedback.
-
|
Also, can you please point me/us to docs on why pattern 2 was chosen in favour of pattern 1. Is it solely because of the massive table passing mentioned above or... more reasons? From my perspective, I see the benefit of both - each asset's effect is separate, it makes debugging easier if there's a usable file on disk, rather than a pickle. On the other hand, maybe for simpler use cases, writing and reading back with pandas gets added everywhere. On that topic, please consider pointing to polars instead of pandas... or at least specify both. Polars deserves to supercede pandas, and it does today, at least for a fair portion of use cases. |
Beta Was this translation helpful? Give feedback.
-
|
"pattern 1" represents Dagster managed inputs/outputs which are handled by a subsystem called IO Managers https://docs.dagster.io/concepts/io-management/io-managers We have found over the years that IO Managers often create a rough learning curve for new users. When using this pattern, new users are able to move values from asset to asset without even knowing what an IO manager is or how it works. Then at some point the default IO manager stops working and the user has to learn a whole bunch of new concepts at a time when they just want to get things running again. For this reason, we have started removing managed IO from the examples in documentation that we expect new users to encounter early on as they learn how to use Dagster. The goal is to make it so that users are introduced to it a more appropriate time, when they are better equipped to understand its value and which IO manager they may need for different assets and different deployments/environments. This is something we are still working on, you can give feedback on the new docs revamp we are working on here #24816 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @alangenfeld for the context. I found this super helpful to know that Pattern 1 is not part of a deprecated feature, but an advanced one. I think it may also be valuable to include the context you shared above somewhere in the documentation. It may be helpful for other users facing the same question. |
Beta Was this translation helpful? Give feedback.
You can switch from
MaterializeResulttoOutputwhen you want to return a value in addition tometadatawill follow up on the right place to put this in the docs