[components] Rendering engine #26468
Open
+136
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
OwenKephart
commented
Dec 13, 2024
Comment on lines
12
to
18
| class Renderer: | ||
| def __init__(self, context: Mapping[str, Any]): | ||
| self.context = context | ||
|
|
||
| def render(self, val: str) -> str: | ||
| return Template(val).render(**self.context) |
There was a problem hiding this comment.
Placeholder name here
OwenKephart
force-pushed
the
12-12-_components_rendering_engine
branch
from
f0cc93a to
cc3280b
Compare
schrockn
reviewed
Dec 13, 2024
Comment on lines
+20
to
+42
| def _should_render( | ||
| valpath: Sequence[Union[str, int]], json_schema: Optional[Mapping[str, Any]] | ||
| ) -> bool: | ||
| """For a given path to an attribute, read the assoicated metadata in the json_schema to determine if the value should be rendered.""" | ||
| if json_schema is None: | ||
| return True | ||
|
|
||
| subschema = json_schema | ||
| for el in valpath: | ||
| if isinstance(el, str): | ||
| subschema = subschema["properties"].get(el) | ||
| else: | ||
| subschema = subschema["items"] | ||
|
|
||
| if not subschema: | ||
| return False | ||
| if "$ref" in subschema: | ||
| subschema = json_schema["$defs"].get(subschema["$ref"][len(REF_BASE) :]) | ||
| if subschema.get("defer_render") is True: | ||
| return False | ||
|
|
||
| return True |
There was a problem hiding this comment.
huge fan of doing this in json schema. We should architect this so as much as possible in components frontend does not require the dagster module in process. This will make it much more amenable in the future to js frontends and UI
schrockn
reviewed
Dec 13, 2024
|
|
||
|
|
||
| def render(renderer: Renderer, val: T, target_type: Type) -> T: | ||
| json_schema = target_type.model_json_schema() if issubclass(target_type, BaseModel) else None |
There was a problem hiding this comment.
can we make this check an attribute in json schema rather than BaseModel?
There was a problem hiding this comment.
still would like to see this addresses
|
I'd like to see how this evolves upstack until I approve, but like the direction. |
OwenKephart
force-pushed
the
12-12-_components_rendering_engine
branch
from
cc3280b to
c0712c5
Compare
OwenKephart
force-pushed
the
12-12-_components_rendering_engine
branch
from
c0712c5 to
d0e42d5
Compare
This was referenced Dec 13, 2024
schrockn
reviewed
Dec 14, 2024
| def _should_render( | ||
| valpath: Sequence[Union[str, int]], json_schema: Optional[Mapping[str, Any]] | ||
| ) -> bool: | ||
| """For a given path to an attribute, read the assoicated metadata in the json_schema to determine if the value should be rendered.""" |
schrockn
reviewed
Dec 14, 2024
Comment on lines
+25
to
+42
| if json_schema is None: | ||
| return True | ||
|
|
||
| subschema = json_schema | ||
| for el in valpath: | ||
| if isinstance(el, str): | ||
| subschema = subschema["properties"].get(el) | ||
| else: | ||
| subschema = subschema["items"] | ||
|
|
||
| if not subschema: | ||
| return False | ||
| if "$ref" in subschema: | ||
| subschema = json_schema["$defs"].get(subschema["$ref"][len(REF_BASE) :]) | ||
| if subschema.get("defer_render") is True: | ||
| return False | ||
|
|
||
| return True |
There was a problem hiding this comment.
some comments here would be helpful. This just looks like magical incantations to me as someone that doesn't know a lot about json schema
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This is the initial layer for a jinja-based rendering engine for the contents of
components.yaml. To keep this diff simpler, I didn't hook this up to anythin quite yet.The key bit here is that in some cases, we want to defer the templating logic until a point in time where we will have more context available. For example, the asset translation logic for the dbt_project component needs to apply its template to each individual node, which won't be available until runtime.
Other fields need to be substituted in before the initial load, such as those being passed into resources used during the component init process.
This allows us to distinguish between these two cases by setting metadata in the generated json schema, which is then used to determine if rendering should be deferred or not.
Note that rendering is done at a per-value level, which caps the amount of craziness someone can do in these (i.e. you can't dynamically generate a nested dictionary of fields). This is generally intentional, I think once you get to that level of complexity you should really be doing that in python land, and attempting to support that sort of thing would make the mental model a lot more complicated.
How I Tested These Changes
Changelog
NOCHANGELOG