Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dry run fails when table/view/assertion to be created does not exists #1805

Open
ashish10alex opened this issue Aug 12, 2024 · 2 comments
Open
Labels

Comments

@ashish10alex
Copy link
Contributor

ashish10alex commented Aug 12, 2024

There are two possible errors relating to dry run:

Tables / views

  1. If the table created by a .sqlx file is not already materialised on BigQuery the dry run will fail due to dry run trying to setMetadata on an view/table that does not exist yet. Relevant lines of code

await this.dbadapter.setMetadata(action);

const metadata = await this.getMetadata(target);
const schemaWithDescription = addDescriptionToMetadata(
actionDescriptor.columns,
metadata.schema.fields
);

In these lines metadata for the table/view that already does not exist will be undefined and trying to access .schema attribute of undefined results in the error shown below

CleanShot 2024-08-12 at 19 16 41@2x


Assertion

Assertion dry run will face if the table it is trying to create does not already exists. Possibly due to trying to run the following query in the current code base which tries to select from a object that does not exist yet

tasks.add(Task.assertion(`select sum(1) as row_count from ${this.resolveTarget(target)}`));


Minimal example to reproduce

  1. Add the following as a new .sqlx file to a dataform project
config {
  type: "table",
  schema: "dataset_name",
  description: "any non empty description",
  tags: ["TEST_DRY_RUN"]
}


SELECT 1 AS A
UNION ALL
SELECT 1 AS B

To recreate this error for view/assertion type, the type can be replaced by either of the two

  1. Run
    dataform run --dry-run --tags="TEST_DRY_RUN"

dataform version: 3.0.2 and also tested by running it from source

@St-Hakky
Copy link

St-Hakky commented Nov 10, 2024

Hi, I'm dataform user.

I’m also encountering the same issue, so I wanted to check if there have been any updates on this.

@Ekrekr
Copy link
Contributor

Ekrekr commented Nov 11, 2024

Agreed, we shouldn't be attempting to set metadata on a dry run!

Any chance you can submit a patch yourself? https://github.com/dataform-co/dataform/blob/main/contributing.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants