Shouldn't an out-of-date dependency trigger an otherwise up-to-date task?

[ConX]

Hi everyone,

First of all, thanks for the great tool!

I just noticed that when a dependency is out-of-date, that doesn't necessarily trigger the target task. The scenario that I have is that of a target task (T) that generates some files and requires a file generated by another task (A). The way I tried to implement that was by listing the task (A) as a dependency of (T) and also listing sources and generates for the task (T). I expected that if task (A) is out-of-date, that will trigger task (T) as well, but that isn't the case.

I assume that I can list the file generated by task (A) in the sources of (T), and that will fix the issue, but I am curious why this isn't implied, given that (A) is a dependency of (T). What is a scenario for which the current approach is desired?

Here is a minimal reproducible example that uses status instead of files:

version: "3"

tasks:
  first-dep:
    cmds:
      - echo First
    status:
      - "exit 1"

  second-dep:
    cmds:
      - echo Second
    status:
      - "exit 0"

  target:
    cmds:
      - "echo Target"
    deps:
      - first-dep
      - second-dep
    status:
      - "exit 0"

and here is the output that I get:

$task target
task: Task "second-dep" is up to date
task: [first-dep] echo First
First
task: Task "target" is up to date

[ConX]

I forgot to mention that I would work on a PR if allowing for such behavior (other by default or through an option) is something others find useful.

[ghostsquad]

Ya, you are right. If you look at traditional Makefile, dependent targets, if changed will cause parent targets to trigger.

Status and Sources is just a way to give flexibility as to how to define if this target should run, assuming it has no dependencies.

The following Makefile has three separate rules. When you run make blah in the terminal, it will build a program called blah in a series of steps:

Make is given blah as the target, so it first searches for this target

blah requires blah.o, so make searches for the blah.o target

blah.o requires blah.c, so make searches for the blah.c target

blah.c has no dependencies, so the echo command is run

The cc -c command is then run, because all of the blah.o dependencies are finished

The top cc command is run, because all the blah dependencies are finished

That's it: blah is a compiled c program

blah: blah.o 	cc blah.o -o blah # Runs third


blah.o: blah.c 	cc -c blah.c -o blah.o # Runs second

blah.c: 	echo "int main() { return 0; }" > blah.c # Runs first

https://makefiletutorial.com/

I think this is a legitimate bug now. Thanks for helping me understand this.

[ghostsquad]

The question now:

How would you indicate that the parent target (task3 in your example) should run even if it's deps are up-to-date?

How would you indicate that the parent target (task3 in your example) should not run if it's deps are up-to-date?

I'd assume also that tasks under cmd should be considered too? How does that work exactly?

Does status/sources apply to all the elements in the cmds array?

If deps or cmds are up-to-date, and you want to force a run everytime (the current behavior if you exclude sources/status), I think you would need to add:

method: none

I'd have to double check the code though.

Although I definitely think this likely should be the default behavior, it is not backwards compatible. 🙁

It would likely need to be part of a version 4 if this were to be the new default behavior.

[ConX]

Thanks for spending the time! I will attempt to answer the questions although the devs know best:

How would you indicate that the parent target (task3 in your example) should run even if it's deps are up-to-date?

That would be based on the status/sources/generates statements of task3. Although, as stated, I think that including file5 and file6 from the previous example in the sources is redundant, there could be other files (not generated by the dependencies) that when included in the sources should determine whether task3 should run. The same goes for generates and status statements. If none of the status/sources/generates is provided irrespective of whether the dependencies are up-to-date or not, task3 should always run when invoked.

How would you indicate that the parent target (task3 in your example) should not run if it's deps are up-to-date?

Following from above, if dependencies are up-to-date and all the provided sources/generates/status resolve to determine task3 an up-to-date task then that should also be skipped.

I'd assume also that tasks under cmd should be considered too? How does that work exactly?
Does status/sources apply to all the elements in the cmds array?

That's a good question I hadn't thought of! I view task under cmd more as shortcuts/functions rather than dependencies, thus I would expect those to not be considered, but I might be off here.

If deps or cmds are up-to-date, and you want to force a run everytime (the current behavior if you exclude sources/status), I think you would need to add:

As I wrote above, I think the current approach of excluding the sources/status/generates should still force the task to run everytime. Am I missing something?

A pseudo-code of how the resolution of the target/parent task (task3 in the example) I think should be handled:

1. Evaluate the dependencies of the target task. If any of the dependencies was out-of-date mark the target task as out-of-date (or tainted per Terraform).
2. When returning to evaluate task3, if it hasn't already been marked as out-of-date, evaluate its sources/generates/status as normal to determine if it should be skipped or not.

To keep the above pseudo-code simple I avoided talking about the dependencies of the dependencies, etc. These should be probably be handled by recursion where each dependency only affects its parents/ancestors. This is probably easier to explain graphically:

For example, if only tasks dep1.2.1 and dep2.2 are out-of-date, this would also force tasks dep1.2, dep1, dep2, and of course the target to be marked as out-of-date and run.

[ghostsquad]

Regarding CMD, the only way to run target deps sequentially would be with cmd.

I feel like deps should be a multi dimensional array. Outer array runs parallel, inner array(s) run sequentially, and Cmds should no longer support the task: syntax. I think that would improve readability too.

Anyways back to deps triggering a run...
Even if this change isn't backwards compatible, it feels like a bug. I would support this change.

[andreynering]

I feel like deps should be a multi dimensional array. Outer array runs parallel, inner array(s) run sequentially, and Cmds should no longer support the task: syntax

There are two reasons to keep this as it works today:

1. Changing that would be backward incompatible
2. task: is very useful because you can run them in whatever order you want, even mixed with commands. Dependencies always run first. Also, not everyone use task: for dependencies, sometimes they are used as helper tasks to run many tasks in sequence, etc

[andreynering]

Hi @ConX, thanks for reaching out (and @ghostsquad for the good inputs).

Regarding this question:

Could you provide an equivalent real-life example where the current behavior of skipping the target task is desired despite the execution of its dependencies?

I think this web app example, although a very basic example, explains what I had in mind when I initially implemented this feature:

In this case js and css are dependencies of assets, which is a dependency of build. If you look at the sources of each task, you'll see it'll automatically work as you expect, because each task has as a source the generated files of its dependencies.

It never come to my mind that someone would want to trigger a run when a dependency change in a scenario where you wouldn't add the generated files of the dependencies as the task source.

Seems that I was wrong, because with DevOps/Infra stuff you not always deal with files. Also, in the beginning status wasn't a thing, and only sources and generates existed. I also didn't know Makefiles behaved this they do.

---

All that said my thoughts are:

• There may be challenges on trying to implement this, given the way the code works currently
• Since this could possibly be backward incompatible, we probably need a global option in the Taskfile for this for now. We could consider enabling it by default on v4, and perhaps keep the option to disable this behavior
• I think it's worth opening a proper issue for this feature request (given this is a discussion). This way we could track 👍s to know if more people are interested in this before implementing it

[ghostsquad]

I've been browsing the existing issues, code base, and also using some of the my own experience from learning the tool, and I am interested in a v4. I think there's a few things that could be done to really polish the tool. I will be trying to put these thoughts and get some more ideas down in a fork in the coming weeks.

For now, I think it's best to make tasks idempotent. If you run into a limitation like this, move the logic of deciding when or when not to act, away from go-task.

[ConX]

Hi @andreynering, thanks for your detailed response. I can create an issue based on this discussion to track the popularity of this feature.

[pavetok]

It could be a local option for dependency declaration. For example

deps:
  - task: foo
    outdated: true

[MoritzWeber0]

Hi together! I have a similar use case. I have several Dockerimages that depend on each other.
Task A is a docker build command that produces the image test. Task B uses the Dockerimage test from Task A as base image. Therefore, Task B has to be rebuild if the Dockerimage of Task A changes.

I have a small workaround for it. I added the checksum file (.task/checksum/task-a) as source to Task B. Task B does then always rerun if the sources of Task A changed.