shadowJar breaks Gradle artifact transforms

[DreierF]

Shadow Version

8.1.1

Gradle Version

8.3

Expected Behavior

I can use artifact transforms on my dependencies in combination with the shadow plugin to shade the result of the transformed dependencies.

Actual Behavior

When the runtimeClasspath requires an artifact transform running the shadowJar task results in

❯ ./gradlew clean shadowJar

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':app:shadowJar'.
> Could not resolve all files for configuration ':app:runtimeClasspath'.
   > Failed to transform lib.jar (project :lib) to match attributes {artifactType=jar, custom-transformed=true, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=17, org.gradle.libraryelements=jar, org.gradle.usage=java-runtime}.
      > Execution failed for CustomTransformAction: .../reproduction-shadow-artifact-transform/lib/build/libs/lib.jar.
         > Transform output .../reproduction-shadow-artifact-transform/lib/build/libs/lib.jar must exist.

According to @jjohannes this is caused by dependency resolution being triggered too early. There is only a particular window between the "main" configuration time and before any task starts in which transforms should/can run. But Gradle does not check that well and you get unexpected/inconsistent errors. This resolve() is triggered during configuration time.
The resolvedConfiguration access here seems to be the problematic part.
That should not be done and instead the filtering implemented there should be done lazily at a later point. It breaks transforms.

Reproduction

A full minimal reproduction setup can be found here:
reproduction-gradle-shadow-artifact-transform.
It consists of an app module and a lib module and a NoOp artifact transform. The error can be reproduced with ./gradlew clean shadowJar.

Workaround

In case other people run into this: I worked around the issue for now by running ./gradlew jar && ./gradlew shadowJar in CI, because it firsts builds the expected jar files and then can pick them up during the second invocation. This is quite a hacky solution, but the only thing I could get to work.

[jjohannes]

@DreierF I re-discovered the workaround for this issue I did in another build:

tasks.withType<ShadowJar>().configureEach {
    // Defer the resolution  of 'runtimeClasspath'. This is an issue in the shadow
    // plugin that it automatically accesses the files in 'runtimeClasspath' while
    // Gradle is building the task graph. The three lines below work around that.
    inputs.files(project.configurations.runtimeClasspath)
    configurations = emptyList()
    doFirst { configurations = listOf(project.configurations.runtimeClasspath.get()) }
}

[jjohannes]

A better workaround that also works with configuration cache:

tasks.withType<ShadowJar>().configureEach {
    // Do not resolve too early through 'dependencyFilter'
    dependencyFilter = NoResolveDependencyFilter()
}

class NoResolveDependencyFilter : DefaultDependencyFilter(project) {
    override fun resolve(configuration: FileCollection): FileCollection {
        return configuration
    }
}

[DanielThomas]

We've run into this too. It occurs to me both the resolution of artifacts at configuration time, and the problem with the missing inputs could be addressed by returning a detached configuration that extendsFrom the requested configurations.

That will let Gradle defer calculation of the classpath input until execution time, and also ensure that implicit dependencies on the tasks that produce the artifacts are wired.

[DanielThomas]

Related upstream issue: gradle/gradle#26155

[jjohannes]

Thanks for the issue link @DanielThomas. I have been searching for that one.