Investigate Using an image instead of building tools each run

[pwelch]

Description

It looks like we run go build on some of the tools used in the probers. It might be worth investigating if we can just pull those in from a prebuilt image each run which would save us time on each run.

[evankanderson]

In particular, rekor-fulcio-e2e spends about 2-3 minutes in "Install (crane, rekor-cli, cosign) tools", which runs:

make -C prober/ crane rekor-cli cosign

If we can publish a docker image (say, nightly) of these, it should reduce the running time from 2-3 minutes to <10s. Since we run 2 probes every 10m, this is about 2 * 6 * 24 = 288 minutes / day of CI time that we'd claw back.

[evankanderson]

@sallyom -- did you get started on this, or do you mind if I take it?

[bobcallaway]

I've tried several times to optimize the module caching to speed this up. it seems to work for a bit, then eventually get worse :/

[evankanderson]

I poked at this a bit, and the one tricky part seems to be the nick-fields/retry action:

https://github.com/sigstore/sigstore-probers/blob/main/.github/workflows/reusable-prober.yml#L90

I may see about re-implementing that elsewhere, but it's a medium-sized ball of retries.

[evankanderson]

... and I just realized I could copy the built go binaries from the built container to the workspace, which should be quick and allow us to use nick-fields/retry for the commands that need to execute.

[evankanderson]

We will still need:

• verify
• crane
• rekor-cli
• cosign