Faster hashing in gatsby core so sites with large files don't get bogged down during build

[FraserThompson]

If there are many large files in the project, builds will hang for a long time between the load plugins and onPreInit steps, and this time goes unreported by the build timer.

For example when building a production build or running the dev server for my site with a lot of large MP3 files, it hangs after the load plugins step for around an hour, and the next step (onPreInit) reports that it took less than a second.

Once it's finished, the total bootstrap time is reported as 3614.893s, but summing the times reported by the build steps only adds up to 306.684s.

After investigating by adding timers to the tasks between those steps, I discovered that 3384s was being spent on the removeStaleJobs function in initialize.ts. Delving deeper I saw that this function creates an SHA1 hash of each file synchronously. This is fine for small files, but for large files it will take a long time, and since it's synchronous if there are many large files it'll stack up.

(This is similar to the issue which we solved here by adding an optional parameter to enable a faster but slightly less robust hashing method: #37425)

I have played with changing createFileHash in utils/jobs/manager.js so that it uses the same method as fastHash in gatsby-source-filesystem (ie. stat the file then sum the mtimeMs string and ino string) and it drastically cut my build time:

Method	removeStaleJobs	Total
SHA1	3384s	3614s
Fast	15s	615s

Would we be able to add a flag to enable faster hashing for gatsby core as well? I'm not sure if this has any wider implications since I'm not 100% sure what it's doing, but if the purpose is to just generate a digest to determine if a file has changed I can't see why it would cause problems. I'm happy to submit a PR though.

Additionally I'm not sure if this should be timed and included in a build step?

Edit: I have added a PR here #38891