When building my massive site a lot of unreported time is spent doing something mysterious between "load plugins" and "onPreInit"

[FraserThompson]

When building a production build or running the dev server, it hangs after the load plugins step for around an hour, but the next step (onPreInit) reports that it took less than a second.

Once it's finished, the total bootstrap time is reported as 3614.893s, but summing the times reported by the build steps only adds up to 306.684s.

So either one of these steps is being mis-timed, or there's something happening between the steps which isn't being timed, possibly something between load plugins and onPreInit.

My site is very huge (over 48,000 nodes) so it presents weird issues like this sometimes, which is why I'm not reporting it as a bug since it's likely very hard to reproduce and probably nobody else is experiencing it.

Does anyone have any ideas or can point me to where to start investigating? I can see there's a few things happening between those steps in initialize.ts but I'm not sure which of them could be causing this.

This is the entire build output from npm start up until bootstrap is finished:

success compile gatsby files - 160.615s
success load gatsby config - 0.072s
success load plugins - 6.301s
success onPreInit - 0.053s
success delete worker cache from previous builds - 0.197s
success initialize cache - 1.424s
success copy gatsby files - 13.918s
success Compiling Gatsby Functions - 12.379s
success onPreBootstrap - 12.994s
success createSchemaCustomization - 0.074s
success Clean up stale nodes - 1.521s - 1101/0 0.00/s
success Checking for changed pages - 0.001s
success source and transform nodes - 87.762s
success building schema - 3.345s
success createPages - 4.242s
success createPagesStatefully - 0.349s
info Total nodes: 48602, SitePage nodes: 1102 (use --verbose for breakdown)
success Checking for changed pages - 0.027s
success write out redirect data - 0.004s
warn The icon(src/assets/site-icon.jpg) you provided to 'gatsby-plugin-manifest' is not square.
The icons we generate will be square and for the best results we recommend you provide a square icon.

success Build manifest and related icons - 0.701s
success onPostBootstrap - 0.705s
info bootstrap finished - 3614.893s

[FraserThompson]

Okay, so I added timer functions to the tasks between load plugins and onPreInit and ran npm start again. This was the result:

success removeStaleJobs - 3384.233s
success flattenPlugins - 0.001s
success startPluginRunner - 0.001s

So that answers the question of where that time is being spent. The next question is what is happening when it's removing stale jobs and why is it taking so long?

Sidenote: Should these tasks be timed by default? Or maybe just with a verbose option? Seems weird that some build tasks are timed and some aren't. Maybe I should start an issue requesting that feature.

edit: Okay, I think I've figured it out. Because my site has so many files (many of which are quite big) file hashing work takes a long time - a lot of problems were solved for me when we added the fastHash feature which skips the cryptographic hash and just uses a slightly less robust digest of mtime and filesize. But it looks like when checking for stale jobs (isJobStale) each file is sha1 hashed, which is gonna take a long time for my site.

I don't fully understand how this job stuff works yet, but ideally we could extend the behaviour of the fastHash parameter (or make a new one) to also do the faster hashing method here in core. I will play around with this and see if I can get something resembling a PR.