Allow the use of S3 bucket to host container build assets

[pditommaso]

Context

Currently, the build process relies on a shared file system (AWS EFS). In a nutshell, the process is the following:

1. A unique work directory is created for a container build
2. Container build assets are copied in the build work directory (e.g. Dockerfile, context files, etc)
3. The work directory is mounted in the build container.
4. The files read by Builkit.

Deliverable

The goal of this issue is to replace the use of the shared file system with an object storage e.g. S3 in order to:

1. simplify the system dependencies
2. simplify the installation process
3. simplify the maintenance

Solution

This could be achieved:

1. storing the build assets into a S3 bucket instead of EFS storage (preserving the same unique path creation logic)
2. making accessible the S3 path in the container a local path via Fusion
3. use the container /tmp as work directory required by Buildkit

[pditommaso]

Adding @jordeu for visibility

[pditommaso]

@munishchouhan We should make a POC simulating a build process pulling the data from S3 via Fusion using a local container

[munishchouhan]

I started working on this today with @pabloaledo, we found the couple of things:

1. We need a custom buildkit image with fusion enabled
2. Buildkit rootless is not working because fusion needs root access

build is still not working, but we can see in the builder container that s3 content has been mounted inside container

[munishchouhan]

I will keep on updating here for discussion

[pditommaso]

1. @pabloaledo is creating (or has created) a scratch container for Fusion. Having that you can make a create that image with wave -i moby/buildkit:v0.14.1-rootless --include <fusion scratch image> 😎
2. Fusion does not require privileged execution. @jordeu may provide more details

[munishchouhan]

we created using
wave --config-file <fusion config file URL> -i moby/buildkit:v0.15.0

[pditommaso]

Same 👍

[munishchouhan]

Another point:

1. buildkit uses its own entrypoint --entrypoint buildctl-daemonless.sh, which is overriding fusion entrypoint

[munishchouhan]

I am able to fix the entrypoint issue by creating a custom image with one entrypoint='' between buildkitd and fusion

[pditommaso]

That happens because you are using --config-file approach

[munishchouhan]

build is working but push is failing because of the lack of config.json file
Working on how to add config.json in container

[pditommaso]

It is supposed to be in the bucket along with the Dockerfile

[munishchouhan]

It is supposed to be in the bucket along with the Dockerfile

yes, but it need to be mounted to /root/.docker folder

[pditommaso]

Indeed, that's not simple to solve. Tagging @fntlnz, he may have some suggestion

[pditommaso]

About the problem of mounting /root/.docker likely it's not possible because Fusion used its own opinionated path prefix. Maybe should consider instead using Mountpoint. @jordeu What do you think?

[fntlnz]

Unfortunately even if fusion can change the mount dir with -mount-point flag it it has the second level directory which is the name of the remote storage (e.g: s3).

However it's easy to use a different directory for docker config so this works with fusion

sudo DOCKER_CONFIG=/fusion/s3/fusion-develop/scratch   docker build -t myimage .

here is how it looks like on s3.

So I would say, just mount fusion as it is and tell the docker cli to point to it.

[pditommaso]

DOCKER_CONFIG sounds a good trick

[munishchouhan]

thanks @fntlnz DOCKER_CONFIG did the trick
Working on code changes now

[munishchouhan]

This change also requires changes in Scan process and for singularity builds too.
I have created draft PR with changes in build process and tested it too, it works with dockerfile

[munishchouhan]

Singularity build and push are working using docker
Now i will work on scan

[munishchouhan]

Scan, SIngularity and build process with docker works with s3 now
I will work to move k8s to s3 now