stage 4 & stage 5 #37

nickdesaulniers · 2022-06-17T23:31:55Z

No description provided.

This is a build of LLVM instrumented with PGO. The two relevant cmake defines for this are: - LLVM_BUILD_INSTRUMENTED - LLVM_VP_COUNTERS_PER_SITE I was hoping to be able to use AutoFDO instead, but I couldn't get the AutoFDO perf data post-processing tool (create_llvm_prof) to work. Some interesting env var's to use to test the resulting binary are - LLVM_PROFILE_FILE - LLVM_PROFILE_VERBOSE Otherwise this instrumented version of clang will try to write profile files to a path that exists in the container, but not outside of it: LLVM Profile Error: Failed to write file "/llvm-project/llvm/build/profiles/*.profraw": No such file or directory

A few issues encountered, will file TODO issues. - Enabling LTO causes the build of llvm-tblgen to segfault. Not sure if this is a ulimit issue related to pids? - The resulting clang is only a few seconds faster, with memory fns and str fns from musl dominating the top twenty functions of a profile.

nickdesaulniers · 2022-06-17T23:35:51Z

llvm-project/Dockerfile.epoch4

+#RUN ninja -C ${LLVM_BUILD_DIR} install-clang install-lld
+#RUN ninja -C ${LLVM_BUILD_DIR} install-clang-resource-headers
+#RUN ninja -C ${LLVM_BUILD_DIR} \
+  #install-llvm-ar \
+  #install-llvm-nm \
+  #install-llvm-objcopy \
+  #install-llvm-objdump \
+  #install-llvm-ranlib \
+  #install-llvm-readelf \
+  #install-llvm-strip
+
+#RUN apk del cmake ninja python3
+
+# Final test
+#RUN llvm-readelf -p .comment $(which clang) | grep -e clang -e LLD
+#RUN llvm-readelf -p .comment $(which clang) | grep -v GCC


probably want to keep these, probably also want to think about exporting the whole suit of tools...

nathanchance · 2022-06-17T23:42:57Z

I know this is WIP but does this need to be a whole new epoch? We should be able to just add stage 4 and stage 5 into the epoch 3 Dockerfile at this point, right?

I'll review this more on Tuesday.

compnerd · 2022-06-18T01:54:19Z

llvm-project/Dockerfile.epoch4

+RUN clang ${SYSROOT} hello.c && ./a.out && \
+  clang ${SYSROOT} hello.c -static && ./a.out && \
+  clang++ ${SYSROOT} hello.cpp && ./a.out && \
+  clang++ ${SYSROOT} hello.cpp -static -lc++abi && ./a.out


Why the SYSROOT ARG?

I did not "install" the sysroot we built to the standard /usr/local/. I think I need to play around with --prefix when configuring.

I somewhat think it's interesting to keep the sysroot separate, but the kernel has some hermiticity issues with host utilities that make building it more difficult.

Specifically, I'm using ARG to shorten repeated use of long command line options; not for someone to override at docker build time.

Should I install the sysroot to /usr/local/ and drop the explicit --sysroot= flags?

compnerd · 2022-06-18T01:56:17Z

llvm-project/Dockerfile.epoch4

+### Zlib
+COPY --from=source zlib-1.2.12.tar.gz .
+RUN tar xf zlib-1.2.12.tar.gz
+ARG ZLIB_DIR=zlib-1.2.12/build


I don't understand the value of this. Do you intend for it to be controllable by the invoker?

No, just making it shorter to refer to multiple times.

compnerd · 2022-06-18T01:58:32Z

llvm-project/Dockerfile.epoch4

+RUN apk del make
+
+### LLVM
+COPY --from=source llvm-project-14.0.1.src.tar.xz .


Whats the value of the COPY instead of just downloading it? If you are trying to save space by avoiding the layer, it seems like building and copying artifacts is what you want.

I'm not trying to optimize space of the docker image at all; rather I'm trying to optimize development time. It's faster to copy the tarball between local images than to keep redownloading it.

nickdesaulniers · 2022-06-21T16:47:44Z

I know this is WIP but does this need to be a whole new epoch? We should be able to just add stage 4 and stage 5 into the epoch 3 Dockerfile at this point, right?

Hmm...I guess we could. Let me think about that question more and play with making such a change. I don't immediately see any reason why not to do so.

nathanchance · 2022-06-21T21:18:20Z

ci/test-clang-docker.sh

+LLVM_PROFILE_FILE=foo.profraw
+LLVM_PROFILE_VERBOSE=1


As shellcheck points out, these should be exported if we are going to keep them around (more on that below).

nathanchance · 2022-06-21T21:21:18Z

llvm-project/Dockerfile.epoch4

+RUN find ${LLVM_BUILD_DIR} -name \*.profraw | xargs rm
+
+### Start doing kernel builds
+# TODO: objtool needs libelf.h and gelf.h? elfutils-dev is the alpine package.


Yes: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/objtool/Makefile#n22

I think this comment can be deleted.

nathanchance · 2022-06-21T21:27:12Z

llvm-project/Dockerfile.epoch4

+### Linux
+COPY --from=source linux-5.18-rc6.tar.gz .
+RUN tar xf linux-5.18-rc6.tar.gz
+RUN apk add make musl-dev rsync
+RUN make -C linux-5.18-rc6 INSTALL_HDR_PATH=/sysroot/usr LLVM=1 -j$(nproc) headers_install
+RUN apk del rsync musl-dev make
+
+### Musl
+COPY --from=source musl-1.2.3.tar.gz .
+RUN tar xf musl-1.2.3.tar.gz
+ARG MUSL_DIR=musl-1.2.3/build
+RUN mkdir -p ${MUSL_DIR}
+RUN cd ${MUSL_DIR} && \
+  CC=clang AR=llvm-ar RANLIB=llvm-ranlib \
+  ../configure --prefix=/usr --syslibdir=/usr/lib
+RUN apk add make
+RUN make -C ${MUSL_DIR} -j$(nproc)
+RUN make -C ${MUSL_DIR} -j$(nproc) DESTDIR=/sysroot install-headers
+RUN make -C ${MUSL_DIR} -j$(nproc) DESTDIR=/sysroot install-libs
+RUN apk del make
+
+# Pause for a quick sanity check
+COPY hello.c hello.cpp /
+ARG SYSROOT=--sysroot=/sysroot
+RUN clang ${SYSROOT} hello.c && ./a.out && \
+  clang ${SYSROOT} hello.c -static && ./a.out && \
+  clang++ ${SYSROOT} hello.cpp && ./a.out && \
+  clang++ ${SYSROOT} hello.cpp -static -lc++abi && ./a.out
+
+### Zlib
+COPY --from=source zlib-1.2.12.tar.gz .
+RUN tar xf zlib-1.2.12.tar.gz
+ARG ZLIB_DIR=zlib-1.2.12/build
+RUN mkdir -p ${ZLIB_DIR}
+RUN cd ${ZLIB_DIR} && \
+  CC="clang ${SYSROOT}" AR=llvm-ar ../configure --prefix=/sysroot/usr
+RUN apk add make
+RUN make -C ${ZLIB_DIR} -j$(nproc)
+RUN make -C ${ZLIB_DIR} -j$(nproc) install
+RUN apk del make


Why do we build the sysroot twice? Wouldn't it be better to build it once in its own image then copy it as we need it?

nathanchance · 2022-06-21T22:11:09Z

llvm-project/Dockerfile.epoch4

+    llvm-profdata merge --output=${arch}.profdata.prof \
+      $(find ${LLVM_BUILD_DIR} -name \*.profraw) && \
+    find ${LLVM_BUILD_DIR} -name \*.profraw | xargs rm


What's the reasoning behind merging the profiles after each kernel build then merging them all at the end? Wouldn't be be simpler to merge them all at the end?

Additionally, if these remain around, I think you can avoid the find call, as the .profraw files should be in ${LLVM_BUILD_DIR}/profiles.

rm ${LLVM_BUILD_DIR}/profiles/*.profraw

nathanchance · 2022-06-21T22:18:08Z

llvm-project/Dockerfile.epoch4

+  ln -s /usr/lib/libcrypto.so /sysroot/usr/lib/.
+# bpf/resolve_btfids doesn't respect HOSTCFLAGS
+RUN cp -r /sysroot/usr/include/ /usr/local/
+# TODO: something in the s390 build isn't respecting HOSTCC.


You don't set HOSTCC in the make command below, is that the problem?

nathanchance · 2022-06-21T22:26:24Z

llvm-project/Dockerfile.epoch4

+      $(find ${LLVM_BUILD_DIR} -name \*.profraw) && \
+    find ${LLVM_BUILD_DIR} -name \*.profraw | xargs rm
+
+# Mips needs bash if building all targets.


Documentation/process/changes.rst was recently updated to include bash as one of the required tools for building the kernel so this comment seems superfluous.

nathanchance · 2022-06-21T22:28:50Z

llvm-project/Dockerfile.epoch4

+      $(find ${LLVM_BUILD_DIR} -name \*.profraw) && \
+    find ${LLVM_BUILD_DIR} -name \*.profraw | xargs rm
+
+# riscv requires perl to build


There are several perl scripts in the kernel and it is listed as required in Documentation/process/changes.rst so I think this can be dropped like the bash comment above.

nathanchance · 2022-06-21T22:29:56Z

llvm-project/Dockerfile.epoch4

+#ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- make CC=clang -j72 defconfig
+RUN ARCH=${arch} HOSTCFLAGS=${SYSROOT} HOSTLDFLAGS=${SYSROOT} \
+      CROSS_COMPILE=s390x-alpine-linux-musl- \
+      CC=clang LLVM_IAS=0 \


As we mentioned offline, if we upgrade to a 5.19 prerelease, we can drop the LLVM_IAS=0 here.

nathanchance · 2022-06-21T22:30:49Z

llvm-project/Dockerfile.epoch4

+RUN apk add make flex bison
+ARG arch=arm
+RUN ARCH=${arch} HOSTCFLAGS=${SYSROOT} HOSTLDFLAGS=${SYSROOT} \
+      make -C linux-5.18-rc6 LLVM=1 -s -j$(nproc) allnoconfig all && \


We should benchmark the performance gains between allnoconfig and defconfig. I worry that allnoconfig might not provide as much coverage.

The following symbols from musl's malloc implementation were appearing in a perf profile of a kernel build (from stage 5, which isn't landed yet, see #37). Samples: 13M of event 'cycles:ppu', Event count (approx.): 9699067643159 Overhead Shared Object Symbol + 1.72% clang [.] __libc_malloc_impl 0.80% clang [.] __libc_free + 0.67% clang [.] alloc_slot + 0.65% clang [.] get_meta Replace musl's malloc with jemalloc. Fixes #36

nickdesaulniers added 4 commits June 13, 2022 13:15

clean up llvm-project container after make clang

0027cb7

build all targets, otherwise we can't profile those builds

0f42830

nickdesaulniers requested review from nathanchance and compnerd as code owners June 17, 2022 23:31

nickdesaulniers changed the title ~~Stage4~~ stage 4 & stage 5 Jun 17, 2022

nickdesaulniers commented Jun 17, 2022

View reviewed changes

nickdesaulniers marked this pull request as draft June 17, 2022 23:36

compnerd reviewed Jun 18, 2022

View reviewed changes

nathanchance reviewed Jun 21, 2022

View reviewed changes

nickdesaulniers mentioned this pull request Jul 1, 2022

llvm-project: link against jemalloc #38

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stage 4 & stage 5 #37

stage 4 & stage 5 #37

nickdesaulniers commented Jun 17, 2022

nickdesaulniers Jun 17, 2022

nathanchance commented Jun 17, 2022

compnerd Jun 18, 2022

nickdesaulniers Jun 21, 2022

compnerd Jun 18, 2022

nickdesaulniers Jun 21, 2022

compnerd Jun 18, 2022

nickdesaulniers Jun 21, 2022

nickdesaulniers commented Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

nathanchance Jun 21, 2022

		LLVM_PROFILE_FILE=foo.profraw
		LLVM_PROFILE_VERBOSE=1

stage 4 & stage 5 #37

Are you sure you want to change the base?

stage 4 & stage 5 #37

Conversation

nickdesaulniers commented Jun 17, 2022

Choose a reason for hiding this comment

nathanchance commented Jun 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nickdesaulniers commented Jun 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment