Arch Linux User Repository

makepkg-optimize.conf isn't properly created

CFLAGS="-march=native -O3 -pipe -fno-plt -fexceptions \
#CFLAGS="-march=native -O3 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fno-plt -fopenmp -pthread -Wno-error -w" #
#CFLAGS="-march=native -Og -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fno-plt" #
        -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security \
        -fstack-clash-protection -fcf-protection"

Mold --separate-debug-file and -fuse-ld=mold needs to be in one section with -Wl

#!/usr/bin/bash
#
#   mold.sh - Compile with the fast mold linker
#

[[ -n "$LIBMAKEPKG_BUILDENV_MOLD_SH" ]] && return
LIBMAKEPKG_BUILDENV_MOLD_SH=1

LIBRARY=${LIBRARY:-'/usr/share/makepkg'}

source "$LIBRARY/util/option.sh"

build_options+=('mold')
buildenv_functions+=('buildenv_mold')

buildenv_mold() {
    if
    check_buildoption "mold" "y"; then
            LDFLAGS="${LDFLAGS/-Wl,/-fuse-ld=mold -Wl,}"
        LDFLAGS+=",--separate-debug-file"
    fi
}

clang pgo: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization isn't easy like pgo in gcc because more opportunity

@brikler

Actually I have never changed anything on the PGO thing from the upstream makepkg-optimize and I did tested it just with gcc - im sorry. Clang uses/needs other options for that. If I have time I will look into this!

it doesn't work as expected. the first compilation breaks with:

clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.used' [-Wunused-command-line-argument]
clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.used' [-Wunused-command-line-argument]
error: error: Could not read profile default.profdata: No such file or directoryCould not read profile default.profdata: No such file or directory

1 error generated.
make[1]: *** [Makefile:92: help.o] Error 1
make[1]: *** Waiting for unfinished jobs....
1 error generated.
make[1]: *** [Makefile:92: main.o] Error 1
make: *** [Makefile:31: all] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

PROFDEST was set in the PKGBUILD, and upx.genwas on the correct place.

there is something wrong in general, because its not on me to create gen and used folder.

edit1: it works with gcc, but not with 'clang'

edit2: a propos UNITY_BUILD it is more or less a developer tool, and it will not work on every project, but if, the time to compile a project can be reduced up to 50%. more about unity build: https://mesonbuild.com/Unity-builds.html

Cause of incremental builds I have something here: https://github.com/ptr1337/dotfiles/blob/master/scripts/toolchain/build_llvm.bash#L242

But it is not very well tested. It does work so for but has some issues and it also uses "check" at the compilation.

@brikler

You need to add in your makepkg.conf for example: PROFDEST=/home/ptr1337/Documents/pgo

In addition your need to create the folder for it. After you did compile the binary it should create a directory with packagename.gen and then you need to use your compiled binary, so that it does create profiles. So running for example at you upx based commands, then it will create profiles. As soon profiles are there it will be merged at the next compilation and change the folder to .used.

Here is it also explained: https://wiki.archlinux.org/index.php/Makepkg-optimize

Because CMAKE_UNITY Ive never watched into it and personally never saw any usage of it but i will take a watch.

Thank you

i noticed pgo doesn't work, there where no <paketname>.gen nor <paketname>.used folder created.

how to debug?

edit upx is my test binary :) this error messages appear when compile upx

clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
clang-14clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
clang-14clang-14: : : warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]

@ptr1337

thank you for this explanation :)

a'porpos clang, have you tried cmakes unity build function to build clang? does it work? it will safe a lot time to build clang, if this function work. https://cmake.org/cmake/help/latest/variable/CMAKE_UNITY_BUILD.html

@brikler

Actually I kept the llvm-bolt-script seperate from the makepkg-optimize-mold. Maybe I go to have it equal.

Actually the buildenv "bolt" is thought to the USE of the optimized toolchain and not for bolting binarys. The stage 1 compiler does provide llvm-bolt itself so that you can optimize binarys/.so files you have instrumented/profiled.

I think the readme is all in all not clear about the usage of the llvm-bolt-scripts. Probably I have to rework them to make it more clear.

The full-workflow.sh provides a complete compilation for a bolted llvm toolchain - that means you will have a optimized clang compiler which has been optimized through following processes: PGO (profiled), THINLTO, and bolted.

Actually it depends also which CPU you have, one which supports LBR(Intel mostly) or you have a CPU which does not support that then you need to intstrument and profile the binary.

Maybe the example I did explain in the readme helps a bit to understand that ? I will do one more example with another binary - for example zstd.