Mixing Binaries and Compiling

From Argent Wiki

Mixing Binaries and Compiling

In devmode, Argent is a binary distribution and a full Gentoo at the same time. Most of your packages come precompiled from our binhost, and the ones you choose are compiled on your machine, with your own configuration.

This page shows how avasile, epkg and emerge work together for that, how to maintain your own packages, and how to tell portage which packages it must always compile, even when our binhost has them.

It uses your own files in /etc/portage all the time, so read Argent Portage Central Configuration first if you haven't yet.

Who does what

Tool What you use it for
avasile puts your system in devmode, once
epkg your daily work: update, install, upgrade, clean, and compile when you ask for it
emerge everything else, epkg is only a wrapper around it

All three read the same /etc/portage, so what you set there applies to all of them. The only exception are the epkg compile commands, see Which epkg commands follow your settings.

Step 1: switch to devmode

avasile --devmode

Check it:

avasile --status

It has to say avasile is on --devmode!.

The other modes don't work for this:

  • user mode has no ebuilds, so nothing can be compiled
  • srcmode compiles everything already, there are no binaries to skip

A mode switch deletes the whole /etc/portage, your own files too. If you already have files there, copy them out before you switch, and put them back after. See Avasile System Management.

How devmode chooses between a binary and compiling

The devmode make.conf has: 

FEATURES="... buildpkg ..."
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --getbinpkg --rebuilt-binaries"

For every package it has to install, portage:

  • takes our binary, when we have that version and it was built with the same USE flags as yours
  • compiles the ebuild when we don't, and saves the result in /var/cache/binpkgs (FEATURES=buildpkg), so next time it can use your binary instead of compiling again
  • replaces what you have installed with our binary of the same version, when ours was built at another time than yours (--rebuilt-binaries). That happens when we rebuild a package on the binhost, and it can happen right after you compiled one yourself

Portage only compares the version and the USE flags. It doesn't know about your patches, your compiler flags, your package.env settings, or an ebuild in your overlay with the same name and version as ours. In all these cases it installs our binary over your work, and --rebuilt-binaries brings our binary back even after you compiled yours. This is what the exclusions below are for.

To see what portage is going to do, before it does it:

emerge -pv <package>

  • a line starting with [binary: it installs a binary, ours or one of yours from /var/cache/binpkgs
  • a line starting with [ebuild: it compiles

The two flows

Figure 1: installing packages 

+---------------------------------+
| user                            |
+---------------------------------+
                 |
                 v
+---------------------------------+
| epkg install                    |
+---------------------------------+
                 |
                 v
+---------------------------------+
| downloads the packages.gpkg.tar |
+---------------------------------+
                 |
                 v
+---------------------------------+
| unarchives the files            |
+---------------------------------+
                 |
                 v
+---------------------------------+
| puts them on the main system    |
+---------------------------------+

Figure 2: compiling mode 

+---------------------------------------------------------------+
| user                                                          |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| epkg install                                                  |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| portage checks .ebuilds source builds                         |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| downloads and verifies the source of a package                |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| prepares the source with patches                              |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| compiles the source which takes time                          |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| installs the compiled source directly                         |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
| creates a package.gpkg.tar for anyone else to use and install |
+---------------------------------------------------------------+

Your own packages

What "your own package" can be, and where it goes:

You want Where Does portage compile it by itself?
different USE flags /etc/portage/package.use/100-my.package.use yes, our binary doesn't match your USE flags
a testing (~amd64) version /etc/portage/package.accept_keywords/100-my.package.keywords yes, unless our binhost has that exact version
a package that is not on our binhost your own overlay yes, there is no binary for it
your own patch /etc/portage/patches/<category>/<package>/ no, exclude it
your own compiler flags or settings for one package /etc/portage/package.env/100-my.package.env and /etc/portage/env/ no, exclude it
your own ebuild, with the same name and version as ours your own overlay no, exclude it
a live ebuild (-9999) your own overlay, or keywords no, it reinstalls your previous build, see Live ebuilds

USE flags, keywords, package.env and overlays are explained in Argent Portage Central Configuration, overlays in the Your own overlays section.

Patches are the Gentoo user patches: every file you put in /etc/portage/patches/<category>/<package>/ is applied when that package is compiled. For example:

mkdir -p /etc/portage/patches/media-video/mpv

cp /root/mpv-my-fix.patch /etc/portage/patches/media-video/mpv/

The name of the patch has a dot (.patch), so epkg update leaves it alone, the same rule as for your other files.

Telling portage to compile instead of using our binary

Permanently in your make.conf

Add the packages to --usepkg-exclude, in your own make.conf, /etc/portage/make.conf/01-my.make.conf: 

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude media-video/mpv --usepkg-exclude app-editors/neovim"

You can repeat --usepkg-exclude for every package, or give it one list in quotes: 

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'media-video/mpv app-editors/neovim'"

You can also add more lines like these later, each one adds to the ones before it.

Always keep ${EMERGE_DEFAULT_OPTS} at the start. It brings in --getbinpkg --rebuilt-binaries from our make.conf. Without it, devmode stops using our binhost and compiles everything.

What you can write in the list:

  • package names: media-video/mpv
  • slots: dev-lang/python:3.13
  • wildcards: kde-apps/*

What you can't write: versions (=media-video/mpv-0.40.0, >=media-video/mpv-0.40), USE flags or repositories (::myoverlay). emerge stops with Invalid Atom(s) in --usepkg-exclude parameter. If you want to hold a version, use package.mask.

Check that portage sees it, ours and yours together:

portageq envvar EMERGE_DEFAULT_OPTS

and that the package is compiled now:

emerge -pv media-video/mpv

From now on:

  • epkg install and epkg upgrade compile mpv every time it has to be installed or upgraded, and take everything else from our binhost
  • our rebuilds of mpv never replace yours
  • its build dependencies are installed too, as binaries when we have them

Keep your own binaries, skip only ours

--usepkg-exclude ignores every binary of the package, also the ones you built yourself, so every reinstall compiles it again. If you only want to keep our binaries away, use --getbinpkg-exclude instead: 

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --getbinpkg-exclude media-video/mpv"

Portage doesn't download our mpv anymore, but when it has to reinstall the same version, it takes your own build from /var/cache/binpkgs instead of compiling it again. New versions are still compiled. It accepts the same kind of list as --usepkg-exclude.

When you change your patch or your settings, your old build is still there and still matches. Use epkg recompile <package> once (see Only once), or use --usepkg-exclude for that package.

Live ebuilds

A live ebuild (-9999) compiles the latest code from git. With binaries enabled, portage reinstalls your previous build of it from /var/cache/binpkgs, and you never get the new code. To always compile live ebuilds: 

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude-live y"

Not keeping a binary at all

FEATURES=buildpkg saves a binary of everything you compile. For a package in --usepkg-exclude that binary is never used, so you can stop saving it and keep the space: 

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude www-client/chromium --buildpkg-exclude www-client/chromium"

Only once

epkg install and epkg upgrade pass extra options to emerge, so you can exclude a package for one run only:

epkg install --usepkg-exclude media-video/mpv media-video/mpv

epkg upgrade --usepkg-exclude media-video/mpv

Or compile it with epkg:

epkg recompile media-video/mpv

epkg compile and epkg recompile don't use any binary in that run, ours or yours: the package, and every dependency it still needs, gets compiled. But it doesn't stick: the next epkg upgrade can put our binary back, with a new version or with --rebuilt-binaries (see How devmode chooses between a binary and compiling). For a package you maintain, use the make.conf way above.

Which epkg commands follow your settings

Most epkg commands run emerge with your make.conf, so your EMERGE_DEFAULT_OPTS, exclusions included, applies to them.

The compile commands don't. For that run, they replace EMERGE_DEFAULT_OPTS with only --rebuilt-binaries, so they ignore our binhost and your options, and compile everything they install.

Your EMERGE_DEFAULT_OPTS applies Replaced for that run, everything is compiled
install, autoinstall, oneshot, autooneshot, install-no-collision, autoinstall-no-collision, upgrade, autoupgrade, upgrade-full, autoupgrade-full, upgrade-specific, autoupgrade-specific, upgrade-nocol, autoupgrade-nocol compile, autocompile, recompile, autorecompile, recompile-depend, autorecompile-depend, src-newuse, recompileallpkg, autorecompileallpkg, buildpkgonly, autobuildpkgonly, module-rebuild, automodule-rebuild, compile-preserved, autocompile-preserved

epkg buildinstall and epkg autobuildinstall are a mix: they compile without our binhost, then install the binary they just built, with only --getbinpkg --rebuilt-binaries.

So the usual way of working is: your exclusions in make.conf, epkg update && epkg upgrade as always, and epkg recompile when you want one package compiled again right now.

Using emerge directly

emerge reads the same make.conf, so everything on this page applies to it too: our binhost when the binary matches, compiling when it doesn't, and your exclusions.

emerge -av <package> is the same as epkg install <package>.

To skip all of EMERGE_DEFAULT_OPTS for one run, ours and yours:

emerge --ignore-default-opts -av <package>

This also drops our --getbinpkg, so everything in that run is compiled, like epkg compile.

Example: your own patched mpv

You have a fix for mpv that is not in Gentoo yet, and you want to keep it on every upgrade, while everything else keeps coming from our binhost.

1. Devmode, once

avasile --devmode

2. Your patch

mkdir -p /etc/portage/patches/media-video/mpv

cp /root/mpv-my-fix.patch /etc/portage/patches/media-video/mpv/

3. Always compile mpv

echo 'EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude media-video/mpv"' >> /etc/portage/make.conf/01-my.make.conf

Keep the single quotes, otherwise your shell replaces ${EMERGE_DEFAULT_OPTS} before it gets into the file.

4. Check

portageq envvar EMERGE_DEFAULT_OPTS

It shows --getbinpkg --rebuilt-binaries --usepkg-exclude media-video/mpv.

emerge -pv media-video/mpv

It shows [ebuild, not [binary.

5. Install it

epkg install media-video/mpv

mpv is compiled, and you see your patch applied in the build output.

6. From now on

epkg update && epkg upgrade

as always. Everything comes from our binhost, and mpv is compiled with your patch whenever it gets a new version. If a new version breaks your patch, the compile fails and tells you, then you fix or remove the patch.

Frequent Questions

Example flow start to end or "What should I do?"

avasile --devmode
avasile --status
echo 'EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude media-video/mpv --usepkg-exclude app-editors/neovim"' >> /etc/portage/make.conf/01-my.make.conf
portageq envvar EMERGE_DEFAULT_OPTS
epkg update
epkg upgrade
etc-update
epkg clean

If you compile your own kernel, replace only the epkg upgrade line with: 

epkg upgrade --keep-going y && epkg module-rebuild

Things to keep in mind

  • every package you exclude is compiled at every upgrade of it. Exclude only what you really maintain, big libraries like dev-qt/qtbase or llvm-core/llvm cost hours, and excluding a library doesn't rebuild what depends on it
  • ⚠️ keep a copy of your 01-my.make.conf, your /etc/portage/patches and your repos.conf files: a mode switch with avasile deletes all of /etc/portage
  • epkg clean removes your older builds from /var/cache/binpkgs together with ours, and epkg wipe removes all of them. See Local binary packages when cleaning
  • don't use exclusions in user mode: there are no ebuilds there, so an excluded package can't be installed or upgraded at all
  • the moment your packages differ from ours, they are yours: if one of them breaks, start by checking your patch or your settings, before reporting it to us

Binary packages only (experimental)

The experimental ibin commands from the EPKG Package Manager page (epkg ibin, epkg iug, ...) use qmerge, not emerge. qmerge never compiles and doesn't read EMERGE_DEFAULT_OPTS, so your exclusions don't apply to it.

If you test them on a system with your own packages, hold your packages at the version you have installed:

epkg iug --exclude media-video/mpv

Retrieved from "https://wiki.argentlinux.io/index.php?title=Mixing_Binaries_and_Compiling&oldid=133"