Gentoo adventures: running a script after a package is updated / installed (emerge hook)

So here's a neat little thing that I learned tonight: how to run a script after a specific package is installed or updated.

Why?

Because I use the net-fs/cifs-utils package to provide mount.cifs and I'd like scripts run by an unprivileged user to be able to mount and dismount entries in /etc/fstab. I'd also like user-space tools like the Dolphin file manager to be able to mount these points.

At some point, there was an suid USE flag for net-fs/cifs-utils but it was decided that the USE flag be removed due to security concerns. Which is probably a good decision to have as a default, but it's a little annoying when my script to sync media to my media player fails because it can't (re-)mount the samba share after net-fs/cifs-utils was updated.

I had a hunch that there would be a way to hook into the emerge process -- and Gentoo (or rather, Portage) didn't let me down. After finding https://dev.gentoo.org/~zmedico/portage/doc/portage.html#config-bashrc-ebuild-phase-hooks (which alluded to creating bash functions in a special file) and https://wiki.gentoo.org/wiki/Handbook:X86/Portage/Advanced#Using_.2Fetc.2Fportage.2Fenv which hinted at where to do such magic, and after a little play-testing, I finally got to adding the following function to the (new) file /etc/portage/bashrc:

function post_pkg_postinst() {
  if test "$CATEGORY/$PN" = "net-fs/cifs-utils"; then
    TARGET=/sbin/mount.cifs
    echo -e "\n\n\e[01;31m >>> Post-install hook: setting $TARGET SUID <<<\e[00m\n\n"
    chmod +s $TARGET
  fi
}

Which automatically sets the SUID bit and gives me a giant red warning that it did so.

https://wiki.gentoo.org/wiki/Handbook:X86/Portage/Advanced#Using_.2Fetc.2Fportage.2Fenv  seemed to suggest that I could create the hook in /etc/portage/env/net-fs/cifs-utils/bashrc, but that didn't work out for me (EDIT: putting stuff under there is a very bad idea unless you're only setting environment variables -- I initially put my hooks under there and found that subsequent emerges installed under /etc/portage/env !!) . The above did and was (reasonably) simple to implement, after a few install attempts and dumping the environment that the script runs in with the env command.

Refactoring time: I'd like to be able to write individual scripts for packages (if I ever do this again), like I do with USE flags, where I store all flags pertinent to the installation of a higher-order package in /etc/portage/package.use/higher-order-package so I know where lower-down USE flags come from, for example my /etc/portage/package.use/chromium file contains:

www-client/chromium -cups -hangouts
>=dev-libs/libxml2-2.9.4 icu
>=media-libs/libvpx-1.5.0 svc cpu_flags_x86_sse4_1 postproc

Wherein I can always reminisce about how I had to set the icu flag on libxml2 to successfully install chromium (https://wiki.gentoo.org/wiki/Qt/FAQ#qtwebkit_vs_chromium_block_caused_by_icu)

Similarly, it would be nice if I could just create code snippets under /etc/portage/env/${CATEGORY}/${PN}, perhaps even have category-wide scripts (which an individual package could override)... hm. Ok, so first, we move the script above to /etc/portage/env/net-fs/cifs-utils/post-install. Then we create a new /etc/portage/bashrc with these contents:

PORTAGE_HOOKS_PREFIX=/etc/portage/hooks
PACKAGE_HOOKS_DIR="$PORTAGE_HOOKS_PREFIX/$CATEGORY/$PN"
CATEGORY_HOOKS_DIR="$PORTAGE_HOOKS_PREFIX/$CATEGORY"
function source_all() {
  if test -d "$1"; then
    for f in $1/*; do
      if test -f "$f"; then
        source "$f"
      fi
    done
  fi
}
source_all "$CATEGORY_HOOKS_DIR"
source_all "$PACKAGE_HOOKS_DIR"

Which, as suggested above, loads all scripts from the category folder (if it exists) then overlays with scripts from the package folder (if that exists). So now it's trivial to add hooks for any of the emerge phases outlined in https://dev.gentoo.org/~zmedico/portage/doc/portage.html#config-bashrc-ebuild-phase-hooks

Side-note: since the scripts above can change any environment variable involved in the build, it's quite important to use nice, long, specific variable names. One of my first attempts used PREFIX instead of PORTAGE_HOOKS_PREFIX and that just ended up installing all new packages under the /etc/portage/hooks directory because, of course, $PREFIX is an environment variable respected by most build scripts... *facepalm*.

Once again, this becomes a testament to how Gentoo provides an operating system which is totally yours. You have the control to deal with all of it, its state is as a result of your actions and choices. Once again, I wish I'd switched years ago.

EF-based testing, with PeanutButter: Shared databases

The PeanutButter.TestUtils.Entity Nuget package provides a few utilities for testing EntityFramework-based code, backed by TempDb instances so you can test that your EF code works as in production instead of relying on (in my experience) flaky substitutions.
One is the EntityPersistenceTester, which provides a fluent syntax around proving that your data can flow into and out of a database. I'm not about to discuss that in-depth here, but it does allow (neat, imo) code like the following to test POCO persistence:

// snarfed from EmailSpooler tests
[Test]
public void EmailAttachment_ShouldBeAbleToPersistAndRecall()
{
    EntityPersistenceTester.CreateFor<EmailAttachment>()
        .WithContext<EmailContext>()
        .WithDbMigrator(MigratorFactory)
        .WithSharedDatabase(_sharedTempDb)
        .WithAllowedDateTimePropertyDelta(_oneSecond)
        .ShouldPersistAndRecall();
}

which prove that an EmailAttachment POCO can be put into, and successfully retrieved from a database, allowing DateTime properties to drift by a second. All very interesting, and Soon To Be Documented™, but not the focus of this entry.

I'd like to introduce a new feature, but to do so, I have to introduce where it can be used. A base class TestFixtureWithTempDb<T> exists within PeanutButter.TestUtils.Entity. It provides some of the scaffolding required to do more complex testing than just "Can I put a POCO in there?". The generic argument is some implementation of DbContext and it's most useful when testing a repository as it provides a protected GetContext() method which provides a spun-up context of type T, with an underlying temporary database. By default, this is a new, clean database every test, but you can invoke the protected DisableDatabaseRegeneration() method in your test fixture's [OneTimeSetup]-decorated method (or constructor, if you prefer) to make this database live for the lifetime of your test fixture. The base class takes care of disposing of the temporary database when appropriate so you can focus on the interesting stuff: getting your tests (and then code) to work. A full (but simple) example of usage can be found here: https://github.com/fluffynuts/PeanutButter/blob/master/source/Utils/PeanutButter.FluentMigrator.Tests/TestDbMigrationsRunner.cs

My focus today is on a feature which can help to eliminate a pain-point I (and others) have experienced with EF testing backed onto a TempDb instance: time to run tests. EF takes a second or two to generate internal information about a database the first time some kind of activity (read/write) to that database is done via an EF DbContext. Not too bad on application startup, but quite annoying if it happens at every test. DisableDatabaseRegeneration() helps, but still means that each test fixture has a spin-up delay, meaning that when there are a few test fixtures, other developers on your team become less likely to run the entire test suite -- which is bad for everyone.

However, after some nudging in the right direction by co-worker Mark Whitfeld, I'd like to announce the availability of the UseSharedTempDb attribute in PeanutButter.TestUtils.Entity as of version 1.2.120, released today.
To use, decorate your test fixture:

[TestFixture]
[UseSharedTempDb("SomeSharedTempDbIdentifier")]
public class TestCrossFixtureTempDbLifetimeWithInheritence_Part1
    : TestFixtureWithTempDb
{
    // .. actual tests go here ..
}

And run your tests. And see an exception:

PeanutButter.TestUtils.Entity.SharedTempDbFeatureRequiresAssemblyAttributeException : 
The UseSharedTempDb class attribute on TestSomeStuff 
requires that assembly SomeProject.Tests have the attribute AllowSharedTempDbInstances.

Try adding the following to the top of a class file:
[assembly: PeanutButter.TestUtils.Entity.Attributes.AllowSharedTempDbInstances]

So, follow the instructions and add the assembly attribute to the top of your test fixture source file and re-run your tests. Congratulations, you're using a shared instance of a TempDb which will be cleaned up when NUnit is finished, providing, of course, that you don't interrupt the test run yourself (:

Gentoo adventures: Diablo 1 HD!

I recently wrote about switching to Gentoo. It's been an overall quite positive experience. The following is a story about getting Diablo 1 HD running through WINE, because:

• Diablo 1 is one of the most iconic games ever
• Diablo 1 (vanilla) has always had some issues running through WINE (most notably, the main menu -- though I discovered today that if you alt-tab (in Plasma shell, at least), then the window preview actually does show the menu)
• I stumbled across someone's post on Reddit stating that it could be done (though it turned out that his advice didn't work for me, but I did eventually get there)

Just in case you didn't follow the link, this is what I'm talking about:

Before we begin: WINE doesn't deal well with an app which changes resolution and then dies unexpectedly. I found this out the hard way. I'd highly recommend installing (if you don't have it already) xrandr and binding a hotkey to set your native resolution. If you're using xbindkeys (yes, I like old-school... it works, reliably and I have an existing config with tweaks to allow volume changing via the buttons on the side of my mouse), you can add this to your .xbindkeysrc:

"xrandr -s 1920x1080"
    Control + Alt + Shift + F12

to be able to swap back to 1920x1080 by pressting ctrl-alt-shift-f12, in case, somewhere along the way, you end up with a strange-looking desktop.

On with it then!

First of all, you're going to need an (ahem) original Diablo 1 CD. Just so happens that I do have one of those, living on a shelf of other games, many of which I own on Steam because optical media is so last century. I'm not advocating other methods for doing this. When I got this Diablo 1 CD, it cost around R50 (~$13). Not really gonna break the bank. If you really can't get one, you could try alternative methods, if you know what I mean.

Naturally, my instructions are going to be in Gentoo dialect, but I expect that there won't be a lot different for others, except the USE flag fiddling I did -- which you won't have to do in a binary distribution and also which may not be entirely necessary, though not hurtful. Some were attempts to get around sound issues and I'll point those out.

Ok, so you have your original Diablo 1 CD, right? First we need to install that, which means we need wine. I installed with the following package.use:

app-emulation/wine dos openal s3tc staging

Note that I added dos simply in case I wanted dosbox later, s3tc because equery u wine claims it may be needed for some games, staging because I was curious what I would be missing out on from pre-release features and openal because of that aforementioned Redditer. So I don't think you actually need any of this.

Once you have WINE installed, run the autorun.exe from your original Diablo 1 CD. Setup should be quick and launch into the game -- cinematics first and then a black screen as the menu isn't properly displayed. You can alt-tab out of this and just kill it.

Now download the latest Diablo 1 HD release and unpack it where you installed Diablo 1. If you stuck with all defaults, you'll find it under ~/.wine/drive_c/Diablo. This adds a Belzebub.exe executable, which you can attempt to WINE -- but you'll get an error about OpenAL. Belzebub is shipped with an OpenAL32 dll -- the problem is that it depends on DirectSound. The false trail to follow is to use the inbuilt openal in WINE -- for me, that stopped the error, but also left me with no in-game sound (though the cinematics did have sound).

What worked better for me was to run winetricks, which I installed with the following package.use:

app-emulation/winetricks rar
 
simply because I didn't want some arbitrary download to fail simply because it came down as a rar archive. I'm pretty sure you don't need it for this process though -- winetricks was downloading the directx9 installer and pulling out dsound.dll (source at time of writing). Once again, I don't think you need my USE flags at all -- but they don't hurt. If your distro doesn't bindle Winetricks, you can always get it here: https://github.com/Winetricks/winetricks.
Run winetricks and go through:

1. Select default wineprefix
2. Install a Windows DLL or component
3. Select dsound and hit the Ok button

Downloading may take a little while, depending on your pipe. Installation was quite quick. Running winecfg afterwards, looking at the Libraries tab, you should see *dsound set native. And now you should be able to run

wine Belzebub.exe

in the ~/.wine/drive_c/Diablo folder, with the original CD in your optical drive and the game should start up and work properly. You can bask in the glory of a 20-year-old game, running at a modern resolution on an OS it wasn't designed for (:

For bonus points, you can pack away that CD if you're prepared to keep an ISO on disk:

1. mkdir ~/DIABLO1_CD
2. dd if=/dev/sr0 of=~/diablo1.iso (substitute your optical drive for /dev/sr0 if it differs)
3. add the following line to your /etc/fstab
/home/{USER}/diablo1.iso /home/{USER}/DIABLO1_CD/ auto loop,users 0 0
4. mount ~/DIABLO1_CD

You could even put that into a script in your Diablo folder:  

#!/bin/sh 
cd $(dirname $0) 
mount ~/DIABLO1_ISO 
wine Belzebub.exe
cd - 

You will probably see some odd flickering as the app starts up. Just hang in there! Mount won't barf on multiple runs -- it will just refuse to remount. Happy hunting!

Like changing religion, only harder, part 2

Gentoo is not the distribution for you if:

• You hate reading
• You don't want to learn new stuff
• You're not willing to do a lot of it manually

The Handbook is an excellent resource, rich in detail. I'd actually recommend that anyone wanting to learn more about how Linux-based stuff works give an installation a go, even if just in a safe VirtualBox VM.

That's where I starrted -- a VM. I need my desktop to stay in a fully working state, so I can't afford to have it partially working or offline for days on end. And even a preliminary look at the handbook was enough to inform me that this was no trivial task.

Now, I've been a little coddled, possibly, by how good even the early Debian (curses-based) installers were (and still are). Debian subscribes to the "chicken-peck philosophy": basically, once the installer is going, you're going to be asked a lot of questions, but the default answer, which a chicken would get by just pecking on the enter key, is sane and will work for most people, Debian also attempts to install with a sane set of default packages which would be useful (or at least not get in the way) for most people.

Gentoo, on the other hand follows the "your machine, your rules" philosophy -- you're basically not guided with much at the terminal. You are going to need that handbook, unless you've done this a few times before.

After a few hours each night for a few nights, I felt like I had a handle on this and discovered that I had a hard drive I wasn't using. My only niggle was that I hadn't managed to get sound to work in the VM -- whenever I tried to play any sound, the player would just lock up. Still, I thought it was worth giving a go, whilst leaving my default boot sequence intact (and manually selecting the spare drive to boot from, when I needed to). I also found that I could do a lot just in a chroot whilst being booted into my existing Debian installation, so that helped a lot.

I'm still learning, but so far, I've gotten to the point where my default boot is the Gentoo installation, launching into a Plasma5 desktop. It's appreciably more responsive than I ever had Plasma5 before (which is why I had been quite happy to downgrade to KDE4 when I last installed Debian over Ubuntu) -- indeed it's more responsive than my Debian install in pretty-much every way. Some points that were interesting in the process:

• The dmix ALSA plugin no longer requires magick to make it work (as it did when I first encountered it, ages ago). DMIX was, for me, this unattainable holy grail -- I never managed to muster the .asoundrc-foo to get it working. Which is why I had resigned myself to having to be bound to PulseAudio for all these years. But the good news is that I didn't have to configure a thing -- ALSA worked with software mixing out of the box, with no coaxing. WIN!
• Installing a service doesn't mean it will be automatically started at system boot. This was a little surprising to me, but it fits in with the Gentoo philosophy -- you need to consciously decide that you want a service running at startup. It doesn't just happen. Fortunately, OpenRC is very easy to deal with, not like SystemD (at least, that's my experience)
• Debian users will know that you pretty-much just need apt-get (install stuff and update lists of stuff you can install) and apt-cache (search for stuff to install). Gentoo has emerge for the searching and installing, but you're really going to want equery (from gentoolkit) and probably genlop.
• Coming from a binary-based distro, USE flags are both interesting and scary. So now I have software compiled on my system with exactly the feature-set I asked for -- unlike binary packages, where I get whatever an upstream developer thought was a good feature-set. Not that that upstream developer was wrong or needed second-guessing: I never had a complaint with that. It's just an interesting paradigm shift and that Gentoo philosophy of consciously selecting stuff comes through again: I've often found that packages have practically the bare minimum of features aligned (Handbrake / ffpmeg is a good example) and I really wish I'd found equery uses a lot earlier. That handy little command not only tells you what use flags are available (which you can figure out with emerge -pv), but (mostly) what they actually mean.
• Compiling. Eish. Be prepared to let your pc grind for an hour or so. Sometimes you can avoid this: there are binary packages for some of the more arduous bits (like LibreOffice and PaleMoon, though I chose to install those from source for funsies). Sometimes you can't (when I found that something I'd installed required that 32-bit versions of 177 other libraries be built). It's an elegant system though.

So I guess it's "So long, Debian, and thanks for all the fish". Perhaps Devuan will get a little more impetus and woo me back again. Who knows? But for now, I'm an habitually-tweaking Gentoo user.

Like changing religion, only harder, part 1.

I've used Debian or some derivative (Ubuntu, Linux Mint and even a brief stint with Corel Linux) as my primary desktop OS at home for around 16 years. Ok, I'll admit there was a brief period when I used Mandrake, before it was Mandriva and long before OpenMandriva. I also used OpenSUSE for a good few months, until RPM reminded me just how much I hate it.

Sure, I reboot to "that other OS" to game and sometimes dev (though that's mainly on a notebook now) and to observe windows updates. Gotta have a hobby. But my home pc spends upwards of 90% of its life in Linux land -- even more recently because I've been too lazy to reboot to game :/

I liked most of Debian, only choosing a derivative when I wanted more current software. I used Ubuntu long before it was the OS of the kool kids, all the way from Warty. It was all I liked about Debian with more current software but not the mad downloads required to keep up with Debian testing. I gave up on Ubuntu partly because of Unity / GNOME3 -- I had swiched to KDE, but it was never particularly fast. But something had already begun to bug me: PulseAudio.

Sound in Linux has traditionally been hard. I started before ALSA, when OSS was the way and there were a few sound daemons you could choose from, but honestly the best approach was to just get a sound card with hardware mixing -- I found a Creative AWE32 and things were good. But this PulseAudio beast was foisted on me and I never quite figured out how to shake it. I dislike it because it introduces noticable latency (even to my non-studio ears) and was always a little flaky. Last time I reloaded (from Ubuntu back to Debian plain, about a year ago), it was after finally simply having had enough of PA dying randomly. Nothing I tried would fix it for long and my desktop was laggy too -- it was just time for a clean house.

Going back to Debian was like a breath of fresh air -- things were faster, but there's still no (easy) hiding from PulseAudio. And Debian has also chosen to go the SystemD route -- another piece of software which I gave a chance, but just grew to hate more and more. Where are my logs? I have to learn journalctl commands, because they're not just in nice little text files under /var/log any more. How do I start/stop/restart a service? I have to learn systemctl commands, because /etc/init.d/(some service) (start|stop|restart) is only available for "antiquated" packages which haven't "upgraded" to systemd yet.

And don't get me started on boot (and even worse: shutdown / reboot) times. I'm running an i7 3770k with 16 gig RAM and an NVidia 660ti. Yeah, none of that is cutting edge -- but none of it is slothful either. Windows 10 on the same machine was making my precious Linux desktop feel positively sluggish.

We can't have that!

So I did what any other pedantic geek would do: downloaded ISOs and started checking out other distros in VirtualBox VMs. Primarily, I was interested in finding something that didn't use systemd, but it also had to be actively maintained, have a decently-sized community, have a lot of packages, have a good history (or be built on the shoulders of a giant with good history, like Debian). I found a nice comparison chart here: https://en.wikipedia.org/wiki/Comparison_of_Linux_distributions#Technical and set to work on evaluating in virtual environments. Ideally, I'd like to have found something Debian-based as I know the layout and tooling fairly well, but I was open to new experiences.

Most of the options I tried out didn't wow me, but a quick glance at the list linked above shows that one of the systemd-independent distributions is Gentoo.

Rewind a decade or so, and I remember having a good friend who loved Gentoo. Personally, it always felt like the OS of choice for an habitual tweaker. It just didn't seem to be worth the effort. For a while, the argument that compiling all packages locally would give a faster desktop was the primary reason I heard that I should consider it. It was also the primary reason for angst back then -- when getting a Gentoo box up and running with a desktop environment, browser, email client, office suite, etc, would take around a week. Those were the days when an Athlon 1800 was the business -- and there's no way an Athlon 1800 can compete with even a low-power current chip for compiling. Gentoo was the distro for people with nothing better to do than watch compilation output scrolling across their screens.

But I noticed that

• OpenRC is the default init system for Gentoo (not some unsupported "you can do it, but good luck" option, like with Arch Linux, another distro I did consider because I've read a lot of good things about it),
• Gentoo had a clear howto on setting up ALSA for sound cards which don't support hardware mixing and 
• Gentoo is documented to the max -- there are wiki page upon wiki page of information

The last point is good -- but it's also a little scary for the same reason that over-documented code is scary: at some point, the documenting comments fall out of sync with the actual software logic and become misleading lies (and I've found some rabbit-holes in Gentoo wiki pages -- but mostly, they've been really helpful)

So Gentoo looked like a good choice.... Onwards and upwards!