Container debugging minihint

What’s in my container?

  1. $ podman ps --ns
    CONTAINER ID  NAMES            PID    CGROUPNS  IPC         MNT         NET         PIDNS       USERNS      UTS
    fe11359293e8  eloquent_austin  11090            4026532623  4026532621  4026532421  4026532624  4026531837  4026532622
  2. $ sudo ls -l /proc/11090/root/
    total 22628
    lrwxrwxrwx.   1 root root        7 Jul 25  2019 bin -> usr/bin
    dr-xr-xr-x.   2 root root        6 Jul 25  2019 boot
    drwxr-xr-x.   5 root root      360 Jan 24 12:03 dev
    drwxr-xr-x.   1 root root      183 Jan 23 16:43 etc
     ...

Thank you.
[28 Jan@1135UTC] UPDATE—Actually, this doesn’t seem to work with newer systems, sorry!

GNOME 3 won’t unlock

Every couple days something on my RHEL 7 box goes into a swapstorm and uses up all the memory. I think it’s Firefox, but I never figured out why, generally I have four different Firefoxes running with four different profiles, so it’s hard to tell which one’s failing (if it even is that). Anyway, sometimes it makes the screen lock crash or something, and I can’t get in, and I can never remember what process you have to kill to get back in, so here it is: gnome-shell. You have to killall -9 gnome-shell, and it lets you back in. Also killall -STOP firefox and killall -STOP "Web Content" are handy if the swapstorm is still under way.

Building GDB on a freshly installed machine FAQ

So you just installed Fedora, RHEL or CentOS and now you want to build GDB from source.

  1. How do you make sure everything you need to build it is installed?
    # dnf builddep gdb
  2. Did it say, No such command: builddep? Do this, then try again:
    # dnf install dnf-plugins-core
  3. Did it say, dnf: command not found…? You’re using yum, try this:
    # yum-builddep gdb
  4. Did it say, yum-builddep: command not found…? Do this, then try again:
    # yum install yum-utils

Thank you, you’re welcome.

“Reformat the filesystem to enable support”

Apparently it’s been a while since I ran containers on my office computer—and by a while, I mean, since November 2016—because if your initial install was RHEL or CentOS 7.2 or older then neither Docker nor Podman will work:

# yum -q -y install podman skopeo buildah
# podman pull registry.access.redhat.com/ubi7/ubi
Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported

So… ugh. I didn’t have any disks it’d work on either:

# for i in $(awk '{ if ($3 == "xfs") print $2 }' /etc/mtab); do xfs_info $i; done | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

I didn’t reformat anything though. podman pull wants overlayFS on /var/run/containers/storage, and buildah bud wants it on /var/lib/containers/storage. I made loopback disks for both:

  1. Find/make space somewhere, then create a directory to put the images in:
    # mkdir -p /store/containers
  2. Create a big file, whatever size you want, for the disk image. I made mine 20GiB. It took a couple minutes, my disks are slow:
    # dd if=/dev/zero of=/store/containers/var_lib_containers.img bs=1M count=20K
  3. Find a free loop device and associate the file to it:
    # losetup -f /dev/loop1
    # losetup /dev/loop1 /store/containers/var_lib_containers.img 
    
  4. Format the “device”, then detach it from the file:
    # mkfs -t xfs -n ftype=1 /dev/loop1
    # losetup -d /dev/loop1
  5. Mount the “disk”, and see if it worked:
    # mount -oloop /store/containers/var_lib_containers.img /var/lib/containers
    # df -h /var/lib/containers
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop1       20G   33M   20G   1% /var/lib/containers
  6. It worked? Make it permanent:
    # echo "/store/containers/var_lib_containers.img /var/lib/containers xfs defaults,loop 1 2" >> /etc/fstab

Rinse and repeat for the other drive it needed. Then try again:

# podman pull registry.access.redhat.com/ubi7/ubi
Trying to pull registry.access.redhat.com/ubi7/ubi...Getting image
source signatures
Copying blob bff3b73cbcc4 done
Copying blob 7b1c937e0f67 done
Copying config 6fecccc91c done
Writing manifest to image destination
Storing signatures
6fecccc91c83e11ae4fede6793e9410841221d4779520c2b9e9fb7f7b3830264

Resetting the root password on Fedora

Yesterday I made a Fedora 30 VM on my RHEL 7 box, and for some reason I couldn’t log in as root after the installation finished. Well, it’s been a while, so I had to look it up, and following the instructions didn’t work either—I finally managed to get a shell, but the terminal was corrupted. Because it was a VM? Because the instructions were out of date? I’ve no idea. Anyway, here’s what I did, with the stuff that wasn’t in the instructions kind of yellowish:

  1. Reboot and wait for the GRUB menu to appear. You may need to be pressing Shift for this to happen.
  2. In the menu, highlight any entry and press e to edit it.
  3. Find the line beginning with linux. Remove the rhgb and quiet options, then add init=/bin/sh at the end of the line.
  4. Press Ctrl-X to boot with those options. After a while you should get a root shell. The prompt was sh-5.0# on my system, not sh-4.2# like the instructions say, but it doesn’t matter.
  5. Run the commands in the instructions:
    /usr/sbin/load_policy -i
    mount -o remount,rw /
    passwd root
    mount -o remount,ro /
  6. The instructions say to reboot now, but none of the commands to reboot the system worked at this point. Probably they expected systemd. No problem, I hit “Force Reset” in Virtual Machine Manager. I probably should have run a sync or two beforehand, but I didn’t think to.

Ta-da, working system!

Full-system Infinity preview coming up

I’ve released bits and pieces of Infinity over the past year, but nothing that really brings everything together. Right now I’m working on an initial full-system release of everything to do with Infinity so far. It’s codenamed “First Flight”, and you’ll be able to actually use it, see how everything hangs together, and generally get a better idea of what the point of it all is.

First Flight will be just enough for GDB to attach to a live process or start with a core file. “info threads” should work, “print errno” will not. First Flight will comprise:

  • A new release of the note compiler I8C.
  • A glibc you can git clone and build to get a libpthread.so with Infinity notes for map_lwp2thr, thr_get_info and thr_iter.
  • A new release of the client library libi8x that can execute those notes.
  • A libthread_db.so shim into libi8x.

The libthread_db.so shim won’t be used in future releases—they’ll have support for Infinity built into GDB—but it’ll likely remain as a nice example of how to use libi8x.

I’m targeting June 23 as the release date for First Flight. You can follow how I’m doing on the Infinity Roadmap (thank you Sergio!)