Container debugging minihint

What’s in my container?

$ podman ps --ns
CONTAINER ID  NAMES            PID    CGROUPNS  IPC         MNT         NET         PIDNS       USERNS      UTS
fe11359293e8  eloquent_austin  11090            4026532623  4026532621  4026532421  4026532624  4026531837  4026532622

$ sudo ls -l /proc/11090/root/
total 22628
lrwxrwxrwx.   1 root root        7 Jul 25  2019 bin -> usr/bin
dr-xr-xr-x.   2 root root        6 Jul 25  2019 boot
drwxr-xr-x.   5 root root      360 Jan 24 12:03 dev
drwxr-xr-x.   1 root root      183 Jan 23 16:43 etc
 ...

Thank you.
[28 Jan@1135UTC] UPDATE—Actually, this doesn’t seem to work with newer systems, sorry!

“Reformat the filesystem to enable support”

Apparently it’s been a while since I ran containers on my office computer—and by a while, I mean, since November 2016—because if your initial install was RHEL or CentOS 7.2 or older then neither Docker nor Podman will work:

# yum -q -y install podman skopeo buildah
# podman pull registry.access.redhat.com/ubi7/ubi
Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported

So… ugh. I didn’t have any disks it’d work on either:

# for i in $(awk '{ if ($3 == "xfs") print $2 }' /etc/mtab); do xfs_info $i; done | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

I didn’t reformat anything though. podman pull wants overlayFS on /var/run/containers/storage, and buildah bud wants it on /var/lib/containers/storage. I made loopback disks for both:

Find/make space somewhere, then create a directory to put the images in:
```
# mkdir -p /store/containers
```
Create a big file, whatever size you want, for the disk image. I made mine 20GiB. It took a couple minutes, my disks are slow:
```
# dd if=/dev/zero of=/store/containers/var_lib_containers.img bs=1M count=20K
```

Find a free loop device and associate the file to it:

# losetup -f /dev/loop1
# losetup /dev/loop1 /store/containers/var_lib_containers.img

Format the “device”, then detach it from the file:

# mkfs -t xfs -n ftype=1 /dev/loop1
# losetup -d /dev/loop1

Mount the “disk”, and see if it worked:

# mount -oloop /store/containers/var_lib_containers.img /var/lib/containers
# df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1       20G   33M   20G   1% /var/lib/containers

It worked? Make it permanent:

# echo "/store/containers/var_lib_containers.img /var/lib/containers xfs defaults,loop 1 2" >> /etc/fstab

Rinse and repeat for the other drive it needed. Then try again:

# podman pull registry.access.redhat.com/ubi7/ubi
Trying to pull registry.access.redhat.com/ubi7/ubi...Getting image
source signatures
Copying blob bff3b73cbcc4 done
Copying blob 7b1c937e0f67 done
Copying config 6fecccc91c done
Writing manifest to image destination
Storing signatures
6fecccc91c83e11ae4fede6793e9410841221d4779520c2b9e9fb7f7b3830264

Resetting the root password on Fedora

Yesterday I made a Fedora 30 VM on my RHEL 7 box, and for some reason I couldn’t log in as root after the installation finished. Well, it’s been a while, so I had to look it up, and following the instructions didn’t work either—I finally managed to get a shell, but the terminal was corrupted. Because it was a VM? Because the instructions were out of date? I’ve no idea. Anyway, here’s what I did, with the stuff that wasn’t in the instructions kind of yellowish:

Reboot and wait for the GRUB menu to appear. You may need to be pressing Shift for this to happen.
In the menu, highlight any entry and press e to edit it.
Find the line beginning with linux. Remove the rhgb and quiet options, then add init=/bin/sh at the end of the line.
Press Ctrl-X to boot with those options. After a while you should get a root shell. The prompt was sh-5.0# on my system, not sh-4.2# like the instructions say, but it doesn’t matter.

Run the commands in the instructions:

/usr/sbin/load_policy -i
mount -o remount,rw /
passwd root
mount -o remount,ro /

The instructions say to reboot now, but none of the commands to reboot the system worked at this point. Probably they expected systemd. No problem, I hit “Force Reset” in Virtual Machine Manager. I probably should have run a sync or two beforehand, but I didn’t think to.

Ta-da, working system!