Container debugging minihint

What’s in my container?

  1. $ podman ps --ns
    CONTAINER ID  NAMES            PID    CGROUPNS  IPC         MNT         NET         PIDNS       USERNS      UTS
    fe11359293e8  eloquent_austin  11090            4026532623  4026532621  4026532421  4026532624  4026531837  4026532622
  2. $ sudo ls -l /proc/11090/root/
    total 22628
    lrwxrwxrwx.   1 root root        7 Jul 25  2019 bin -> usr/bin
    dr-xr-xr-x.   2 root root        6 Jul 25  2019 boot
    drwxr-xr-x.   5 root root      360 Jan 24 12:03 dev
    drwxr-xr-x.   1 root root      183 Jan 23 16:43 etc
     ...

Thank you.
[28 Jan@1135UTC] UPDATE—Actually, this doesn’t seem to work with newer systems, sorry!

“Reformat the filesystem to enable support”

Apparently it’s been a while since I ran containers on my office computer—and by a while, I mean, since November 2016—because if your initial install was RHEL or CentOS 7.2 or older then neither Docker nor Podman will work:

# yum -q -y install podman skopeo buildah
# podman pull registry.access.redhat.com/ubi7/ubi
Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported

So… ugh. I didn’t have any disks it’d work on either:

# for i in $(awk '{ if ($3 == "xfs") print $2 }' /etc/mtab); do xfs_info $i; done | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

I didn’t reformat anything though. podman pull wants overlayFS on /var/run/containers/storage, and buildah bud wants it on /var/lib/containers/storage. I made loopback disks for both:

  1. Find/make space somewhere, then create a directory to put the images in:
    # mkdir -p /store/containers
  2. Create a big file, whatever size you want, for the disk image. I made mine 20GiB. It took a couple minutes, my disks are slow:
    # dd if=/dev/zero of=/store/containers/var_lib_containers.img bs=1M count=20K
  3. Find a free loop device and associate the file to it:
    # losetup -f /dev/loop1
    # losetup /dev/loop1 /store/containers/var_lib_containers.img 
    
  4. Format the “device”, then detach it from the file:
    # mkfs -t xfs -n ftype=1 /dev/loop1
    # losetup -d /dev/loop1
  5. Mount the “disk”, and see if it worked:
    # mount -oloop /store/containers/var_lib_containers.img /var/lib/containers
    # df -h /var/lib/containers
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop1       20G   33M   20G   1% /var/lib/containers
  6. It worked? Make it permanent:
    # echo "/store/containers/var_lib_containers.img /var/lib/containers xfs defaults,loop 1 2" >> /etc/fstab

Rinse and repeat for the other drive it needed. Then try again:

# podman pull registry.access.redhat.com/ubi7/ubi
Trying to pull registry.access.redhat.com/ubi7/ubi...Getting image
source signatures
Copying blob bff3b73cbcc4 done
Copying blob 7b1c937e0f67 done
Copying config 6fecccc91c done
Writing manifest to image destination
Storing signatures
6fecccc91c83e11ae4fede6793e9410841221d4779520c2b9e9fb7f7b3830264