GitLab YAML Docker Registry client

Have you written a Docker Registry API client in GitLab CI/CD YAML? I have.

# Delete candidate image from CI repository.
clean-image:
  stage: .post
  except:
    - main

  variables:
    AUTH_API: "$CI_SERVER_URL/jwt/auth"
    SCOPE: "repository:$CI_PROJECT_PATH"
    REGISTRY_API: "https://$CI_REGISTRY/v2/$CI_PROJECT_PATH"

  before_script:
    - >
      which jq >/dev/null
      || (sudo apt-get update
      && sudo apt-get -y install jq)

  script:
    - echo "Deleting $CANDIDATE_IMAGE"
    - >
      TOKEN=$(curl -s
      -u "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD"
      "$AUTH_API?service=container_registry&scope=$SCOPE:delete,pull"
      | jq -r .token)
    - >
      DIGEST=$(curl -s -I
      -H "Authorization: Bearer $TOKEN"
      -H "Accept: application/vnd.docker.distribution.manifest.v2+json"
      "$REGISTRY_API/manifests/$CI_COMMIT_SHORT_SHA"
      | tr -d "\r"
      | grep -i "^docker-content-digest: "
      | sed "s/^[^:]*: *//")
    - >
      curl -s
      -X DELETE
      -H "Authorization: Bearer $TOKEN"
      "$REGISTRY_API/manifests/"$(echo $DIGEST | sed "s/:/%3A/g")

Use stdbuf to tee without buffering

Do you want to tee the output of a command to a file, but see it in your terminal too, without buffering? The stdbuf command can do this for you:

$ sudo stdbuf --output=L tcpdump -i any -tttt -n 'udp port 5353' | tee -a tcpdump-mdns
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
2023-06-15 11:55:16.637670 eth0 M   IP 10.0.0.23.5353 > 224.0.0.251.5353: 0 A (QM)? winnebago.local. (28)
2023-06-15 11:55:16.744660 eth0 M   IP 10.0.0.42.5353 > 224.0.0.251.5353: 0*- [0q] 1/0/0 (Cache flush) A 10.0.0.42 (38)
...

Which uninstalled package provides a file?

$ apt-file find guestmount
guestmount: /usr/bin/guestmount
guestmount: /usr/share/bash-completion/completions/guestmount
guestmount: /usr/share/doc/guestmount/changelog.Debian.gz
guestmount: /usr/share/doc/guestmount/copyright
guestmount: /usr/share/man/ja/man1/guestmount.1.gz
guestmount: /usr/share/man/man1/guestmount.1.gz
guestmount: /usr/share/man/uk/man1/guestmount.1.gz

Flask on Elastic Beanstalk

I had a play with Elastic Beanstalk the other day. It’s one of those things people turn their noses up at, but it seems pretty good for prototyping and small things. My biggest issue so far has been that, for Python applications, it expects a WSGI callable called application in the file application.py… but I was using Flask, and every single Flask application ever has a WSGI callable called app in the file app.py. I tried to not care, but it got too much after about an hour so I went and found how to override it:

$ cat .ebextensions/01_wsgi.config
option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: "app:app"

Thank you Nik Tomazic for that! (=⌒‿‿⌒=)

The other AWS IAM SSO problem

The other thing you’ll run into using IAM users in AWS CLI is that a lot of things don’t support SSO sessions anyway. If you configure an IAM user with an SSO session name as recommended you’ll get errors like this:

$ eb init -p python-3.8 eb-flask-app
ERROR: InvalidConfigError - The profile "default" is configured to use SSO but is missing required configuration: sso_start_url, sso_region

and this:

$ terraform apply
| Error: configuring Terraform AWS Provider: loading configuration: profile "default" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

You can fix these by configuring without an SSO session:

$ aws configure sso
SSO session name (Recommended): 
WARNING: Configuring using legacy format (e.g. without an SSO session).
Consider re-running "configure sso" command and providing a session name.
SSO start URL [None]: https://whatever.awsapps.com/start
SSO region [None]: us-east-1
 ...

You can also fix them by just editing your ~/.aws/config, and copying the sso_start_url and sso_region keys from the [sso-session ...] section into the relevant user’s section, but that might be a hack too far!

Using AWS CLI with IAM users

When you configure AWS CLI to use an IAM user, the first thing it asks for is an SSO session name. Don’t put whitespace or punctuation in it. The command doesn’t tell you this, but it’s going to use what you enter as an identifier, and fail with a cryptic error:

$ aws configure sso
SSO session name (Recommended): AWS CLI on Gary's Chromebook
SSO start URL [None]: https://whatever.awsapps.com/start
SSO region [None]: us-east-1
SSO registration scopes [sso:account:access]:

An error occurred (InvalidClientMetadataException) when calling the RegisterClient operation:

You need to put something like “gbenson” in there.