Seeing "Installing adwaita-cursor-theme-3.36.1-1.fc32.noarch" scroll past as create_container.sh was running, I looked into where that dependency came from. It's java-11-openjdk:

https://github.com/monogon-dev/monogon/blob/6ef7f9bb94890748cc7c635f187fce7c5f497fe3/build/ci/Dockerfile#L34

I then removed this package from build/ci/Dockerfile and re-ran create_container.sh, scripts/bin/bazel clean && scripts/bin/bazel test //... and tried the IntelliJ plugin, and everything appeared to continue to work as expected.

My recommendation would be:

• If the included JDK is indeed unnecessary, it can be removed -- no need to include it and its dependencies in the builder image!
• If the JDK is necessary, it seems likely that java-11-openjdk-headless would suffice. This would avoid a few themes, fonts and Xorg dependencies from being pulled in, which are almost certainly unused.

(originally created by @q3k in T933)

Implementation of the cluster lifecycle part of our lifecycle design doc:

• node/cluster state machine
• multi-node clusters (replace golden ticket mechanism)

We will need network/storage firmware for some platforms sooner or later.

We should also almost definitely ship Intel Microcode (and other CPU vendor?) updates.

Both of these fall under a generic umbrella of 'shipping blobs', be it from linux-firmware or elsewhere. One important thing to investigate is whatever licensing issues might arise from this, perhaps look into how some Linux Distribution Vendors do it.

Currently, the Curator cannot function in follower mode, and the roleserver only dials the first node of the cluster (ie. the curator that should be the leader, until failover).

We should implement this and exercise this in some end-to-end test.

Observed on https://jenkins.monogon.dev/job/gerrit-presubmit-monogon/job/44%252F444%252F1/1/console

--- FAIL: TestPanic (0.05s)
    supervisor_test.go:372: runnable 'one' didn't acknowledge cancel
FAIL

We currently don't have a way to get Kubernetes credentials without using the Debug Service.

We should add an RPC endpoint that peforms yet another credential escrow: from cluster manager/client credentials to Kubernetes credentials.

Or, we should get kube-apiserver to accept Metropolis credentials? I think we don't want to do that thought, because the apiserver has no CRL functionality... To investigate.

We currently have no way to log anything in the Curator's gRPC handlers. This isn't great.

We should use the opportunity to introduce this properly, eg. integrate with some tracing system, or at least make it possible to easily integrate with such a tracing system later.

Some design decisions to be made:

• Do we make supervisor.Logger(ctx) just work? How supervisor-agnostic do we want to make this? We likely want to use logtree interfaces, but maybe not refer to the supervisor package unless necessary? Is the clean separation worth having to do supervisor.Logger in runnables vs. somethingelse.Logger in RPCs?
• Where do we actually log things from requests? Somewhere within the logtree of the supervisor runnable, or to a totally separate set of buffers?

What we should bump:

• [X] Go (to 1.17),
• [X] gVisor (to a recent release)
• [x] Linux (to newest release within LTS, or new LTS)
• [x] Kubernetes (to 1.22/1.23)
• [x] Fedora Container (to 35)

We we should consider:

• [x] Other Go dependencies?

Metropolis needs to check whether that the rootfs (a.k.a. system partition) mounted by the kernel hasn't been tampered with, at least according to the built/shipped kernel. For that, we need to put more work into how we build images.

The image build flow currently is:

• //metropolis/node:rootfs generates an erofs image containing all node userland code.
• //third_party/linux builds a kernel with EFI stub that hardcodes its boot command line to console=ttyS0 root=PARTLABEL=METROPOLIS-SYSTEM rootfstype=erofs init=/init
• //metropolis/node:image takes the above rootfs and EFI image and combines them into a disk image with an EFI system partition, the erofs image as its own partition, and an empty data partition for Metropolis to use. This is used by the //metropolis/test/launch code to actually run Metropolis, alongside a config protobuf.

We should generally uncouple these things, as currently we have some unwritten expectations about how things should work, but they effectively only work for our test code.

First, we need some way to enforce integrity on erofs system partition images (“Checksum” them). For example, using dm-integrity or dm-verity. I don't know how these works, but this should likely checksum an erofs image (perhaps adding some supporting structures to it) and yield the resulting image and an accompanying checksum (probably in some small protobuf file).

Then, we need to have a way to “Lock” a Kernel that it should expect an erofs with such-and-such integrity guarantees (ie. a given checksum). This should be done without having to rebuild the kernel. @lorenz knows of some UEFI/Linux feature that should allow us to do this by adding a command line to an existing EFI payload. We shouldn't end up with something that itself isn't further checkable, so keeping to a single EFI image is IMO the best option.

Finally, the kernel and erofs images must be “Joined” into a pair where a we get an EFI binary and erofs image that will work together. This can then be piped again into //metropolis/node:image (which perhaps should live in //metropolis/test/launch)?

I think these features should be part of the build process, and not an end-user tool, as users are unlikely to want to dynamically build such images outside of the build system - instead, they will either use a release from Monogon or do their own build if they want to run a patched/forked version of Metropolis. Perhaps later we will make up some sort of artifact format (ie. tarball :) ) that contains both of these together as a single file for redistribution, but for now just a Bazel target with two outputs should be good enough.

Or, graphically:

   .---------------------------.          .---------------------------.
   | erofs image (no checksum) |          | kernel image (no cmdline) |
   |  //metropolis/node:rootfs |          |    //third_party/linux    |
   '---------------------------'          '---------------------------'
               |                                      |
               | “Checksum”                           |  
               :----------------------.               |
               |                      |               |
               V                      V               |
   .-------------------------.  .----------------.    |
   | erofs image  (checksum) |  | checksum proto |    |
   '-------------------------'  '----------------'    |
               |                      | .-------------'
               |                      | | “Lock”
               |                      V V
               |              .---------------------------.
               |              | kernel image (expects     |
               |              | erofs image with checksum |
               |              '---------------------------'
               |                      |
               | .--------------------'
               | | “Join”
               V V
      .--------------------------------------.
      | joined kernel EFI and erofs image    |
      | //metropolis/node:joined.{efi,img} ? |
      '--------------------------------------'

     

Note: I came up with all these names right now - these are not written in stone, and neither is this design. Whoever implements this should probably first write a short design document for this :).

Not in scope: further signing this joined pair into some sort of self-standing release.

Not in scope: building an installer or anything that is 'ready to use'.

@msgctl and @lorenz have been doing some troubleshooting of Metropolis on real hardware.

The symptoms are that after the firmware boots into the image, we get a blank screen. This might be our EFI stub, this might be the Linux EFI stub we then launch, it might be some Linux configuration. It probably doesn't come back after calling ExitBootServices.

@lorenz has just got some hardware to debug this (LPC bus sniffing for outb() tombstone debugging)

(originally reported by @lorenz in T968)

Hit during tests (<<0.1% probability):

supervisor E0412 12:24:11.249739 supervisor_processor.go:198] Runnable root.enrolment died: returned error when NODE_STATE_NEW: could not make and mount data partition: formatting encrypted block device: wait: no child processes

Likely caused by a race condition between pid1 reaping and Go's command.Run().