Fully Open Edge Cloud

How To Install The Scanning Agent On A Server

How To Install The Scanning Agent On A Server
  • Last Update:2022-06-27
  • Version:001
  • Language:en

Agenda

  • Some context on the project
  • Enroll the keys in UEFI (optional)
  • Create and install the UEFI image

This HowTo describes the process to install the scanning agent on a rapid.space server with Secure Boot activated.

Here is the project :
https://lab.nexedi.com/nexedi/initramfs-with-mca/

Prerequisites

  • None
 

Some Context

When booting a system, different programs are run. As a very rough and simplified overview, for a system supporting UEFI we can say that:

  • First UEFI is in control.
  • Then it calls an EFI application (.efi), generally a bootloader (e.g. GRUB).
  • The bootloader in turn calls the OS.
  • To find the user's root file system, a Linux Kernel may use an already prepared temporary file system called initramfs.
  • In our case, the Linux Kernel is directly embedded in the efi application and it uses a custom initramfs.

UEFI provides a feature called Secure Boot (SB) which can verify the efi applications' signature with keys stored by UEFI, so that an attacker cannot modify the boot sequence and tamper with what is run on the machine early during the boot process. Such an attack would make the threat undetectable with the classic forensic tools, i.e. from the already booted OS.

Nexedi developped for Rapid.Space a system going further, which scans the user's root file system to be mounted by initramfs during the initramfs stage, i.e. before the Linux Kernel has taken over, and send the scan to Wendelin for further analysis. To do so, we create a customed initramfs embedding the scanner and sender and we sign it with our own keys. Then, if our keys are enrolled by UEFI, it can verify the integrity of our customed initramfs, loaded by UEFI as an efi application. With this process, we get a complete scan of the to-be-loaded file system each time a server boots. We can detect abnormal file signatures and permissions while being sure than an attacker could not modify the tools we used to create and send the scan.

Note: The scanning tool is called Metadata Collect Agent (MCA), see the project at

https://lab.nexedi.com/nexedi/metadata-collect-agent

So the main components for a user to install this project on a machine are:

  • A set of keys
  • Key enrollment in UEFI.
  • Initramfs image creation and signature.

With the hardest part to master being the second one, documented in uefi-keys/README.md. (But don't worry, it's not THAT hard if you read carefully!)

UEFI keys context

This stage is about enrolling Rapid.Space's keys in the UEFI firmware of the machine. It all happens in the uefi-keys/ directory. cd uefi-keys/

Note that it is usually required one time. If Rapid Space's keys are already enrolled on the machine and they are the one in uefi-keys/keys, then you can proceed to the next step of the current document.

 

UEFI keys enrollment

This stage is about enrolling Rapid.Space's keys in the UEFI firmware of the machine. It all happens in the uefi-keys/ directory. cd uefi-keys/

Note that it is usually required one time. If Rapid Space's keys are already enrolled on the machine and they are the one in uefi-keys/keys, then you can proceed to the next step of the current document.

This steps are only needed if Rapid Space's keys are not already enrolled on the machine. To display the current keys, run efi-readvar (no root access needed). If the Common Name (CN) of the Platform Key (PK) contains the string "Rapid.Space", then Rapid Space's keys are already enrolled and you can probably go back to the root directory of this project. Else, please proceed reading this document. To see the Common Name, scroll up to the top of the efi-readvar output, in the field corresponding to the variable PK, then after "CN=" is the Common Name.

By default, the command listed below require to be logged as root. It will be indicated otherwise.

Install the dependencies

To perform the following instructions, some packages are needed. They are installed by dependencies.sh of this directory.

Run ./dependencies.sh

Create the keys

WARNING: A set of keys is already created here. So the following instruction is probably NOT what you want to do. This section is simply for documentation and you should proceed to the next section.

Run the script ./make_keys.sh (no root access needed). The script creates the keys in keys/.

Sign an UEFI application

You have nothing special to do in this section. This step is documented in the README.md at the root of this project. Once the keys are created, you do not have to follow the rest of this document to sign an image. Although, it is required to actually boot on such an image.

UEFI variables

To display the current keys, run efi-readvar (no root access needed).

  • The PK variable correponds to the Platform Key. There can be only one per machine and it is the top-level key. It is needed to modify the KEK variable.
  • The KEK variable corresponds to the Key Exchange Keys. At least one of them are needed in order to modify the db and dbx variables.
  • The db variable corresponds the databases of authorized keys and certificates. It can be viewed as a whitelist and it contains the signing key of the initramfs images we create and use.
  • The dbx variable corresponds to the databases of forbidden keys and certificates. It can be viewed as a blacklist and has a greater priority than db. That means that if it contains a key or certificate also in db, then the key or certificate is still unauthorized and it cannot be used to boot on the system. We do not use dbx here, as at the moment we enroll only our own key.
  • The MokList variable is not used here, nor is it documented.

If the PK's Common Name (CN) contains "Rapid.Space", then you probably don't need to do anything unless you are in an undocumented use case.

Else, if at least one of the PK, KEK, db and dbx variables has an entry (i.e. efi-readvar displays something that seems meaningful), then proceed to the next section.

Else, the output of efi-readvar should be clear about the fact that the variables has no entries, and you can bypass the next section and jump directly to the section after.

Remove the current keys

Boot the machine and enter the BIOS. There, disable Secure Boot (SB) and remove every keys. The following instructions are based on the UEFI interface of the capri in Lille office and may vary depending on the hardware and the UEFI firmware version.

To remove the current keys, you have to:

  1. Disable Secure Boot

  2. Switch Secure Boot to Custom Mode

  3. Go in Key Management and delete the keys: Platform Key (PK), Key Exchange Keys (KEK), Authorized Signatures (db), Forbidden Signatures (dbx).

Save and exit, and this time efi-readvar should tell that these variables have no entries.

 

Add the keys from Rapid Space

If all the PK, KEK, db and dbx variables have no entries, or after the previous stage is done, add the keys to UEFI as follow. We assume the keys have the name make_keys.sh gave them:

  • Add the Platform Key (the authority key for the machine): efi-updatevar -f PK.auth PK

  • Add the Key Exchange Key (the authority key for the db and dbx certificates and hashes): efi-updatevar -a -c KEK.crt -k PK.key KEK

  • Add the whitelist certificate: efi-updatevar -a -c db.crt -k KEK.key db

  • Run efi-readvar again to check that PK, KEK and db have exactly one entry with the Common Name (CN) containing "Rapid.Space".

 

Troubleshooting

Some general informations:

  • As a general matter, with Secure Boot (SB) disabled and no key, every key should be addable.
  • A Platform Key is needed for Secure Boot (SB) to be enabled.
  • To add or modify a key, you need the private key of the upper level (the Platform Key is its own upper level key).
Common errors
  • Error: Failed to update <var>: Operation not permitted It happens when trying to delete or replace a key. It may be resolved by making it mutable:
lsattr /sys/firmware/efi/efivars/{db,dbx,KEK,PK}*

"----i--------------" before the key means it is set as immutable. To make it mutable:

chattr -i /sys/firmware/efi/efivars/<key>

Example:

chattr -i /sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f

Recommanded: make everything immutable after your operation. To do so, perform the same command with +i instead of -i. Same example:

chattr +i /sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f
  • Error: Cannot write to db, wrong filesystem permissions It may be resolved by providing the upper level private key (after the -k option of efi-updatevar). Note that it can happen even in root mode, as the keys are stored on a separate filesystem.
 

Initramfs image

This stage is about creating and signing an initramfs image with the Metadata Collect Agent (MCA) enabled in it.

 

Install dependencies

Run sudo ./installation/dependencies.sh

 

Install the dracut module and signed initramfs image

Run sudo ./installation/install.sh

Notes:

  • The installation of the dracut module is performed by dracut.module/Makefile.
  • The initramfs image is created by dracut according to the preferences in dracut.module/dracut.conf and the command-line.
  • The (unsigned) initramfs image is then signed by dracut, again according to dracut.module/dracut.conf
 

Secure Boot

Make sure Secure Boot is enabled. It should be in Custom Mode.

 

For developpers...

 

Installation

At the moment, the package names are in installation/packages.sh and should be modified there. This script is sourced by installation/install.sh and dracut.module/configure. Not that at the moment, the configure script is very simple and non-standard.