Overview

Oniro offers comprehensive solutions for updating IoT devices both during development, in the factory and in the field. This document acts as a walk-through for system integrators that want to get familiar with the concepts and prototype the technology on their own.

Linux-based devices

Oniro systems derived from the oniro-image-base recipe support a range of options for performing system updates. To take advantage of those features, system integrator must perform some steps specific either to the product or to the vendor.

In general, the updates are applied to the immutable operating system image. Mutable application and system state is not modified by the update process. For firmware Oniro integrates with LVFS and the firmware update manager. Application updates, for programs that live outside of the system image, need a specific system, for example a specific container manager.

Per-vendor RAUC signing keys

All products based on Oniro reference images MUST generate and maintain a signing key which is used to verify updates. Oniro provides a default, known and insecure key with short validity span (365 days) for testing. This key must not be used for production. During the build, bitbake will emit a message similar to this one, if the insecure key is used:

WARNING: rauc-1.6-r0 do_install: The image is using a known, insecure test key for verifying RAUC bundles. Do not use this in production systems.

To provide a custom signing set the bitbake variable RAUC_KEYRING_FILE. See the rauc_%.bbappend file in meta-oniro-core for details. The gen-keys.sh script present in the repository can be used as a starting point to obtain production keys.

IMPORTANT Production keys should be valid for much longer than one year. A device which was manufactured and left in storage for a long time will verify validity of updates using this key. At minimum, we recommend generating keys that are valid for ten years. See RAUC documentation about bundle verification for additional documentation. Consider using a key-chain to allow providing updated keys over time.

Per-product image and bundle recipes

Products based on Oniro reference images should maintain a pair of recipes - one which constructs a system image ready to be programmed into the device - and another one which can be used to update an existing system. Please look at oniro-image-base and oniro-bundle-base recipes for inspiration and as a starting point.

Both RAUC and NetOTA contain a system for identifying updates compatible with the device. RAUC uses a compatible string, which is placed both inside the system image and inside each update bundle. NetOTA uses several fields that provide greater flexibility into selecting the image to pick, simplifying configuration requirements in typical cases. At the extreme end, NetOTA allows to restrict updates down to specific MACHINE value. Details about those constraints are important to understand if an image and bundle recipe are shared across different physical hardware systems.

Per-product RAUC configuration

RAUC needs to know about the compatible updates and about the slots that describe the location of block devices used for A/B updates. RAUC also needs to be able to reject correctly signed updates aimed at other devices, by checking the compatible string stored inside the update bundle and inside the running system image.

Modify rauc/rauc_%.bbappend and create a number of machine-specific variable overrides:

1) Set RAUC_COMPAT to a string identifying compatible devices. The value should be human readable, for example "Foo Corp. Base Board X".
2) Set RAUC_SLOT_A and RAUC_SLOT_B to a pair of Linux device paths that identify the block devices used for storing RAUC system slots. Normally those are two consecutive partitions on the primary storage device like eMMC, NVME or SATA.

You may also consider using a per-product signing key by setting RAUC_KEYRING_FILE. Consult the section about per-vendor RAUC signing keys for more details.

Per-product SysOTA configuration

SysOTA needs to know about the product bootloader type and should know about the make and model of your product.

Modify sysota/sysota_%.bbappend and create a number of machine-specific variable overrides:

1) Set SYSOTA_BOOTLOADER_TYPE to to the type of bootloader used by your product. For example, pi-boot for all Raspberry Pi-based systems, GRUB for all UEFI systems or u-boot for certain embedded platforms. Note that depending on the bootloader type, some additional work may be required. This is documented below this list.
2) Set SYSOTA_MAKER to the name of the vendor, for example "Foo Corp.". This is similar to the first portion of RAUC_COMPAT but is kept distinct to allow for interactions where a generic white box device becomes re-programmed and re-labeled as a new device. Some future interactions will happen when the maker value changes. Those are not defined yet.
3) Set SYSOTA_MODEL to the name of the device, for example "Gizmo 2000". Unlike the second part of RAUC_COMPAT this string should identify the actual software that is running on top of the hardware to make it something unique, not the hardware inside the box.

Local update with RAUC

The first step of verifying that the update system is working is to build the update bundle (for example oniro-bundle-base), copying the bundle file to the device and performing a local update. This provides the following feedback:

1) Bootloader is set up correctly across all the involved components
2) Signing keys are set up correctly
3) RAUC compatible string is set up correctly

Before the update store the output of the rauc status command. Fresh out of the build system you should see that RAUC is using the A slot. Note that RAUC numbers slots internally and they may have the opposite ordering to what you may expect (i.e. slot A may be numbered 1 and slot B may be numbered 0). This is expected.

RAUC update bundles are using the .raucb extension. Copy the file by any means available (e.g. scp or via removable storage medium) and run the command rauc install ./path/to/your/bundle.raucb as root. RAUC will display a number of messages as it makes progress. The the board will reboot automatically. If the update applies correctly and the system is in a good state after rebooting the update transaction will be automatically committed. On soft-failure the board will reboot back to the old system. In case of deeper problems either the hardware watchdog or a human operator will power-cycle the device. Rollback happens automatically.

After the update consult the output of rauc status. You should now see that the other slot is being used. After the first update you should see slot B as active. Subsequent updates should cycle between the two available slots. There's some more information shown by rauc status --detailed, notably the detailed status of each slot.

In case something goes wrong double check if you did all the customization steps described here correctly. In particular look at the contents of /etc/rauc/system.conf and /etc/sysota/sysotad.conf. They should not have placeholder values. RAUC should not be configured to use oniro-insecure-cert.pem.

If all else fails please reach out to Onrio engineers.

Over-the-air update with HawkBit

This step assumes that local updates work fine. You should verify that before attempting to use HawkBit.

This step verifies that your HawkBit system is configured correctly and that the devices can reach it over the network. Setting up HawkBit is out of scope of this document as there is extensive first-party documentation provided by that project. A minimal local deployment can be done with Docker and the one-liner docker run -p 8080:8080 hawkbit/hawkbit-update-server:latest.

HawkBit needs to know about the software that will be delivered. The model defined by HawkBit is well documented but a short summary is provided here. All actual files that are uploaded to the update server are associated with software modules. Each software module has a type that defines some constraints. Software modules are like Linux system packages at a specific version. Software modules are collected inside distribution sets. Distribution sets are like Linux system images with a set of packages, each at a specific version. Each distribution set also has a type that defines additional constraints. A default HawkBit installation comes with both software module types and distribution set types sufficient for our operation but you will have to define a software module and distribution set to complete the first update.

1) HawkBit needs to know about each device in the fleet. In HawkBit, those are known as targets. Using the web dashboard, the API or command line tool like hawkbitctl create a single target. Define a controller ID string which identifies the device. For small-scale testing any identifier may be used (e.g. "foo-1"). Practical fleets need a scheme to define this namespace. It is important to note, that an unique controller ID needs to be provisioned into each device.
2) Create a distribution set (the web dashboard calls those simply distributions) describing a versioned snapshot of your product build. Going with the naming convention described earlier, it may be "Gizmo 2000" with a date-based version "2022.04.21-1", the -1 suffix lets us have many images a day. The exact version scheme is entirely up to the integrator, there is no prescribed model.
3) Create a software module containing the system update bundle. Pick the OS type, version just like before and give it a name that describes both the content (RAUC bundle) the image (Gizmo 2002) and machine if there may be any ambiguity.
4) Upload the bundle to the newly created module.
5) Assign the newly created module to the distribution. This is done by dragging the module onto the distribution set in the distributions tab. HawkBit dashboard will prompt to confirm the assignment.
6) Assign the newly created distribution set to the target created earlier. This is similarly done by dragging the distribution set onto the target name and following with the confirmation prompt.
7) In the system configuration page allow devices to authenticate using their target security token. This is off by default.

This is completes the HawkBit side of the story. We need to provision the device to talk to our instance before anything happens. There is no support for structured provisioning yet. For the moment you need to interactively create a file in the device file system. Create /etc/rauc-hawkbit-updater/config.conf based on /usr/share/rauc-hawkbit-updater/example.conf and change the ownership to rauc-hawkbit (this is relevant).

You must set the following fields, at minimum:

• target_name - this is the controller ID of the target you've created in step 1). Consult the dashboard for the value.
• auth_token - HawkBit auto-generates the token for each created target. Consult the dashboard for the value.
• hawkbit_server - domain name or IP of your HawkBit server. When running locally from Docker it can be the IP address of your computer, port number is 8080 by default. Use ip addr list (Linux) or ipconfig (Windows) to find the IP addresses of your computer. Pick the one that is reachable by the device.

Once you do this, the service will start automatically and if you had followed earlier instructions, will download and apply the update automatically. The device will then schedule a reboot which occurs after three minutes. You can look at the system logs (journalctl) to see what had occurred before the reboot takes place. Note that the logs are ephemeral and will not persist across reboots.

Similarly to how local update was tested, please consult the output of rauc status after the boot. In addition, depending on your bootloader type you may see interesting messages on the serial port. For example GRUB will note that a one-time boot into a specific slot is performed. Reboot again to see that the update is persisted. Reference images do not contain specific service health checks and will succeed if the kernel and userspace boot enough for SysOTA to commit the transaction.

In case something goes wrong check the status of the rauc-hawkbit-updater.service with systemctl status rauc-hawkbit-updater.service. You may look at more complete journal logs with journalctl -u rauc-hawkbit-updater.service. You may need to start the service manually in case you are making changes on the first try. The service will start automatically on subsequent boots, once the device is provisioned.

Over-the-air update with NetOTA

This step assumes that local updates work fine. You should verify that before attempting to use HawkBit.

This step verifies that your NetOTA repository is configured correctly and that the devices can reach it over the network.

(TBD)

Zephyr-based devices

TBD

Known Issues

There are some known limitations of the current implementation.

Features and bugs on the 2022 roadmap

The following known issues are planned to be addressed during the Oniro 2022 "Goofy" development cycle.

1) GRUB EFI binary is not updated yet (https://gitlab.eclipse.org/eclipse/oniro-core/sysota/-/issues/8)
2) GRUB configuration lives outside of the EFI binary and is not updated yet. https://gitlab.eclipse.org/eclipse/oniro-core/oniro/-/issues/523
3) U-boot is not supported yet (https://gitlab.eclipse.org/eclipse/oniro-core/sysota/-/issues/9, https://gitlab.eclipse.org/eclipse/oniro-core/sysota/-/issues/10 and
4) rauc-hawkbit-updater has limited feature set. An update process which fails and gets rolled back is not reported back to HawkBit as failed. (https://gitlab.eclipse.org/eclipse/oniro-core/sysota/-/issues/20)
5) rauc-hawkbit-updater is incompatible with rauc handlers which need to reboot the device directly, and cannot use the automatic reboot functionality offered by the updater tool. To work around that, the reboot is scheduled in the background, causing a configurable delay that only exists to avoid this limitation. (https://gitlab.eclipse.org/eclipse/oniro-core/sysota/-/issues/20)
6) Firmware updates are not coordinated with OS updates (https://gitlab.eclipse.org/eclipse/oniro-core/sysota/-/issues/16)

Features not on the 2022 road-map

There are some known limitations that will not be addressed during the 2022 release. Those are likely to happen next year but consult an updated version of this document for details.

1) x86-64 Secure Boot is not supported yet.
2) ARM Trusted firmware is not supported yet.

Glossary

SysOTA

Transnational update system for immutable Linux-based operating systems. SysOTA implements boot loader integration, transactional update and rollback and certain cloud connectors, such as NetOTA.

SysOTA depends on RAUC for slot management and bundle authentication. By using RAUC as a lower-level library we make it easy to integrate with existing software, such as HawkBit.

RAUC

Robust, flexible and configurable update system.

RAUC defines a collection of slots, handlers and optional hooks that organize the update process. RAUC handles bundle verification and authentication as well as the actual process of writing updated data to slots.

HawkBit

IoT fleet management and configuration service.

HawkBit allows tracking the state of individual devices, offers vast APIs for configuring and updating devices with precision and accuracy required in industrial environments.

NetOTA

Stateless network service, protocol and repository format for delivering software updates to devices without having to track device identity and operational state.

Oniro devices may be provisioned with NetOTA configuration to receive automatic updates on demand, allowing each device to update at its own pace and when convenient.

NetOTA allows creating products that are not centrally managed by a fleet operator. The model allows software publishers to offer updates to a wide range of devices easily, minimizing configuration and complexity for the user.

Advanced publishing features allow providing multiple software versions at the same time, aiding development in CI/CD environments and offering image best optimized for each device type.

Update bundle

Self-sufficient update element which can be installed on an existing Oniro device to update the kernel, operating system components and some boot components.

Oniro uses RAUC bundle format which is a squashfs file system image with meta data, signatures and update images for the system slot.