citadelhub
Inapoi la Blog
Embedded
·12 min read

Embedded Linux in Production: What It Takes to Ship Yocto-Based Devices

Getting embedded Linux running on a dev board is the easy part. Shipping a production Yocto image with secure boot, OTA updates, and regulatory compliance is where the real engineering begins.

There is a moment in every embedded Linux project where the euphoria of seeing a login prompt on your eval board gives way to a sobering realization: you are nowhere close to shipping. The delta between "it boots" and "it ships" is enormous — spanning security, reliability, update infrastructure, regulatory paperwork, and years of ongoing maintenance.

This article walks through the full production pipeline for Yocto-based embedded Linux devices, drawn from real-world experience shipping products on NXP i.MX8, TI AM62x, and STM32MP1 platforms. If you are evaluating whether to build this capability in-house or looking for a partner who has done it before, this should give you a clear picture of what the work actually entails.

BSP Development: The Foundation

The Board Support Package is where everything starts. Vendor-provided BSPs from NXP, Texas Instruments, or STMicroelectronics give you a baseline — a kernel, a bootloader, and a machine configuration — but they are rarely production-ready out of the box.

Machine Configuration

Your machine.conf defines the hardware contract between Yocto and your board. A typical production machine config goes well beyond what the vendor provides:

# meta-myproduct/conf/machine/myproduct-imx8mm.conf
#@TYPE: Machine
#@NAME: MyProduct i.MX8M Mini
#@SOC: i.MX8MM

MACHINEOVERRIDES =. "imx-boot-container:mx8mm:"

require conf/machine/include/imx8mm-evk.inc

# Kernel
PREFERRED_PROVIDER_virtual/kernel = "linux-imx"
PREFERRED_VERSION_linux-imx = "6.6%"
KERNEL_DEVICETREE = "freescale/imx8mm-myproduct-rev2.dtb"

# U-Boot
PREFERRED_PROVIDER_virtual/bootloader = "u-boot-imx"
UBOOT_CONFIG = "myproduct"
SPL_BINARY = "spl/u-boot-spl.bin"

# eMMC partitioning
WKS_FILE = "imx8mm-myproduct.wks.in"

# Serial console
SERIAL_CONSOLES = "115200;ttymxc1"

# Firmware
MACHINE_FIRMWARE += "linux-firmware-ath10k"

The critical detail most teams miss: pin your kernel and U-Boot versions explicitly. Vendor BSP layers frequently bump versions between releases. An uncontrolled kernel upgrade mid-project has derailed more schedules than any single technical issue I have seen.

Device Tree Overlays

Production boards inevitably diverge from eval kits. You will have custom PMICs, different DDR configurations, peripheral connections that do not match any reference design. Device tree overlays let you layer these customizations without forking the vendor's base DTS:

/dts-v1/;
/plugin/;

&i2c3 {
    status = "okay";
    clock-frequency = <400000>;

    fuel_gauge: bq27441@55 {
        compatible = "ti,bq27441";
        reg = <0x55>;
        design-capacity = <3200>;
        design-energy = <12160>;
    };
};

&usdhc2 {
    /* Remap SD card detect for custom carrier board */
    cd-gpios = <&gpio2 12 GPIO_ACTIVE_LOW>;
};

Use overlays from the start. Teams that modify vendor device trees directly end up in merge hell when a BSP update lands.

Vendor BSP Integration

This is the most politically charged part of any Yocto project. Vendor BSP layers (like meta-freescale, meta-ti) are maintained with varying degrees of quality. The NXP meta-imx layer, for instance, carries hundreds of patches against the upstream kernel that may never be mainlined.

The pragmatic approach:

  • Track the vendor's LTS kernel (e.g., NXP's 6.6.x LTS for i.MX8). Do not chase mainline unless you have a compelling reason and the engineering budget to validate it.
  • Isolate your customizations in a separate BSP layer (meta-myproduct) that sits above the vendor layer. Never modify vendor layer files directly.
  • Document every vendor patch you depend on. When the next BSP release drops, you need to know which of your workarounds are still necessary.

Image Architecture: Designing for Reliability

Consumer electronics and industrial devices do not get the luxury of a sysadmin who can SSH in and fix things. Your image architecture must assume that once deployed, the device is on its own.

Read-Only Root Filesystem

A writable rootfs on an embedded device is a ticking time bomb. Power loss during a write can corrupt the filesystem. Wear leveling on cheap eMMC modules is unpredictable. The solution is straightforward:

# In your image recipe
IMAGE_FEATURES += "read-only-rootfs"

# Overlay for writable state
DISTRO_FEATURES += "overlayfs"

Layer a tmpfs or a dedicated partition for runtime state using OverlayFS:

/           -> read-only ext4 or squashfs
/var        -> overlayfs (tmpfs upper + rootfs lower)
/etc        -> overlayfs (persistent partition upper + rootfs lower)
/data       -> dedicated ext4 partition for application data

SquashFS is worth considering for the rootfs. It compresses well (typically 40-60% smaller than ext4), reads are fast due to block-level decompression, and it is inherently read-only. The trade-off is slightly higher CPU load on random reads.

A/B Partition Scheme

An A/B (or "dual-copy") partition layout is the industry standard for devices that need reliable updates:

+--------+--------+--------+--------+----------+--------+
| Boot   | Root A | Root B | Config | Data     | OTA    |
| (FAT)  | (ext4) | (ext4) | (ext4) | (ext4)  | (ext4) |
| 64MB   | 512MB  | 512MB  | 32MB   | dynamic  | 256MB  |
+--------+--------+--------+--------+----------+--------+

Partition layout is defined in a WIC kickstart file:

bootloader --ptable gpt
part /boot --source bootimg-partition --fstype=vfat --label boot --active --align 8192 --size 64
part / --source rootfs --fstype=ext4 --label rootfs-a --align 8192 --size 512
part --source rootfs --fstype=ext4 --label rootfs-b --align 8192 --size 512
part /config --fstype=ext4 --label config --align 8192 --size 32
part /data --fstype=ext4 --label data --align 8192 --size 1024
part /ota --fstype=ext4 --label ota --align 8192 --size 256

The key requirement: the bootloader must know which slot is active and be able to switch on failure. U-Boot's bootcount mechanism handles this — after a configurable number of failed boots, it automatically rolls back to the previous slot.

OTA Update Strategies

Over-the-air updates are table stakes for any connected device. The three dominant open-source frameworks each make different trade-offs.

SWUpdate

SWUpdate is the most flexible option. It processes .swu files — essentially CPIO archives containing update artifacts and a Lua-scriptable installation logic:

# sw-description for a dual-copy rootfs update
software = {
    version = "2.4.1";
    hardware-compatibility: ["rev2", "rev3"];

    images: ({
        filename = "rootfs.ext4.gz";
        type = "raw";
        compressed = "zlib";
        device = "/dev/mmcblk0p3";  /* inactive slot */
        installed-directly = true;
    });

    scripts: ({
        filename = "post_update.sh";
        type = "shellscript";
    });
};

Strengths: Extremely customizable, supports symmetric and asymmetric updates, Lua scripting for complex logic, good hawkBit integration for fleet management.

Weaknesses: You own the update orchestration logic. The flexibility means more engineering work to get right.

Mender

Mender provides a more opinionated, full-stack solution — client, server, and fleet management UI included. It handles A/B rootfs updates out of the box with automatic rollback.

Strengths: Fastest time-to-first-OTA. Managed server option (Mender Enterprise) removes backend maintenance burden. Delta updates via mender-binary-delta.

Weaknesses: Less flexible than SWUpdate for non-standard update flows. The managed service has per-device pricing that adds up at scale. Tighter coupling to their server infrastructure.

RAUC

RAUC sits between SWUpdate and Mender in philosophy. It uses a bundle format with a manifest and cryptographic verification, integrating with D-Bus for status reporting:

Strengths: Clean architecture, strong cryptographic verification model, native systemd integration, good asymmetric update support for complex multi-slot setups.

Weaknesses: Smaller community than SWUpdate or Mender. Documentation assumes significant prior knowledge.

Delta Updates

Full rootfs updates over cellular connections are expensive. Delta update support is critical for bandwidth-constrained deployments:

  • Mender offers mender-binary-delta as a commercial add-on (typically 50-70% bandwidth reduction)
  • SWUpdate can integrate with librsync or zchunk for delta generation
  • RAUC supports adaptive updates via casync for chunk-based deduplication

For fleets larger than a few thousand devices, the bandwidth savings from delta updates typically pay for the engineering investment within months.

Security Hardening

A compromised embedded device is not just a data breach — it is a physical safety risk. The secure boot chain must be airtight from the first instruction executed by the SoC.

Secure Boot Chain

On NXP i.MX platforms, the Hardware Assurance Block (HAB) provides ROM-level verified boot:

  1. SoC ROM verifies the signed Secondary Program Loader (SPL) using keys burned into OTP fuses
  2. SPL verifies U-Boot proper using HAB
  3. U-Boot verifies the kernel and device tree using FIT image signatures
  4. Kernel verifies the rootfs using dm-verity

The critical step that makes this irreversible: fusing. Once you blow the SRK_LOCK fuse on an i.MX8, the chip will only boot images signed with your keys. There is no undo. Fuse in production, never on your only prototype.

# Generate HAB PKI tree (do this ONCE, store keys in HSM)
hab4_pki_tree.sh \
    --key-length 4096 \
    --duration 20 \
    --num-srk 4 \
    --srk-ca-cert SRK1_sha256_4096_65537_v3_ca_crt.pem

# Sign U-Boot SPL
cst -i csf_spl.txt -o csf_spl.bin

dm-verity for Rootfs Integrity

dm-verity provides block-level integrity verification of the rootfs using a Merkle hash tree:

veritysetup format /dev/mmcblk0p2 /dev/mmcblk0p5
# Root hash: 4a8b3c...

# In kernel command line:
root=/dev/dm-0 rootfstype=ext4
dm-mod.create="verity,,,ro,0 2097152 verity 1 /dev/mmcblk0p2 /dev/mmcblk0p5 4096 4096 262144 1 sha256 4a8b3c..."

Combine dm-verity with a read-only rootfs and you have a rootfs that is both immutable and cryptographically verified on every read. Any tampering causes I/O errors rather than silent corruption.

Mandatory Access Control

For devices that handle sensitive data or control physical actuators, add SELinux or AppArmor as a second defense layer:

# In local.conf or distro.conf
DISTRO_FEATURES:append = " selinux"
PREFERRED_PROVIDER_virtual/refpolicy = "refpolicy-targeted"

SELinux is more comprehensive but significantly harder to configure correctly for embedded. AppArmor is often the pragmatic choice — profile-based confinement is easier to reason about when you control every process on the device.

Key Management

Never store signing keys on developer workstations. Use an HSM (Hardware Security Module) — even a YubiHSM 2 at $650 is dramatically better than a PGP key on someone's laptop. For CI pipelines, cloud HSMs (AWS CloudHSM, Azure Dedicated HSM) integrate with most signing toolchains.

Rotate signing keys on a defined schedule. Embed multiple SRK keys in the OTP fuses (i.MX supports four) so you can revoke a compromised key without bricking the fleet.

Build Reproducibility

A Yocto build that works on one engineer's machine but fails on another is a project risk. Reproducibility is not optional.

Layer Management

Pin every layer to a specific commit. Use a repo manifest or kas configuration:

# kas/myproduct.yml
header:
  version: 14

machine: myproduct-imx8mm
distro: myproduct

repos:
  poky:
    url: "https://git.yoctoproject.org/poky"
    commit: "ab4c2ef..."  # scarthgap-5.0.4
    layers:
      meta:
      meta-poky:

  meta-openembedded:
    url: "https://git.openembedded.org/meta-openembedded"
    commit: "f3e7a90..."
    layers:
      meta-oe:
      meta-networking:
      meta-python:

  meta-freescale:
    url: "https://github.com/Freescale/meta-freescale.git"
    commit: "de89b31..."

  meta-swupdate:
    url: "https://github.com/sbabic/meta-swupdate.git"
    commit: "c4f1d72..."

  meta-myproduct:
    path: "sources/meta-myproduct"

kas has become the de facto standard for managing Yocto build configurations. It handles layer checkout, bblayers.conf generation, and local.conf setup in a single declarative YAML file. This eliminates the "works on my machine" class of build failures entirely.

SSTATE Cache and Build Performance

A clean Yocto build for an i.MX8 target takes 4-8 hours on a modern 16-core workstation. SSTATE (shared state) caching is what makes iterative development viable:

# local.conf — shared SSTATE mirror
SSTATE_MIRRORS = "file://.* https://sstate.mycompany.com/PATH;downloadfilename=PATH"
SSTATE_DIR = "${TOPDIR}/sstate-cache"

# Enable hash equivalence server for even better cache hits
BB_HASHSERVE = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"

For CI, run a dedicated SSTATE mirror server. A well-maintained cache reduces incremental build times from hours to minutes. The hash equivalence server (introduced in Yocto 3.1) further improves cache hit rates by recognizing when different inputs produce identical outputs.

Autobuilder CI

Your CI pipeline should build every commit that touches BSP or image recipes. A typical setup:

# GitLab CI example
build-image:
  stage: build
  tags: [yocto-builder]
  script:
    - kas build kas/myproduct.yml
    - kas build kas/myproduct.yml:kas/test-image.yml
  artifacts:
    paths:
      - build/tmp/deploy/images/myproduct-imx8mm/
  cache:
    key: sstate-${CI_COMMIT_BRANCH}
    paths:
      - build/sstate-cache/

Dedicate beefy machines for this — 32 cores, 64GB RAM, NVMe storage. Yocto builds are I/O and CPU intensive. Trying to run them on standard CI runners is a recipe for 6-hour pipelines and frustrated engineers.

Testing

Embedded testing is uniquely challenging because you cannot separate the software from the hardware it runs on.

QEMU-Based CI

QEMU testing catches a surprising number of issues without any hardware:

# Build a QEMU-compatible image alongside your hardware image
MACHINE = "qemux86-64"
# or for ARM targets:
MACHINE = "qemuarm64"

Use QEMU in your CI pipeline for userspace testing, init system validation, and application-level integration tests. It will not catch hardware-specific bugs, but it will catch regressions in recipes, configuration, and application logic — which account for the majority of issues.

Hardware-in-the-Loop with LAVA

For hardware-specific validation, LAVA (Linaro Automated Validation Architecture) is the standard framework. LAVA manages a pool of physical devices, flashing images, running test suites, and collecting results:

# LAVA job definition
device_type: imx8mm-myproduct
job_name: smoke-test-nightly
timeouts:
  job:
    minutes: 30
  action:
    minutes: 10

actions:
  - deploy:
      to: flasher
      images:
        image:
          url: https://ci.mycompany.com/images/myproduct-image-dev.wic.gz
          compression: gz

  - boot:
      method: u-boot
      commands: nfs
      prompts:
        - "myproduct login:"

  - test:
      definitions:
        - repository: https://git.mycompany.com/qa/lava-tests.git
          from: git
          path: tests/boot-smoke.yaml
          name: boot-smoke-test
        - repository: https://git.mycompany.com/qa/lava-tests.git
          from: git
          path: tests/peripheral-check.yaml
          name: peripheral-validation

The infrastructure investment for LAVA is significant — you need relay boards, serial consoles, power distribution units, and network-bootable device farms. But for any product shipping more than a few hundred units, it pays for itself by catching hardware interaction bugs before they reach the field.

Test Strategy

A practical test pyramid for embedded:

  • Unit tests (host-compiled): Business logic, parsers, protocol handlers. Run on every commit.
  • QEMU integration tests: Init sequence, service startup, configuration management. Run on every commit.
  • Hardware smoke tests: Boot, peripheral enumeration, basic I/O. Run nightly.
  • Hardware soak tests: Stress testing, power cycling, thermal cycling, OTA update loops. Run weekly.
  • Compliance pre-scans: EMC pre-compliance measurements. Run before every hardware revision.

Regulatory Compliance

Software engineers often underestimate the regulatory burden. CE marking, FCC certification, and industry-specific standards all have software implications.

EMC and Radio Compliance

If your device has wireless connectivity, FCC (US) and CE/RED (EU) certification is mandatory. From the software side, this means:

  • Transmit power limits must be enforced in firmware/driver configuration. The FCC tests the device as shipped, including software-configurable power levels. If your driver allows a region override that exceeds legal limits, certification fails.
  • Frequency band restrictions must match the target market. Linux's CRDA/wireless-regdb handles this, but you must ensure the correct regulatory domain is set and locked at boot.
  • DFS (Dynamic Frequency Selection) on 5GHz bands requires radar detection that meets specific timing requirements. This is typically handled in firmware, but verify your driver/firmware combination passes the relevant test suite.
# Ensure regulatory database is included and region is locked
IMAGE_INSTALL += "wireless-regdb"
# In system config:
# REGDOMAIN=DE

Documentation Trail

Certification labs require detailed documentation of the software configuration:

  • Exact kernel version, config, and wireless driver versions
  • All RF-related parameters and their default/maximum values
  • Evidence that end users cannot modify regulatory-controlled parameters
  • Software version management showing that OTA updates cannot alter certified RF behavior

Build your CI pipeline to generate this documentation automatically. A script that extracts LINUX_VERSION, driver versions, and RF parameters from the build output saves weeks of back-and-forth with the test lab.

Common Pitfalls

BSP Vendor Lock-In

Every vendor BSP layer carries out-of-tree patches. The deeper your product integrates with vendor-specific kernel features, the harder it becomes to migrate. Mitigations:

  • Abstract hardware access behind well-defined interfaces. If your application talks to a sensor through a vendor-specific ioctl, wrap it.
  • Track upstream progress on the features you depend on. NXP's i.MX kernel patches are slowly being upstreamed — know which ones have landed in mainline so you can plan your migration.
  • Budget for BSP updates in your project plan. A vendor BSP update is not a git pull — it is typically 2-4 weeks of integration and regression testing.

Kernel Version Management

Long-lived products face a painful choice: stay on the vendor's kernel and lose upstream security fixes, or move to a newer kernel and lose vendor support. The answer is almost always to stay on the vendor kernel and backport security fixes. The LTS kernels maintained by vendors (NXP's 6.6.x, TI's 6.6.x) receive critical CVE patches, but the lag can be weeks or months.

Maintain a CVE tracking process. Subscribe to linux-cve-announce, cross-reference against your kernel config (unused subsystems mean many CVEs are not applicable), and have a defined SLA for patching critical vulnerabilities.

Long-Term Maintenance

Consumer electronics have a 3-5 year product lifecycle. Industrial and automotive devices can be 10-15 years. Your Yocto layer must be maintainable over that entire period:

  • Use LTS Yocto releases (Kirkstone, Scarthgap). Community support lasts 2+ years, with commercial support options extending further.
  • Minimize layer dependencies. Every third-party layer is a maintenance liability. Evaluate whether pulling in meta-some-library is worth it, or if adding a single recipe to your own layer is simpler.
  • Automate CVE scanning. Yocto's built-in cve-check class cross-references your package versions against the NVD:
# In local.conf
INHERIT += "cve-check"
CVE_CHECK_SKIP_RECIPE += "linux-imx"  # vendor handles kernel CVEs

Run this in CI. A monthly CVE review is the minimum for any shipping product.

Closing Thoughts

Shipping a Yocto-based product is a significant engineering undertaking. The build system, the update infrastructure, the security chain, the regulatory compliance — each of these domains requires deep specialized knowledge. Teams that underestimate the scope end up with delayed launches, insecure devices, or products that cannot be updated in the field.

The investment is worth it. A well-architected embedded Linux platform gives you a foundation that scales across product variants, supports decade-long lifecycles, and provides the security and reliability that customers expect.

At Citadel Tech Hub, our embedded systems team has shipped Yocto-based products across industrial IoT, automotive, and consumer electronics. We work across the full stack — from Yocto BSP development and custom kernel work to AUTOSAR Classic/Adaptive integration and Zephyr RTOS for constrained MCU targets. Whether you need a production BSP built from scratch, an OTA update infrastructure designed and deployed, or ongoing maintenance for a long-lived product line, we bring the experience to get it right the first time. Get in touch to discuss your embedded project.

Pregatit sa Incepi Proiectul?

Hai sa discutam cum te putem ajuta sa-ti aduci ideile la viata. Obtine o consultatie gratuita astazi.