How to use Twister to cycle test Zephyr devices

Twister is the test runner tool built into Zephyr RTOS. At Golioth we use it extensively for our Hardware-in-the-Loop testing. I also use it locally for device testing. However, it can be a pain to set up environment variables and remember the test syntax.

A couple of weeks ago I wrote a small helper script to assist in cycle testing. It configures everything, calls Twister, and counts the successes.

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

#export CI_MIMXRT1024_EVK_PORT=/dev/ttyACM0
#export PORT_VAR=/dev/ttyACM0
#export SNR_VAR=728343547
#export GOLIOTH_API_KEY="golioth-project-api-key"
#export GOLIOTH_CREDENTIALS_FILE="${SCRIPT_DIR}/credentials_empty.yml"
#export hil_board=mimxrt1024_evk
#export west_board=mimxrt1024_evk

export CI_NRF52840DK_NRF52840_PORT=/dev/ttyACM0
export PORT_VAR=/dev/ttyACM0
export SNR_VAR=1050266122
export GOLIOTH_API_KEY="golioth-project-api-key"
export GOLIOTH_CREDENTIALS_FILE="${SCRIPT_DIR}/credentials.yml"
export hil_board=nrf52840dk
export west_board=nrf52840dk_nrf52840

i=0

zephyr/scripts/twister \
    --platform ${west_board} \
    -T modules/lib/golioth-firmware-sdk/examples/zephyr/fw_update \
    --prep-artifacts-for-testing

ret=$?

echo -e "\n############ Build set complete; beginning tests ##############\n"

until [ "$ret" -ne 0 ]
do
    zephyr/scripts/twister \
        --platform ${west_board} \
        -T modules/lib/golioth-firmware-sdk/examples/zephyr/fw_update \
        --device-testing \
        --device-serial ${PORT_VAR} --test-only \
        --west-flash="--skip-rebuild,--dev-id=${SNR_VAR}" \
        -v

    ret=$?
    ((i=i+1))

    echo -e "\n############ Iteration: $i ########## Return Code: $ret ##############\n"
done

play bell.mp3

Code Walkthrough

This was initially used to cycle test OTA updates using the fw_update sample in the Golioth Firmware SDK.

export CI_NRF52840DK_NRF52840_PORT=/dev/ttyACM0
export PORT_VAR=/dev/ttyACM0
export SNR_VAR=1050266122
export GOLIOTH_API_KEY="golioth-project-api-key"
export GOLIOTH_CREDENTIALS_FILE="${SCRIPT_DIR}/credentials.yml"
export hil_board=nrf52840dk
export west_board=nrf52840dk_nrf52840

Our pytest file for that sample looks for a number of key values to be present as environment variables so I added two blocks to this file to set that all up, commenting or uncommenting for the device under test.

zephyr/scripts/twister \
    --platform ${west_board} \
    -T modules/lib/golioth-firmware-sdk/examples/zephyr/fw_update \
    --prep-artifacts-for-testing

I run this from the root of the Zephyr install. The first Twister command builds the firmware. The --prep-artifacts-for-testing flag saves only the binaries and other files necessary to flash a device into a twister-out directory. I’ll explain the other flags later in this post.

ret=$?

echo -e "\n############ Build set complete; beginning tests ##############\n"

until [ "$ret" -ne 0 ]
do
    #Run the twister device test here

    ret=$?
    ((i=i+1))

    echo -e "\n############ Iteration: $i ########## Return Code: $ret ##############\n"
done

play bell.mp3

Because I’m looking to cycle test this code (run many tests one after the other to ensure there are no failures), I set up a loop in the bash script. Immediately after each Twister command, ret=$? is called to store the Twister return code. If successful, a 0 is returned and the loop continues. If an error code is returned the loop breaks and a bell sound plays (using the Linux sox package).

zephyr/scripts/twister \
    --platform ${west_board} \
    -T modules/lib/golioth-firmware-sdk/examples/zephyr/fw_update \
    --device-testing \
    --device-serial ${PORT_VAR} \
    --test-only \
    --west-flash="--skip-rebuild,--dev-id=${SNR_VAR}" \
    -v

The second Twister command uses the binaries from the first to test the code on actual hardware. Here is what each flag is used for:

• --platform: supplies the Zephyr board name
• -T: path of the Zephyr directory of your desired sample.yml file that details each test. Here I’m targeting a specific folder that has two tests. If you leave this blank, Twister will crawl the tree looking for all sample files and running the tests described within.
• --device-testing: tells Twister to run on actual hardware
• --device-serial: serial address of the actual hardware
• --test-only: don’t rebuild the firmware, use binaries from previous Twister build
• --west-flash: use the west flash command for programming and optionally pass some west flags. Here I’m telling west not to rebuild the project, and passing the serial number of the J-Link device as I usually have multiple J-Link devices connected at the same time.
• -v: verbose mode; I like to use this so I can tail -f the twister_harness.log to see which tests pass/fail in real time.

Wrapup

Twister is a sledge-hammer of a test suite, but with a helper script like this you can use it as if it were precision tweezers.

Recently, I needed to increase the network buffers available to an application. This is kind of guesswork as you need enough buffers that you don’t run out but allocating extra buffers wastes RAM. Since I already had this script on hand, I started with a high number of RAM buffers and modified the script to reduce the number by one each cycle (The firmware was also compiled on each step by removing the --test-only flag). Once the test failed I knew exactly where the buffer threshold was located.

I hope this helps. I’ll certainly be using this a lot and will post more as I develop addition improvements.