My OpenBMC userspace development workflow

I recently pushed a couple of tools (overlay and bbdbg) into the openbmc-tools repository that help me develop userspace software for OpenBMC. This post describes how I work with all the different tools involved.

Without modifying anything in a stock OpenBMC image, the process to generate the build artifacts is to issue bitbake obmc-phosphor-image. Building a stock image doesn’t help much with developing userspace applications though, so there needs to be more to the story.

An approach to building specific applications to run in the target environment generated by bitbake is to use the SDK, configuring your shell session so you can build arbitrary applications for the target. But there are at least two pitfalls to the SDK: It takes a long time to build and install, and in OpenBMC’s case there’s no promise of ABI stability. In general this means building an SDK that’s specific to the state of the target environment, down to the specific commit the target image was built from.

With those constraints in mind, I prefer to try routes that don’t involve the SDK. bitbake is the tool for building the images, so it would be useful if we could reuse it or its environment. With devtool it turns out we can. Here’s a run-down of the process I follow:

On the development machine, build a base image on which we’ll test our work

1. cd ~/src/openbmc/openbmc
2. . setup p10bmc
3. bitbake obmc-phosphor-image

From here we deploy the resulting image onto the target in the usual fashion.

On the development machine, set up a repository to work on

1. cd ~/src/openbmc
2. git clone https://github.com/openbmc/dbus-sensors.git
3. cd dbus-sensors
4. git checkout -b my-hacks
5. vi src/NVMeSensorMain.cpp
6. git commit -a -m "my hacks"

In this case we’ve modified src/NVMeSensorMain.cpp which forms part of the source for the nvmesensor application installed into /usr/bin on the target.

On the development machine, configure and build our modified application

1. devtool modify dbus-sensors
2. cd ~/src/openbmc/openbmc/build/p10bmc/workspace/sources/dbus-sensors
3. git remote add local ~/src/openbmc/dbus-sensors
4. git fetch local my-hacks && git merge FETCH_HEAD
5. devtool build dbus-sensors

On the target machine, prepare to deploy the modified application

1. overlay add /usr/bin /lib/systemd/system ...

For systems with a read-only rootfs, this makes the directory arguments writable by overlaying them with a tmpfs. The overlay script handles the grunt work of setting up the necessary tmpfs mountpoints, whose location we don’t really care about in the general case.

The overlay script lives here and should just be copied onto the target using scp.

From the development machine, deploy the modified application to the target

1. devtool deploy-target --no-host-check --show-status --no-check-space --strip dbus-sensors

Note this may require a rather hacky patch to devtool

On the target machine, test the modified application and discover issues

1. Test /usr/bin/nvmesensor
2. /usr/bin/nvmesensor generates a core-dump

On the target machine, extract the application core dump

1. coredumpctl dump $PID -o /tmp/core

On the development machine, prepare the bitbake environment for core analysis

1. bitbake -c do_rootfs obmc-phosphor-image

devtool build doesn’t run the step of packaging up the application for opkg to deploy and updating the package index, so we have to explicitly invoke this step ourselves.

On the development machine, analyse the core dump

1. scp $BMC:/tmp/core .
2. bbdbg ~/src/openbmc/openbmc/build/p10bmc /usr/bin/nvmesensor core dbus-sensors dbus-sensors-dbg

Using bbdbg we create ourselves a gdb session where we can perform sensible analysis of the nvmesensor core with all the necessary symbols loaded in. The easiest way to access bbdbg is to clone the openbmc-tools repository onto your development machine.

Interactive debugging with gdbserver and bbdbg

If we return to the step 16 where we’re testing nvmesensor on the target, we can also do interactive debug rather than core debug if we use gdbserver on the target.

On the target machine, attach gdbserver to a running application

1. gdbserver --attach localhost:1234 $PID

Note that gdbserver typically ignores the host portion of host:port, so we just pick localhost here.

On the development machine, use bbdbg to attach to the remote gdbserver

1. bbdbg ~/src/openbmc/openbmc/build/p10bmc /usr/bin/nvmesensor - dbus-sensors dbus-sensors-dbg
2. (gdb) target remote $BMC:1234