The word on the street is DevOps. There’s technology out there to make both developers’ and sysadmins’ lives easier and more intertwined, such as:

• Containerization: packaging software in a lightweight virtualization container with a low overhead, with bonuses in maintainability and portability, and
• Large-scale cluster management: configuration and management of thousands of servers using easy-to-use tooling and scripting,

but it’s currently exclusively focused on the let’s-call-it-the-cloud domain. So we – a very small team of network engineers, information security experts and mobile platform wizards – secured some funding from the Dutch Ministry of Defence and very limited time to do a small-scale project to see if aforementioned technologies work in the mobile domain.

Mobile is a sysadmin’s nightmare

So, you’re a sysadmin and charged with keeping stuff running in an organization which increasingly relies on mobile devices to get information to its employees and vice versa. Particularly, these mobile devices are typically meant to be used in extreme environments and are typically suited towards one specific use case. These devices are closed-off and proprietary, each with their own management solution – if you’re lucky. Yes, typical safe stovepipe solutions, forcing your employees to lug around multiple devices with all the typical implications for training and support. And often, configuring devices means getting your boots muddy and manually inserting a proprietary memory card in every single device.  Not to mention flying to all corners of the earth and getting into all kinds of unsafe situations. Rinse, repeat for thousands of devices and you can imagine there’s your worst nightmare.

Along came smartphones and tablets. These devices make a range of functionality and tailored information available on one relatively open hardware platform. Everyone knows apps and they are a part of our daily life. And to top things off, on the hardware side, things are getting interesting with modular smartphones and a lower barrier-to-entry on hardware design for mobile devices. So what if you would replace all those proprietary devices with smartphone-ish devices which are modular in both software and hardware? It certainly means more choice and better information for the person operating the phone, plus a generic platform that can be used for a variety of very specific use cases. So maybe cheaper and more effective. But seemingly an even larger nightmare from a manageability point of view.

DevOps to the rescue?

Are there parallels between managing thousands of servers and thousands of mobile devices? We believe there are and we set out to test a couple of DevOps technologies. Specifically, we wanted to see if both containerization and large-scale cluster management technology could be applied to configure and manage mobile devices in a way that beats today’s options. To do this, we needed a compelling use case.

We settled on bootstrapping, configuring and personalizing mobile devices during a typical briefing for a military patrol mission. Our line of thinking was if we could make this into a zero-effort process, both end users and sysadmins would benefit. Other use cases we considered were context-defined reconfiguration while on patrol (think the Cover launcher for Android on steroids), dynamically updating the whole network infrastructure during an extended mission and crypto key distribution to a large number of devices.

The proof of the pudding

Our proof-of-principle had a couple of elements. First, we chose a hardware platform. Did I mention smartphones are our target devices? Today’s smartphones seem open, but are hard to develop for if you don’t want to spend too much time fighting the mobile OS of your choice. And don’t forget poor or lacking developer documentation for hardware components. Smartphones are also not very extensible hardware-wise and we needed a plug-and-play hardware platform. So we chose the Raspberry Pi. Added bonus is that the Pi is so underpowered that a solution for the Pi could theoretically run on any smartphone hardware. We added a display module, a case, a  battery, a Bluetooth adapter and a WiFi adapter.

Now to the platform software stack. For the OS, we chose Arch Linux since it is generally more bleeding-edge than Raspbian and should support most of our other bleeding-edge software stack. As a container mechanism, we chose Docker. It uses Linux kernel functionality to application containers in an isolated environment, and Android does something quite similar for apps. And as a configuration mechanism, we went for Puppet. Both Docker and Puppet run on the Raspberry Pi with a little effort.

Finally, we chose our software components to run on the platform. Generally, in the Defense domain, the focus is on networking and information security. We chose to package an ad-hoc routing protocol (OLSRv2), an ad-hoc VPN solution (TINC) and an end-user application (a homebrew text-chat app working with the display module) in Docker containers. These choices made us cover all our bases in regard to which kind of software was expected to run in our target environment.

Will it blend?

Short answer: yes. We demonstrated three neat prototype devices in a cluster that could work through three stages of configuration:

1. Node initialization. This stage deploys system essentials on top of a minimal Arch Linux system images. It also configures an initial device identity.
2. Mission-specific configuration. This stage deploys mission-specific containers (ad-hoc networking, VPN and chat) to the Pi.
3. Personalization. Via an Android companion app and Bluetooth 4 LE, a person can log in to the device, effectively transferring his / her relevant crypto keys to the device and personalizing the device for his / her needs. Once the device is personalized, the ad-hoc network is started and the user can use the chat app to communicate with other devices.

We tied all technology together using custom shell scripts. We did some development towards using physical network interfaces in Docker, and we adapted the Docker plugin for Puppet to work on the Raspberry Pi. We also used the pipework and bleno helper libraries to pull physical network interfaces into Docker containers and make our life easier with Bluetooth.

Will it really blend?

Er, yes, but with some caveats. The process is fixed to our specific use case. Also, development took more time with an ever-changing software stuck; all mentioned projects are under heavy development. And initially running the demonstration takes more time than we had hoped for (30 minutes for stage 1, a couple of minutes for stage 2 and 3). However, there are some quick-win optimizations we have thought of but haven’t gotten around to doing just yet, so we can hopefully improve that soon.

In the future

We are really interested to see the concept of dynamic configuration of mobile devices brought further for real use cases in real operating environments. Our priority is a convincing demonstration at scale in a real military, first responder or other extreme situation. Technologically, there is still a lot of spit and polish to apply to our current implementation and we are looking towards using all kind of crazy mechanisms in dynamic configuration, including distributed Docker repositories and Puppet masters. We want context and sensor information so that we can manage stuff way more tailored than we can at the moment. And we want a modular smartphone to try it on.

We’re happy write more in-depth posts on parts of the project that may be of interest to others, so please leave some comments towards what part you’re interested in. Our next post will probably be on using physical network interfaces in Docker containers.