Lately I’ve been looking at some tools to make it easier to package containerized applications as lightweight virtual machines, to get the best of both worlds: the developer experience of a Dockerfile with the added protection of a hypervisor.

Using Linuxkit Generated Iso In Docker Command

As part of this process, I’ve been digging into Juniper’s containerized routing stack called cRPD, and trying to get that into a virtual form factor, so that I can effectively have a Linux router that happens to use the Junos routing stack. I’ve worked out an approach for doing this that could, in theory, extend to all kinds of other software, including other disaggregated routing stacks, such as Free-Range Routing.

Once you have built the tool, use. Linuxkit build linuxkit.yml to build the example configuration. You can also specify different output formats, eg linuxkit build -format raw-bios linuxkit.yml to output a raw BIOS bootable disk image, or linuxkit build -format iso-efi linuxkit.yml to output an EFI bootable ISO image. Using LinuxKit generated ISO in Docker For Mac #2580. Closed Copy link Collaborator docker-desktop-robot commented May 9, 2018. Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity. Using Linuxkit Generated Iso In Docker For Mac Linuxkit is a new project presented by Docker during the DockerCon 2017. If we look at the description of the project on: A secure, portable and lean operating system built for containers I am feeling already exited.

NOTE - you need access to download the cRPD software from Juniper if you wish to follow the instructions in this post, and at the time of this writing, there is no free trial for cRPD. However, a lot of us at Juniper are actively working on getting cRPD into your hands more easily, so stay tuned!

This post is written for folks like me who are looking to define their own modular, and automated OS build. As a result, this requires some advance knowledge of concepts pertaining to Linux and Docker. You’ll need a few things set up in advance:

• Docker
• A linux-based host OS (I’m using Ubuntu)
• A hypervisor (these instructions are for QEMU)
• Git

What is cRPD? What is LinuxKit?

cRPD takes some of the best parts of Junos (RPD, MGD), disaggregates and packages them in a lightweight container image.

Because it’s so stripped down, it is not designed/able to function as a full-blown operating system, or manage network interfaces, as we’re accustomed to being able to do with a typical Junos device - it’s just a routing stack with a management process. This means that in order to turn it into a real routing element, we need to run it on an existing OS, like Linux.

Linuxkit is “a toolkit for building custom minimal, immutable Linux distributions”. It allows you to build your own lightweight Linux distribution using containers. It uses a YAML-based specification where you can specify the components that go into your distribution, like the kernel, init systems, and the userspace processes you wish to deploy inside - all running as containers, but with the added layer of true virtual isolation.

For our purposes, this is a match made in heaven - pairing the powerful, mature routing stack and management daemon/CLI of cRPD with the automated, modular, and lightweight Linux distribution built with LinuxKit, we get the best of both worlds.

Building Our Custom cRPD Image

Before we get to building our own Linux distribution, we need to make some modifications to the Docker image we get with cRPD. Out of the box, cRPD is designed to be minimal, meaning there are a number of options that are not enabled or configured by default. To get things like SSH and NETCONF working, we’ll actually build our own custom Docker image that takes the base image we’ll download from the Juniper website, and adds the relevant configurations to make it useful for our purposes.

I created the crpd-linuxkit repo with all of the files we’ll be using forthis build, so start off by cloning that repository:

Next, follow the cRPD installation instructions (and specifically the section titled Download the cRPD Software ) to download a .tgz file containing the crpd image. This is a tarball that Docker will recognize and allow you to import into your local Docker image cache. Continue only once you see your image in the output of docker image ls | grep crpd.

We’re going to build a custom Docker image that uses this cRPD image as its base, so we can add things like a proper SSH configuration. This Dockerfile uses crpd:latest as its base image reference, which doesn’t yet exist. The following command will look up the ID of the image we just imported, and re-tag it as crpd:latest so we can use it in our custom build:

If you take a peek at the files in the repository, you’ll notice with have a few things that will go into our customDocker image:

• Dockerfile - copies our modified configuration files into the container image, and also sets up authentication options
• launch.sh - a launch script which is executed when the container starts, to copy our configuration files into the correct location, and also start relevant services like the ssh service.
• sshd_config - this is our modified SSH configuration, which includes a very important step of pointing the NETCONF subsystem to the appropriate path. This way, NETCONF requests will go straight to cRPD.

This walkthrough won’t go into much more detail on the files here, as well as the other files in the repository, so if you see something you don’t understand, cat its contents and read it for youself! I added comments where I could.

These are all tied together with the Dockerfile, so you should be able to run the below to build everything:

Using Linuxkit Generated Iso In Docker Container

To be clear, our ultimate goal is to run cRPD inside a virtual machine that we create with LinuxKit, but why don’t we take a second to marvel at the fact that we now have a container image that runs the Junos routing stack! Let’s start an instance of it to play around with real quick for funsies:

We can interactively enter the Junos CLI with:

This should start looking a lot more familiar:

However, it’s not without its oddities - for instance, there is no show interfaces terse command!

Keep in mind, this is just the Junos routing stack and management daemon. It has no control over the network interfaces themselves, that’s still firmly in control of the underlying operating system, which in this case is my laptop since I’m running this natively in docker.

We can view things like the routing status for network interfaces, as that’s relevant to what cRPD is designed to do:

This is just the beginning. If you’re new to cRPD and just want to play with it instead of setting it up, one of the reasons I’ve been working on this is building a reproducible image for the NRE Labs curriculum. Follow along there or on Twitter, as I’m hoping to be able to publish something like this in the next few months.

Let’s exit and delete our cRPD container. From now on, we’ll be running cRPD inside a LinuxKit virtual machine.

Building the Router VM Image with LinuxKit

Using Linuxkit Generated Iso In Docker Windows 10

Okay, so we have our cRPD container image, but again, it’s not designed to function as a full-blown operating system. To actually pass traffic, we will use this container image as an “app” to be deployed in a brand-new Linux distribution that LinuxKit will create for us. The end-result is that we have a Linux-based VM that runs Junos software.

At the time of this writing, LinuxKit’s latest release is v0.7. Since I’m running on Linux, I’ll grab the precompiled binary for my platform, and pop it into /usr/local/bin:

LinuxKit comes with a linuxkit run command, but I already have existing scripts for running VMs the way I want, soI just want a virtual hard drive image to be generated that I can just execute myself.

Take a look at the contents of crpd.yml, as the entire manifest for defining this virtual machine, including picking a Linux kernel version, even including an init system, and finally, running our cRPD image, is all defined there. We can feed this filename into the linuxkit build command to build our VM.

Using Linuxkit Generated Iso In Docker

LinuxKit is able to package to a variety of form factors. I’ll be running my VMs with QEMU, so packaging as qcow2-bios is most appropriate. This is possible with the -format flag:

Note that the above requires KVM extensions to be available and enabled, and your user to be added to the kvm group. If you run into problems here, check that KVM virtualization support is configured properly.

This may take a little time, as the linuxkit tool needs to download relevant images referenced in the build manifest, and then assemble them into a working distribution. At the end, you should end up with a file called crpd.qcow2 in your current directory.

Running a Routed Topology

We now have a virtual disk image we can use to boot instances of our cRPD virtual machine. We’ll look to start two instances, crpd1 and crpd2, and connect them via their eth1 interfaces, both connected to a bridge on the host. We’ll consider success to be that we’ve formed an OSPF adjacency, and have learned the route to crpd1’s loopback interface, and can ping it from crpd2.

So, first we’ll need to prep our host network configuration by creating a bridge and adding tap interfaces. This will allow our VMs to communicate with each other. Run the following as root:

Next, we need to start the virtual machines. I provided a helper script you can run to copy our disk image once for each VM we want to start, as well as running the VMs with QEMU:

Note that this runs the VMs as detached screen sessions, so either make sure screen is installed, or you can run the commands in that script in separate terminal sessions/tabs.

Don’t forget to stop these VMs with ./stop-vms.sh when you’re done with this walkthrough!

Once the script returns, you can use telnet 127.0.0.1 5000 and telnet 127.0.0.1 5001 to connect to the serial port for crpd1 and crpd2 accordingly. Once you see the below prompt (If you don’t see it after a while, try hitting enter), the VM is booted, and cRPD should be running :

Using Linuxkit Generated Iso In Docker Linux

If you are still connected to the serial port, hit Ctrl+] and type quit to disconnect and return to the host shell.

At this point, we have two VMs that we’ll call crpd1 and crpd2 that are connected via their eth1 interfaces using a host bridge. So, the name of the game is to configure networking on these VMs, as well as routing within cRPD so we can have an OSPF adjacency.

First, we’ll access crpd1 by connecting via ssh with the command ssh [email protected] -p 2022 (password is Password1!):

Note that we still have a bash shell here, not the Junos CLI. This is because cRPD is built within a Ubuntu container image, so we still have familiar Linux primitives here. Better still, the container is running with NET_ADMIN permissions, so we can make network changes here, and it will apply to the VM as a whole.

Then, in another terminal session, run ssh [email protected] -p 2023 (password is Password1!) to access crpd2 and paste the following commands:

If you ran the above commands, you should still be in crpd2’s Junos CLI. You can verify that you have an established OSPF adjacency, and have learned the route to crpd1’s loopback address here:

Type exit to return to the bash shell, and run a ping to verify connectivity:

The Junos configuration we loaded into the Docker image early in this walkthrough was minimal. The point of this exercise was to develop an image configuration that could be programmed further via NETCONF at runtime.

So, for all the marbles, you can run these commands to install a Python virtualenv, install PyEZ (the Python library for Junos), and execute a python script that outputs the OSPF neighbors from crpd1. This ensures that we can communicate to the instance via NETCONF (be sure you’ve exited from your SSH session in previous steps - these are meant to be run back in the crpd-linuxkit repository you cloned earlier):

You should see this:

There are a number of projects that could be useful as alternatives - both in the Linux build portion, as well as the disaggregated software. This post is about proposing one possible path, not that this is the only one you could choose.

In addition to using something like Free-Range Routing as your routing stack, you don’t even have to run a routing stack at all. There are a bunch of use cases where the UX of a Dockerfile can/should be married with the isolation of a hypervisor. You also don’t have to use LinuxKit for the Linux build portion - there are a number of alternatives that could be used, each with its own pros/cons:

• Firecracker Ignite - This is a tool the folks at Weave built to add a GitOps workflow for building Firecracker images. I looked at this briefly but it seems to be fairly opinionated about how it runs things. It’s simpler than linuxkit, but it seems that I don’t have the ability to export a VM image that I can then run myself in a traditional hypervisor. This tight coupling doesn’t really give me what I want today. YMMV.
• RancherOS - This seems to be much closer to the traditional Docker toolchain, and could very well work for this.
• CoreOS - Also might work, though not sure of the state of this post RH acquisition. May also work.

After I published this blog post, a few folks gave me a few more to keep an eye on:

• Vorteil (Don’t ask me how to pronounce it)
• BottleRocketOS - Seems to be made by the AWS folks, so I have to assume it has something to do with Firecracker, but not sure how. Seems cool, and looks like it’s written in Rust, which is cooler.

I hope this was helpful to someone that might be thinking of doing the same things I am. For a long time, I’ve viewed the UX of a Dockerfile as something I could only get if I was willing to part with the security and isolation of virtual machines. I love that there are a number of ways in 2020 that I can bypass that tradeoff entirely.