.. include:: stylesheets/article-html.txt

================================================
Extending QEMU User-Mode with PRoot: Why and How
================================================

:Original Title:  PRoot: a Step Forward for QEMU User-Mode
:Published at:    1st International QEMU Users' Forum
:Authors:         Cédric Vincent & Yves Janin
:PRoot Version:   0.5

:Abstract:

    This paper introduces PRoot, a new tool specifically designed to
    extend QEMU user-mode.  We detail our motivations, compare with
    other similar solutions and give first results.

Extending QEMU User-Mode
========================

QEMU [C1]_ is an open-source hardware emulator with two main usages:
in *system-mode* a whole guest machine is emulated to run a full
operating system.  It can benefit from virtualization support like KVM
to avoid the emulation of most CPU operations.  By contrast, QEMU
*user-mode* only emulates the CPU to run one guest process at a time,
and communications with the host kernel are converted by a thin layer.

Our initial motivation was to use QEMU to ease the development of
embedded Linux applications by emulating their build system and
test-suites directly on developers' workstations.  During our
experiments we found out that the system-mode was not fast enough and
we decided to focus on the user-mode.  Unfortunately this latter
cannot execute a tree of guest processes within a guest Linux
distribution for reasons we detail in the next section.  To get the
best of these two modes we developed PRoot, a tool fully compatible
with QEMU that can be seen as an *extension* of the user-mode.


Overcoming QEMU user-mode limitations with PRoot
------------------------------------------------

As illustrated by Figure 1 (a) and (b), QEMU should run faster for a
given process in user-mode than in system-mode since it does not
emulate the peripherals nor the guest kernel.  However the boundary
between the guest environment and the host becomes *thinner*, and the
host kernel now has to support some of the operations that were
devoted to the guest kernel.  We can define three requirements that
will guide us throughout the rest of this paper:

R1
    an efficient and correct path translation mechanism between the
    guest and the host.  This feature is typically required to
    emulate programs that perform absolute accesses to architecture
    specific files such as the dynamic linker ``/lib/ld-linux.so``.
    Note that a correct translation scheme must support relative paths
    and symbolic links too.

R2
    the ability to spawn a new process and keep emulation running for
    this new process.

R3
    the ability to run a host process program.  Strictly speaking R3
    is not mandatory, but using *un-emulated* cross-compilers and
    interpreters is particularly useful to speed-up build process and
    test-suites.

QEMU user-mode status for R1 and R2 is currently not satisfying.  For
R1, QEMU user-mode has the ability to redirect file operations made by
the guest process but it can take several minutes at each process
creation to scan the guest Linux distribution.  Moreover there is no
support for relative paths and symbolic links.

Let us now consider R2.  From the point-of-view of the host kernel, an
emulated process image P1 is actually an image of QEMU |Q1P1|.  When
|Q1P1| executes a new program, the resultant process P2 is totally out
of the control of QEMU.  As a conclusion R2 is not supported either.

.. |Q1P1| replace:: Q1\ :sub:`P1`

.. figure:: extending_qemu-fig1.svg

   Figure 1: Guest process[es] running under QEMU, arrows are system calls flow

The ambition of PRoot is to fulfill R1, R2 and later on R3, while
keeping design, implementation and use as simple as possible.  Since
PRoot is also used for validation purpose, it has to reproduce kernel
results and errors as if the emulated applications were run in a
native environment.

PRoot is based on ``ptrace``, a Linux system call [C2]_ that is mainly
used by debuggers or system call tracers such as ``strace`` and the
GNU Debugger ``gdb``.  However since it provides unique facility to
monitor and control processes, it can also be applied to a much wider
range of applications bringing kernel-like features in user-space
[C3]_ [C4]_.  With ``ptrace``, PRoot is able to catch every system
call emitted by a QEMUlated process and dynamically redirect them as
shown in Figure 2.  One of the main task in PRoot is the path
canonicalization that translates paths from the guest world to the
host world and vice versa (R1).  Additionally, when PRoot detects a
new process about to be created, it can change the initial request and
get a new instance of QEMU running the new process (R2) as shown in
Figure 1 (c).

.. figure:: extending_qemu-fig2.svg

   Figure 2: Control flow of a syscall done by a guest process under QEMU+PRoot


Results
=======

Experiments presented in this section were performed on a Pentium-4
2.4GHz workstation with 1GiB of RAM running Slackware-13.1.
Concerning the guest system, we used ARMedSlack-12.2 with QEMU-0.12.5
configured to provide 256MiB of RAM when used in system-mode.


Functional Comparison With Similar Tools
----------------------------------------

Table 1 exemplifies two Linux packages that are representative of the
complexity of an embedded Linux distribution; Perl-5 requires a guest
environment for its bootstrap whereas the Coreutils test-suite
exhibits numerous limit configurations that can help to find
unexpected behaviors or bugs.

Many path translation tools already exist but few pass these tests
(R1).  Fakechroot-2.14 and Fakeroot-Ng-0.17 fail.  Plash is a modified
GNU C library and was therefore not considered, and we did not
consider ``chroot`` either since it requires administration
privileges.  Generally speaking, all these tools were not designed to
support the execution of processes through an emulator (R2).

Only PRoot and ScratchBox2 [C5]_ proved complete enough to configure,
build and validate the two complex Linux packages.  Table 1 gives the
results of their test-suites run with PRoot and ScratchBox2, it also
gives system-mode results as a reference.

.. table:: Table 1: Results of the packages' test-suites

    ====================  ===============  ==========  ===========
    Package                ScratchBox-2.2   PRoot-0.5  system-mode
    ====================  ===============  ==========  ===========
    Perl-5.10.0                     99.6%       99.6%        99.8%
    GNU Coreutils-6.12              94.9%       97.3%        96.7%
    ====================  ===============  ==========  ===========

ScratchBox2 and PRoot have similar path canonicalization algorithms
but differ in their design [#]_ and usage.  Unlike ScratchBox2, PRoot
does not need any configuration and does not assume anything about the
guest and host environments.  By contrast, ScratchBox2 is a highly
flexible tool with scripting facility.  A complete technical
comparison between ScratchBox2 and PRoot is beyond the scope of this
paper, but we can safely conclude that both are functional enough to
be already used in an industrial environment.  Choosing one instead
another one depends on your expectations: simplicity with PRoot vs
powerful customization with ScratchBox2.

.. [#] ScratchBox2 relies on the dynamic linker to override system
       call wrappers in the C library.


Performance Comparison With QEMU System-Mode
--------------------------------------------

Table 2 compares performance of QEMU-0.12.5, first in user-mode with
PRoot-0.5 and then in system-mode.  Timings were measured with the
command ``time`` and we checked that the system-mode results were
coherent with the host clock.

.. table:: Table 2: Performance of PRoot+QEMU user-mode vs QEMU system-mode

    ==================  ===============  ===============
    stage                   Perl-5.10.0   Coreutils-6.12
    ==================  ===============  ===============
    archive extraction      3.6x faster      2.7x faster
    configuration           2.0x faster      4.0x faster
    build                   2.9x faster      3.5x faster
    validation              4.1x faster      3.6x faster
    ==================  ===============  ===============

The speed-up with QEMU user-mode is quite impressive and comes from
the fact that the guest kernel and the hardware are not emulated.  We
think that all these results will be further improved when PRoot
handles *un-emulated* cross-compilers and interpreters (R3).  This
feature is still under development and looks promising.


Conclusion
==========

The simple fact that PRoot requires no privilege or setup is a great
advantage that should ease the usage of this *extended* user-mode.  It
is worth mentioning that PRoot should work with any version of QEMU
user-mode since no patch is needed, and in any Linux environment as
well.  Finally, the current version of PRoot is mature enough to be
already used in an industrial environment.


References
==========

.. [C1] Bellard, F.: QEMU, a Fast and Portable Dynamic Translator.
        USENIX Annual Technical Conference, FREENIX Track (2005)
        41--46

.. [C2] Kerrisk, M.: The Linux Programming Interface.  No Starch Press
        (2010) 23, 43--46, 367--370

.. [C3] Spillane, R. P., Wright, C. P., Sivathanu, G., Zadok, E.: Rapid
        File System Development Using ptrace.  Proceedings of the 2007
        Workshop on Experimental Computer Science (ExpCS'07), 22. ACM
        (2007)

.. [C4] Dike, J.: User-mode Linux.  Proceedings of the 5th Annual Linux
        Showcase & Conference - Volume 5 (ALS'01). USENIX Association
        (2001)

.. [C5] ScratchBox2. http://www.freedesktop.org/wiki/Software/sbox2
