DEV Community

Cover image for Bringing VNC Support to VirtualBox for Windows
Stefor07
Stefor07

Posted on Edited on

Bringing VNC Support to VirtualBox for Windows

Introduction

VirtualBox is one of the most widely used virtualization platforms, with support for multiple host operating systems, including Windows and Linux. There is, however, a notable difference between the two: VNC support is available out of the box on Linux, while Windows users have no equivalent option.

Rather than relying on third-party remote desktop software or the standard VRDP implementation, I decided to build a custom Windows version of VirtualBox with native VNC support. What appeared to be a straightforward task turned out to be a considerably more demanding project, requiring approximately two and a half days of work.

Why VNC?

VNC is a widely adopted remote desktop protocol, supported by many clients such as TigerVNC, RealVNC and TightVNC. My goal was to replicate the Linux experience on Windows: connecting directly to a virtual machine with a standard VNC client, without installing additional desktop-sharing software on the host.

VirtualBox already offers a modular remote desktop architecture through VRDE, and its source code includes a VNC implementation. The challenge was making it work reliably on Windows.

The Build Environment

Compiling VirtualBox on Windows requires an extensive toolchain. The environment I prepared included:

  • Microsoft Visual Studio 2022 Build Tools
  • Git
  • Windows 10 and Windows 11 SDKs
  • Windows Driver Kit (WDK)
  • Yasm and NASM
  • JDK 17
  • Open Watcom
  • Qt 6.12.0
  • WiX Toolset
  • Flex and Bison

In addition, I built libvncserver separately using MSVC and integrated the resulting library into the VirtualBox source tree.

The full list of tools, versions and configuration files is documented in the project wiki.

Why Building the VNC Extension Pack Alone Was Not Enough

A fundamental obstacle was the fact that simply compiling the VNC Extension Pack was not enough. The official VirtualBox distribution provided by Oracle does not load custom extension packs that have been compiled independently and lack a digital signature. Consequently, it was not possible to create the VNC component and install it alongside the official version.

Instead, it was necessary to recompile VirtualBox itself, integrating VNC support directly into the build. This required adjusting the build configuration and making several modifications to the KMK override files. The compilation process did not succeed on the first attempt; various issues had to be identified and resolved over the course of several iterations.

Specifically, an official VirtualBox release, such as the one distributed by Oracle, is not compiled with the following flag:

VBOX_WITHOUT_HARDENING = 1
Enter fullscreen mode Exit fullscreen mode

This means that the official build runs with Windows hardening enabled. In this mode, VirtualBox verifies the integrity and digital signature of the executables and modules loaded by its processes, and refuses to load anything that does not meet these requirements. An independently compiled Extension Pack, which is not signed by Oracle, therefore cannot be loaded.

Building VirtualBox with this flag set disables these checks, so that custom, unsigned components can be loaded. This is the reason why a full rebuild was unavoidable. It is also worth noting that the flag only affects user-mode binaries: it does not remove the requirement for signed kernel-mode drivers, which is addressed separately in the section on driver signing.

Packaging the Installer

Once compilation succeeded, I generated a Windows MSI installer with the WiX Toolset. I intentionally excluded the following components, which were not needed for my use case:

  • VirtualBox Web Service
  • Guest Additions ISO

Rebuilding the Guest Additions ISO was unnecessary, as the official ISO distributed by Oracle is fully suitable for this setup. The outcome was a custom MSI installer containing my modified VirtualBox build.

Tip

The ISO files for the Guest Additions are not included in the source code or in the precompiled versions; however, they can be downloaded via VirtualBox itself when you choose to install the VirtualBox Guest Additions from within a VM.

Driver Signing

Driver signing was another significant challenge. VirtualBox depends on kernel-mode drivers, and my custom build produces drivers that are not signed with Oracle's certificates. To load them on my Windows host, I used my own test-signing certificate, which involved:

  • Signing the generated drivers with the custom certificate
  • Installing the corresponding public certificate in the appropriate Windows certificate store
  • Enabling Windows Test Signing Mode

This is an important limitation of the current build. Windows Test Signing Mode must remain enabled for these test-signed drivers to load. Installing the certificate alone is not enough to satisfy Windows' standard driver-signing policies.

Anyone who wishes to use the precompiled version will therefore need the MSI installer, the public certificate, and the appropriate test-signing configuration. The private signing key will not be distributed. The installation steps are described in the repository.

Results

After approximately two and a half days of troubleshooting, configuration changes and rebuilds, the project achieved its goal. VNC works correctly on my custom Windows build of VirtualBox. I can connect to a running virtual machine with a standard VNC client by specifying the host address, port and password configured through VBoxManage. No external VNC server is required on the Windows desktop.

The result is native VNC access to VirtualBox virtual machines through the VRDE architecture, which is exactly what I set out to build.

The GitHub Repository

GitHub logo St3vSoft / VirtualBox-VNC-Windows

Oracle VirtualBox for Windows with VNC Support

VirtualBox VNC for Windows

A custom Windows build of Oracle VirtualBox with integrated VNC support.

Overview

VirtualBox VNC for Windows is a community-maintained custom build of Oracle VirtualBox that enables VNC support through the VirtualBox Remote Desktop Extension (VRDE).

While VNC support is available out of the box on Linux, Windows users do not have an equivalent ready-to-use binary distribution.

This project aims to bridge that gap by integrating the existing VNC implementation into a custom Windows build of VirtualBox, using libvncserver.

The goal is to allow users to connect directly to their virtual machines using standard VNC clients, such as TigerVNC, RealVNC and TightVNC, without requiring an external VNC server running on the Windows host.

Features

  • Integrated VRDE VNC support for Windows.
  • Based on the original Oracle VirtualBox source code.
  • Uses libvncserver for VNC functionality.
  • Custom Windows MSI installer.
  • Compatible with standard VNC clients.
  • Guest Additions ISO can…

It contains the VirtualBox source tree with the VNC modifications already applied and libvncserver precompiled, so it can be built without any changes. The wiki provides a step-by-step guide covering:

  • setting up the build environment
  • example build configurations
  • building libvncserver
  • building VirtualBox from source
  • known issues and workarounds

There are three ways to get the source code: clone the official VirtualBox repository, use the repository above, or download the source archive from the official VirtualBox website. The last two require you to build libvncserver and apply the modifications manually, as described in the wiki.

The aim is not to replace the official Oracle distribution, but to offer a community-built alternative for Windows users who need VNC support.

Conclusion

This project proved considerably more complex than anticipated. The VNC integration itself was only one part of the work: preparing the Windows build environment, integrating libvncserver, modifying the KMK configuration, producing the MSI installer and handling kernel driver signing were all essential steps.

I am pleased with the outcome. When a feature is missing on your platform, sometimes the most effective solution is to build it yourself.

Feedback, issues and pull requests are welcome on the GitHub repository.

Top comments (0)