forked from Qortal/Brooklyn
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
214 lines
8.1 KiB
214 lines
8.1 KiB
Virtio-GPU Venus |
|
================ |
|
|
|
Venus is a Virtio-GPU protocol for Vulkan command serialization. The protocol |
|
definition and codegen are hosted at `venus-protocol |
|
<https://gitlab.freedesktop.org/olv/venus-protocol>`__. The renderer is |
|
hosted at `virglrenderer |
|
<https://gitlab.freedesktop.org/virgl/virglrenderer>`__. |
|
|
|
The protocol is still under development. This driver and the renderer are |
|
both considered experimental. |
|
|
|
Requirements |
|
------------ |
|
|
|
The Venus renderer requires |
|
|
|
- Vulkan 1.1 |
|
- ``VK_EXT_external_memory_dma_buf`` |
|
- ``VK_EXT_image_drm_format_modifier`` |
|
- ``VK_EXT_queue_family_foreign`` |
|
|
|
from the host driver. However, it violates the spec in some places currently |
|
and also relies on implementation-defined behaviors in others. It is not |
|
expected to work on all drivers meeting the requirements. It has only been |
|
tested with |
|
|
|
- ANV 21.1 or later |
|
- RADV 21.1 or later (the host kernel must have |
|
``CONFIG_TRANSPARENT_HUGEPAGE`` disabled because of this `KVM issue |
|
<https://github.com/google/security-research/security/advisories/GHSA-7wq5-phmq-m584>`__) |
|
|
|
The Venus driver requires supports for |
|
|
|
- ``VIRTGPU_PARAM_RESOURCE_BLOB`` |
|
- ``VIRTGPU_PARAM_HOST_VISIBLE`` |
|
- ``VIRTGPU_PARAM_CROSS_DEVICE`` |
|
- ``VIRTGPU_PARAM_CONTEXT_INIT`` |
|
|
|
from the virtio-gpu kernel driver, unless vtest is used. Currently, this |
|
means the `context-init |
|
<https://gitlab.freedesktop.org/virgl/drm-misc-next/-/tree/context-init>`__ |
|
kernel branch paired with `crosvm |
|
<https://chromium.googlesource.com/chromiumos/platform/crosvm>`__. |
|
|
|
vtest |
|
----- |
|
|
|
The simplest way to test Venus is to use virglrenderer's vtest server. To |
|
build virglrenderer with Venus support and to start the vtest server, |
|
|
|
.. code-block:: console |
|
|
|
$ git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git |
|
$ cd virglrenderer |
|
$ meson out -Dvenus-experimental=true |
|
$ ninja -C out |
|
$ ./out/vtest/virgl_test_server --venus |
|
|
|
In another shell, |
|
|
|
.. code-block:: console |
|
|
|
$ export VK_ICD_FILENAMES=<path-to-virtio_icd.x86_64.json> |
|
$ export VN_DEBUG=vtest |
|
$ vulkaninfo |
|
$ vkcube |
|
|
|
If the host driver of the system is not new enough, it is a good idea to build |
|
the host driver as well when building the Venus driver. Just remember to set |
|
:envvar:`VK_ICD_FILENAMES` when starting the vtest server so that the vtest |
|
server finds the locally built host driver. |
|
|
|
Virtio-GPU |
|
---------- |
|
|
|
Because the driver requires ``VIRTGPU_PARAM_CONTEXT_INIT`` from the virtio-gpu |
|
kernel driver, one must make sure the guest kernel includes the changes from |
|
the `context-init |
|
<https://gitlab.freedesktop.org/virgl/drm-misc-next/-/tree/context-init>`__ |
|
branch. |
|
|
|
To build crosvm, |
|
|
|
.. code-block:: console |
|
|
|
$ mkdir crosvm |
|
$ cd crosvm |
|
$ wget https://storage.googleapis.com/git-repo-downloads/repo |
|
$ chmod +x repo |
|
$ ./repo init -g crosvm -u https://chromium.googlesource.com/chromiumos/manifest.git |
|
$ ./repo sync |
|
$ cd src/platform/crosvm |
|
$ RUSTFLAGS="-L<path-to-virglrenderer>/out/src" cargo build \ |
|
--features "x virgl_renderer virgl_renderer_next default-no-sandbox" |
|
|
|
Note that crosvm must be built with ``default-no-sandbox`` or started with |
|
``--disable-sandbox`` in this setup. |
|
|
|
This is how one might want to start crosvm |
|
|
|
.. code-block:: console |
|
|
|
$ sudo LD_LIBRARY_PATH=<...> VK_ICD_FILENAMES=<...> ./target/debug/crosvm run \ |
|
--gpu vulkan=true \ |
|
--display-window-keyboard \ |
|
--display-window-mouse \ |
|
--host_ip 192.168.0.1 \ |
|
--netmask 255.255.255.0 \ |
|
--mac 12:34:56:78:9a:bc \ |
|
--rwdisk disk.qcow2 \ |
|
-p root=/dev/vda1 \ |
|
<path-to-bzImage> |
|
|
|
assuming a working system is installed to partition 1 of ``disk.qcow2``. |
|
``sudo`` or ``CAP_NET_ADMIN`` is needed to set up the TAP network device. |
|
|
|
Virtio-GPU and Virtio-WL |
|
------------------------ |
|
|
|
In this setup, the guest userspace uses Xwayland and a special Wayland |
|
compositor to connect guest X11/Wayland clients to the host Wayland |
|
compositor, using Virtio-WL as the transport. This setup is more tedious, but |
|
that should hopefully change over time. |
|
|
|
For now, the guest kernel must be built from the ``chromeos-5.10`` branch of |
|
the `Chrome OS kernel |
|
<https://chromium.googlesource.com/chromiumos/third_party/kernel>`__. crosvm |
|
should also be built with ``wl-dmabuf`` feature rather than ``x`` feature. |
|
|
|
To build minigbm and to enable minigbm support in virglrenderer, |
|
|
|
.. code-block:: console |
|
|
|
$ git clone https://chromium.googlesource.com/chromiumos/platform/minigbm |
|
$ cd minigbm |
|
$ CFLAGS=-DDRV_<I915-or-your-driver> OUT=out DESTDIR=out/install make install |
|
$ cd ../virglrenderer |
|
$ meson configure out -Dminigbm_allocation=true |
|
$ ninja -C out |
|
|
|
Make sure a host Wayland compositor is running. Replace |
|
``--display-window-keyboard --display-window-mouse`` by |
|
``--wayland-sock=<path-to-wayland-socket>`` when starting crosvm. |
|
|
|
In the guest, build and start sommelier, the special Wayland compositor, |
|
|
|
.. code-block:: console |
|
|
|
$ git clone https://chromium.googlesource.com/chromiumos/platform2 |
|
$ cd platform2/vm_tools/sommelier |
|
$ meson out -Dxwayland_path=/usr/bin/Xwayland -Dxwayland_gl_driver_path=/usr/lib/dri |
|
$ ninja -C out |
|
$ sudo chmod 777 /dev/wl0 |
|
$ ./out/sommelier -X --glamor |
|
--xwayland-gl-driver-path=<path-to-locally-built-gl-driver> \ |
|
sleep infinity |
|
|
|
sommelier requires ``xdg-shell-unstable-v6`` rather than the stable |
|
``xdg-shell`` from the host compositor. One must make sure the host |
|
compositor still supports the older extension. |
|
|
|
Optional Requirements |
|
--------------------- |
|
|
|
When virglrenderer is built with ``-Dminigbm_allocation=true``, the Venus |
|
renderer might need to import GBM BOs. The imports will fail unless the host |
|
driver supports the formats, especially multi-planar ones, and the DRM format |
|
modifiers of the GBM BOs. |
|
|
|
In the future, if virglrenderer's ``virgl_renderer_export_fence`` is |
|
supported, the Venus renderer will require ``VK_KHR_external_fence_fd`` with |
|
``VK_EXTERNAL_FENCE_HANDLE_TYPE_SYNC_FD_BIT`` from the host driver. |
|
|
|
A WSI image of the Venus driver is an external image to the host driver. When |
|
the WSI image is transitioned from ``VK_IMAGE_LAYOUT_UNDEFINED`` after image |
|
acquisition, the Venus driver does not request the Venus renderer to perform |
|
an ownership transfer on the external image. It is unclear if the ownership |
|
transfer is required or not. A specification issue has been filed for |
|
clarifications. See the comment before ``vn_cmd_fix_image_memory_barrier`` |
|
for more details. |
|
|
|
VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT |
|
----------------------------------- |
|
|
|
The Venus renderer makes assumptions about ``VkDeviceMemory`` that has |
|
``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT``. The assumptions are illegal and rely |
|
on the current behaviors of the host drivers. It should be possible to remove |
|
some of the assumptions and incrementally improve compatibilities with more |
|
host drivers by imposing platform-specific requirements. But the long-term |
|
plan is to create a new Vulkan extension for the host drivers to address this |
|
specific use case. |
|
|
|
The Venus renderer assumes a device memory that has |
|
``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`` can be exported as a mmapable dma-buf |
|
(in the future, the plan is to export the device memory as an opaque fd). It |
|
chains ``VkExportMemoryAllocateInfo`` to ``VkMemoryAllocateInfo`` without |
|
checking if the host driver can export the device memory. |
|
|
|
The dma-buf is mapped (in the future, the plan is to import the opaque fd and |
|
call ``vkMapMemory``) but the mapping is not accessed. Instead, the mapping |
|
is passed to ``KVM_SET_USER_MEMORY_REGION``. The hypervisor, host KVM, and |
|
the guest kernel work together to set up a write-back or write-combined guest |
|
mapping (see ``virtio_gpu_vram_mmap`` of the virtio-gpu kernel driver). CPU |
|
accesses to the device memory are via the guest mapping, and are assumed to be |
|
coherent when the device memory also has |
|
``VK_MEMORY_PROPERTY_HOST_COHERENT_BIT``. |
|
|
|
When a ``VkImage`` or a ``VkBuffer`` is created, the Venus renderer does not |
|
know if the image or the buffer will be bound to such a device memory or not. |
|
As a result, the Venus renderer unconditionally chains |
|
``VkExternalMemoryImageCreateInfo`` to ``VkImageCreateInfo`` and chains |
|
``VkExternalMemoryBufferCreateInfo`` to ``VkBufferCreateInfo`` without |
|
checking for the host driver support.
|
|
|