An Open Virtualization Framework for Heterogeneous SoC-FPGA Platforms

Abstract

Heterogeneous SoC-FPGA platforms are suitable for a wide range of applications, from automotive systems to industrial automation, as they offer a unique combination of flexibility, reconfigurability, and high-performance computing capabilities. Moreover, due to the escalating demand for higher performance in computing systems and the need for efficient of shared resources, the adoption of Virtual Machines (VMs) on SoC-FPGAs is becoming increasingly prevalent. Although the VMs are isolated for security and fault containment, communication between them is necessary to fully utilize the system's potential and resources, especially when dealing with heterogeneous VMs. Enabling efficient communication between different processing elements in heterogeneous VMs is one of the main challenges that hinders the full potential utilization of SoC-FPGA devices. The current communication mechanisms to transfer data between different processing elements in heterogeneous SoC-FPGA platforms result in significant data transfer overhead and huge latencies. On the other hand, deploying cache coherency in a heterogeneous system presents unique difficulties due to distinct architectures, memory hierarchies, and communication protocols. Our research introduces an open and full-system stack virtualization framework designed to address these challenges and a novel approach to enable hardware cache coherent communication between heterogeneous VMs on SoC-FPGA devices. The framework supports a number of software and hardware components out of the box. The framework, being open-source, facilitates collaboration and innovation in the field of embedded systems and virtualization. It enables the seamless virtualization of heterogeneous components, including CPUs and FPGAs, within SoC-FPGA platforms, providing a foundation for resource isolation, efficient management, and enhanced communication among virtualized entities.

The framework offers a ready-to-deploy specialized hardware components for cache management, firmware, hypervisor, and Operating System (OS) support, to establish seamless coherence across heterogeneous VMs. To demonstrate the effectiveness of the plug-and-play features within the proposed framework, we support three hypervisors on the Processing System (PS) and one hypervisor on the FPGA, along with two OSes and bare-metal application support. Furthermore, we offer two distinct hardware support on the FPGA. These components can be combined to create a full system capable of generating a multitude of configurations. This flexibility allows researchers to utilize and expand the framework for comprehensive evaluations of their proposals, encompassing a wide range of hardware, hypervisors, and operating systems.

We evaluate the proposed coherent communication mechanism deployed on top of the proposed framework with different scenarios to measure the latencies of three ports: the Advanced Extensible Interface (AXI) Coherency Extensions (ACE), which is a two-way hardware coherency port; the High-Performance Coherency (HPC), which is a one-way hardware coherency port; and the High-Performance (HP), which is a non-coherent port. We then execute multiple benchmarks to assess the effectiveness and efficiency of our proposed solution. ACE consistently offers lower latency in various scenarios. In the best case, ACE offers 220 times less latency compared to HPC and HP, especially when data can be snooped from the PS caches. The advantage of ACE is most significant when maintaining coherency requires flushing in HPC and HP scenarios. In the memcpy benchmark, ACE consistently exhibits lower latency compared to HPC and HP ports, with a remarkable 48% lower latency in the best-case scenario. In the sparse matrix vector multiplication benchmark, ACE outperforms HPC and HP ports with a substantial 65% reduction in latency.