mirror of https://github.com/Qortal/Brooklyn
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
348 lines
15 KiB
348 lines
15 KiB
.. SPDX-License-Identifier: GPL-2.0 |
|
|
|
================= |
|
PCI NTB Function |
|
================= |
|
|
|
:Author: Kishon Vijay Abraham I <[email protected]> |
|
|
|
PCI Non-Transparent Bridges (NTB) allow two host systems to communicate |
|
with each other by exposing each host as a device to the other host. |
|
NTBs typically support the ability to generate interrupts on the remote |
|
machine, expose memory ranges as BARs, and perform DMA. They also support |
|
scratchpads, which are areas of memory within the NTB that are accessible |
|
from both machines. |
|
|
|
PCI NTB Function allows two different systems (or hosts) to communicate |
|
with each other by configuring the endpoint instances in such a way that |
|
transactions from one system are routed to the other system. |
|
|
|
In the below diagram, PCI NTB function configures the SoC with multiple |
|
PCI Endpoint (EP) instances in such a way that transactions from one EP |
|
controller are routed to the other EP controller. Once PCI NTB function |
|
configures the SoC with multiple EP instances, HOST1 and HOST2 can |
|
communicate with each other using SoC as a bridge. |
|
|
|
.. code-block:: text |
|
|
|
+-------------+ +-------------+ |
|
| | | | |
|
| HOST1 | | HOST2 | |
|
| | | | |
|
+------^------+ +------^------+ |
|
| | |
|
| | |
|
+---------|-------------------------------------------------|---------+ |
|
| +------v------+ +------v------+ | |
|
| | | | | | |
|
| | EP | | EP | | |
|
| | CONTROLLER1 | | CONTROLLER2 | | |
|
| | <-----------------------------------> | | |
|
| | | | | | |
|
| | | | | | |
|
| | | SoC With Multiple EP Instances | | | |
|
| | | (Configured using NTB Function) | | | |
|
| +-------------+ +-------------+ | |
|
+---------------------------------------------------------------------+ |
|
|
|
Constructs used for Implementing NTB |
|
==================================== |
|
|
|
1) Config Region |
|
2) Self Scratchpad Registers |
|
3) Peer Scratchpad Registers |
|
4) Doorbell (DB) Registers |
|
5) Memory Window (MW) |
|
|
|
|
|
Config Region: |
|
-------------- |
|
|
|
Config Region is a construct that is specific to NTB implemented using NTB |
|
Endpoint Function Driver. The host and endpoint side NTB function driver will |
|
exchange information with each other using this region. Config Region has |
|
Control/Status Registers for configuring the Endpoint Controller. Host can |
|
write into this region for configuring the outbound Address Translation Unit |
|
(ATU) and to indicate the link status. Endpoint can indicate the status of |
|
commands issued by host in this region. Endpoint can also indicate the |
|
scratchpad offset and number of memory windows to the host using this region. |
|
|
|
The format of Config Region is given below. All the fields here are 32 bits. |
|
|
|
.. code-block:: text |
|
|
|
+------------------------+ |
|
| COMMAND | |
|
+------------------------+ |
|
| ARGUMENT | |
|
+------------------------+ |
|
| STATUS | |
|
+------------------------+ |
|
| TOPOLOGY | |
|
+------------------------+ |
|
| ADDRESS (LOWER 32) | |
|
+------------------------+ |
|
| ADDRESS (UPPER 32) | |
|
+------------------------+ |
|
| SIZE | |
|
+------------------------+ |
|
| NO OF MEMORY WINDOW | |
|
+------------------------+ |
|
| MEMORY WINDOW1 OFFSET | |
|
+------------------------+ |
|
| SPAD OFFSET | |
|
+------------------------+ |
|
| SPAD COUNT | |
|
+------------------------+ |
|
| DB ENTRY SIZE | |
|
+------------------------+ |
|
| DB DATA | |
|
+------------------------+ |
|
| : | |
|
+------------------------+ |
|
| : | |
|
+------------------------+ |
|
| DB DATA | |
|
+------------------------+ |
|
|
|
|
|
COMMAND: |
|
|
|
NTB function supports three commands: |
|
|
|
CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before |
|
invoking this command, the host should allocate and initialize |
|
MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the |
|
Endpoint). The endpoint on receiving this command will configure |
|
the outbound ATU such that transactions to Doorbell BAR will be routed |
|
to the MSI/MSI-X address programmed by the host. The ARGUMENT |
|
register should be populated with number of DBs to configure (in the |
|
lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). |
|
|
|
CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The |
|
host invokes this command after allocating a buffer that can be |
|
accessed by remote host. The allocated address should be programmed |
|
in the ADDRESS register (64 bit), the size should be programmed in |
|
the SIZE register and the memory window index should be programmed |
|
in the ARGUMENT register. The endpoint on receiving this command |
|
will configure the outbound ATU such that transactions to MW BAR |
|
are routed to the address provided by the host. |
|
|
|
CMD_LINK_UP (0x3): Command to indicate an NTB application is |
|
bound to the EP device on the host side. Once the endpoint |
|
receives this command from both the hosts, the endpoint will |
|
raise a LINK_UP event to both the hosts to indicate the host |
|
NTB applications can start communicating with each other. |
|
|
|
ARGUMENT: |
|
|
|
The value of this register is based on the commands issued in |
|
command register. See COMMAND section for more information. |
|
|
|
TOPOLOGY: |
|
|
|
Set to NTB_TOPO_B2B_USD for Primary interface |
|
Set to NTB_TOPO_B2B_DSD for Secondary interface |
|
|
|
ADDRESS/SIZE: |
|
|
|
Address and Size to be used while configuring the memory window. |
|
See "CMD_CONFIGURE_MW" for more info. |
|
|
|
MEMORY WINDOW1 OFFSET: |
|
|
|
Memory Window 1 and Doorbell registers are packed together in the |
|
same BAR. The initial portion of the region will have doorbell |
|
registers and the latter portion of the region is for memory window 1. |
|
This register will specify the offset of the memory window 1. |
|
|
|
NO OF MEMORY WINDOW: |
|
|
|
Specifies the number of memory windows supported by the NTB device. |
|
|
|
SPAD OFFSET: |
|
|
|
Self scratchpad region and config region are packed together in the |
|
same BAR. The initial portion of the region will have config region |
|
and the latter portion of the region is for self scratchpad. This |
|
register will specify the offset of the self scratchpad registers. |
|
|
|
SPAD COUNT: |
|
|
|
Specifies the number of scratchpad registers supported by the NTB |
|
device. |
|
|
|
DB ENTRY SIZE: |
|
|
|
Used to determine the offset within the DB BAR that should be written |
|
in order to raise doorbell. EPF NTB can use either MSI or MSI-X to |
|
ring doorbell (MSI-X support will be added later). MSI uses same |
|
address for all the interrupts and MSI-X can provide different |
|
addresses for different interrupts. The MSI/MSI-X address is provided |
|
by the host and the address it gives is based on the MSI/MSI-X |
|
implementation supported by the host. For instance, ARM platform |
|
using GIC ITS will have the same MSI-X address for all the interrupts. |
|
In order to support all the combinations and use the same mechanism |
|
for both MSI and MSI-X, EPF NTB allocates a separate region in the |
|
Outbound Address Space for each of the interrupts. This region will |
|
be mapped to the MSI/MSI-X address provided by the host. If a host |
|
provides the same address for all the interrupts, all the regions |
|
will be translated to the same address. If a host provides different |
|
addresses, the regions will be translated to different addresses. This |
|
will ensure there is no difference while raising the doorbell. |
|
|
|
DB DATA: |
|
|
|
EPF NTB supports 32 interrupts, so there are 32 DB DATA registers. |
|
This holds the MSI/MSI-X data that has to be written to MSI address |
|
for raising doorbell interrupt. This will be populated by EPF NTB |
|
while invoking CMD_CONFIGURE_DOORBELL. |
|
|
|
Scratchpad Registers: |
|
--------------------- |
|
|
|
Each host has its own register space allocated in the memory of NTB endpoint |
|
controller. They are both readable and writable from both sides of the bridge. |
|
They are used by applications built over NTB and can be used to pass control |
|
and status information between both sides of a device. |
|
|
|
Scratchpad registers has 2 parts |
|
1) Self Scratchpad: Host's own register space |
|
2) Peer Scratchpad: Remote host's register space. |
|
|
|
Doorbell Registers: |
|
------------------- |
|
|
|
Doorbell Registers are used by the hosts to interrupt each other. |
|
|
|
Memory Window: |
|
-------------- |
|
|
|
Actual transfer of data between the two hosts will happen using the |
|
memory window. |
|
|
|
Modeling Constructs: |
|
==================== |
|
|
|
There are 5 or more distinct regions (config, self scratchpad, peer |
|
scratchpad, doorbell, one or more memory windows) to be modeled to achieve |
|
NTB functionality. At least one memory window is required while more than |
|
one is permitted. All these regions should be mapped to BARs for hosts to |
|
access these regions. |
|
|
|
If one 32-bit BAR is allocated for each of these regions, the scheme would |
|
look like this: |
|
|
|
====== =============== |
|
BAR NO CONSTRUCTS USED |
|
====== =============== |
|
BAR0 Config Region |
|
BAR1 Self Scratchpad |
|
BAR2 Peer Scratchpad |
|
BAR3 Doorbell |
|
BAR4 Memory Window 1 |
|
BAR5 Memory Window 2 |
|
====== =============== |
|
|
|
However if we allocate a separate BAR for each of the regions, there would not |
|
be enough BARs for all the regions in a platform that supports only 64-bit |
|
BARs. |
|
|
|
In order to be supported by most of the platforms, the regions should be |
|
packed and mapped to BARs in a way that provides NTB functionality and |
|
also makes sure the host doesn't access any region that it is not supposed |
|
to. |
|
|
|
The following scheme is used in EPF NTB Function: |
|
|
|
====== =============================== |
|
BAR NO CONSTRUCTS USED |
|
====== =============================== |
|
BAR0 Config Region + Self Scratchpad |
|
BAR1 Peer Scratchpad |
|
BAR2 Doorbell + Memory Window 1 |
|
BAR3 Memory Window 2 |
|
BAR4 Memory Window 3 |
|
BAR5 Memory Window 4 |
|
====== =============================== |
|
|
|
With this scheme, for the basic NTB functionality 3 BARs should be sufficient. |
|
|
|
Modeling Config/Scratchpad Region: |
|
---------------------------------- |
|
|
|
.. code-block:: text |
|
|
|
+-----------------+------->+------------------+ +-----------------+ |
|
| BAR0 | | CONFIG REGION | | BAR0 | |
|
+-----------------+----+ +------------------+<-------+-----------------+ |
|
| BAR1 | | |SCRATCHPAD REGION | | BAR1 | |
|
+-----------------+ +-->+------------------+<-------+-----------------+ |
|
| BAR2 | Local Memory | BAR2 | |
|
+-----------------+ +-----------------+ |
|
| BAR3 | | BAR3 | |
|
+-----------------+ +-----------------+ |
|
| BAR4 | | BAR4 | |
|
+-----------------+ +-----------------+ |
|
| BAR5 | | BAR5 | |
|
+-----------------+ +-----------------+ |
|
EP CONTROLLER 1 EP CONTROLLER 2 |
|
|
|
Above diagram shows Config region + Scratchpad region for HOST1 (connected to |
|
EP controller 1) allocated in local memory. The HOST1 can access the config |
|
region and scratchpad region (self scratchpad) using BAR0 of EP controller 1. |
|
The peer host (HOST2 connected to EP controller 2) can also access this |
|
scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This |
|
diagram shows the case where Config region and Scratchpad regions are allocated |
|
for HOST1, however the same is applicable for HOST2. |
|
|
|
Modeling Doorbell/Memory Window 1: |
|
---------------------------------- |
|
|
|
.. code-block:: text |
|
|
|
+-----------------+ +----->+----------------+-----------+-----------------+ |
|
| BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 | |
|
+-----------------+ | +----------------+ +-----------------+ |
|
| BAR1 | | | Doorbell 2 +---------+ | | |
|
+-----------------+----+ +----------------+ | | | |
|
| BAR2 | | Doorbell 3 +-------+ | +-----------------+ |
|
+-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 | |
|
| BAR3 | | | Doorbell 4 +-----+ | +-----------------+ |
|
+-----------------+ | |----------------+ | | | | |
|
| BAR4 | | | | | | +-----------------+ |
|
+-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3|| |
|
| BAR5 | | | | | | +-----------------+ |
|
+-----------------+ +----->-----------------+ | | | | |
|
EP CONTROLLER 1 | | | | +-----------------+ |
|
| | | +---->+ MSI-X ADDRESS 4 | |
|
+----------------+ | +-----------------+ |
|
EP CONTROLLER 2 | | | |
|
(OB SPACE) | | | |
|
+-------> MW1 | |
|
| | |
|
| | |
|
+-----------------+ |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
+-----------------+ |
|
PCI Address Space |
|
(Managed by HOST2) |
|
|
|
Above diagram shows how the doorbell and memory window 1 is mapped so that |
|
HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access |
|
buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and |
|
memory window 1 regions are allocated in EP controller 2 outbound (OB) address |
|
space. Allocating and configuring BARs for doorbell and memory window1 |
|
is done during the initialization phase of NTB endpoint function driver. |
|
Mapping from EP controller 2 OB space to PCI address space is done when HOST2 |
|
sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. |
|
|
|
Modeling Optional Memory Windows: |
|
--------------------------------- |
|
|
|
This is modeled the same was as MW1 but each of the additional memory windows |
|
is mapped to separate BARs.
|
|
|