High-speed I/O

Overview

Most of the data entering or leaving a modern FPGA goes through a small number of very fast pins. Gigabit transceivers carry PCIe, Ethernet, JESD204 and proprietary links at tens of gigabits per second per lane. Memory interfaces drive DDR4, DDR5 and LPDDR at several gigabits per pin. On the largest parts, high-bandwidth memory sits in the same package. This hub is for engineers whose design has to move data at those rates, and for the board designers who have to make the copper work.

The FPGA’s role is unusual here. The hard transceiver and memory PHY are fixed silicon with hundreds of configuration bits, the protocol layers are vendor IP or your own logic, and the board is yours. A link that does not come up can fail at any of those three levels, and the hard part is finding which.

Key ideas

What a transceiver does

A transceiver channel serialises a parallel word from the fabric onto a differential pair and recovers a parallel word at the far end. On the transmit side it adds pre-emphasis to compensate for the channel. On the receive side an equaliser (CTLE and DFE) undoes the channel’s loss, a clock and data recovery circuit (CDR) extracts the bit clock from the data itself, and a deserialiser widens the stream back to 32, 64 or 80 bits for the fabric. Line coding (8b/10b, 64b/66b, or 128b/130b for PCIe) keeps enough transitions for the CDR and bounds the DC balance. Above about 28 Gb/s per lane, vendors switch from NRZ to PAM4 signalling, which carries two bits per symbol at the cost of a much smaller eye. At the time of writing, 112 Gb/s PAM4 is shipping on AMD Versal Premium, Altera Agilex 7 and Achronix Speedster7t, and 224 Gb/s is on roadmaps.

Protocols and their IP

PCIe appears as a hard block on most mid-range and larger devices, with Gen3 at 8 GT/s, Gen4 at 16 GT/s and Gen5 at 32 GT/s common, and Gen6 at 64 GT/s with PAM4 arriving on the newest parts. The hard block implements the transaction and data link layers and exposes AXI or a vendor-specific streaming interface. Ethernet ranges from a soft 1G MAC through hard 10G, 25G, 100G and 400G MACs with forward error correction. JESD204B and JESD204C connect high-speed ADCs and DACs over serial lanes with deterministic latency and need a careful multi-device clock (SYSREF) distribution. Aurora and Altera’s SerialLite are lightweight point-to-point protocols for FPGA-to-FPGA links. Vendor IP for all of these is free to use on the vendor’s devices, and third-party IP exists where it is not.

Memory interfaces

External DDR goes through a hard or soft memory PHY and a controller. DDR4 runs to 3,200 MT/s on most FPGAs, DDR5 to 5,600 MT/s or more on the newest, and LPDDR4 and LPDDR5 appear on power-sensitive parts. The controller handles calibration at power-up, refresh and bank scheduling, and presents an AXI or Avalon port that is far wider and slower than the pins. HBM stacks memory dies beside the FPGA die in the same package, with thousands of connections and bandwidth of 400 GB/s to over 800 GB/s at the time of writing, on AMD Virtex UltraScale+ HBM, Versal HBM and Altera Agilex M-series. A design that needs that bandwidth has to be written to use many independent channels at once.

Signal integrity

A 10 Gb/s link has a unit interval of 100 ps, so trace loss, impedance discontinuities, via stubs and connector reflections all matter. Vendors publish IBIS and IBIS-AMI models of their I/O so that board designers can simulate the channel before building it. Reference clocks need low jitter, usually from a dedicated oscillator rather than a divided fabric clock. Every transceiver has an eye scan feature that reports the received eye opening from inside the chip, and it is the first thing to run when a link fails. For DDR, length matching, fly-by topology and termination are decided at layout, and the FPGA tool’s pin planner will refuse combinations it cannot calibrate.

Where to start

• /learn/beginner/04-registers-clocks-and-state/ covers clocks and clock domains, which every serial interface multiplies.
• /learn/beginner/09-memories-block-ram-and-fifos/ covers the FIFOs that buffer every high-speed interface into the fabric.
• /learn/beginner/10-from-rtl-to-board/ introduces pin constraints and I/O standards.

Intermediate lessons on a first transceiver loopback, a 1G Ethernet MAC and a DDR controller bring-up are coming.

Tools and vendors

Rates and generations change with each family release. Check the current datasheet.

Common mistakes

• Feeding a transceiver reference clock from a fabric PLL output instead of a dedicated low-jitter source.
• Swapping the polarity of a differential pair at layout and not knowing the transceiver can invert it in configuration.
• Placing via stubs on 25 Gb/s pairs without back-drilling.
• Assuming the PCIe hard block gives you a DMA engine. It gives you a bus. The DMA is separate IP or your own.
• Running DDR calibration once at the bench and not over the full temperature range.
• Ignoring the elastic buffer and clock compensation needs of a link with two independent clocks.

Going deeper

• AMD UG576, UltraScale Architecture GTH Transceivers: a complete description of a modern transceiver lane.
• AMD UG583, UltraScale Architecture PCB Design: layout rules for transceivers and memory on a real board.
• PCI-SIG specifications: the PCIe base specification (membership required for the full text).
• IEEE 802.3 Ethernet working group: the Ethernet standard and its amendments.
• JEDEC standards: DDR4, DDR5, LPDDR, HBM and JESD204 are all JEDEC documents, free with registration.
• IBIS Open Forum: the I/O buffer model specification used for channel simulation.