Glider
"In het verleden behaalde resultaten bieden geen garanties voor de toekomst"
About this blog

These are the ramblings of Matthijs Kooijman, concerning the software he hacks on, hobbies he has and occasionally his personal life.

Most content on this site is licensed under the WTFPL, version 2 (details).

Questions? Praise? Blame? Feel free to contact me.

My old blog (pre-2006) is also still available.

March
Sun Mon Tue Wed Thu Fri Sat
   
25
   
Powered by Blosxom &Perl onion
(With plugins: config, extensionless, hide, tagging, Markdown, macros, breadcrumbs, calendar, directorybrowse, entries_index, feedback, flavourdir, include, interpolate_fancy, listplugins, menu, pagetype, preview, seemore, storynum, storytitle, writeback_recent, moreentries)
Valid XHTML 1.0 Strict & CSS
USB, Thunderbolt, Displayport & docks

USB-C logos

After I recently ordered a new laptop, I have been looking for a USB-C-connected dock to be used with my new laptop. This turned out to be quite complex, given there are really a lot of different bits of technology that can be involved, with various (continuously changing, yes I'm looking at you, USB!) marketing names to further confuse things.

As I'm prone to do, rather than just picking something and seeing if it works, I dug in to figure out how things really work and interact. I learned a ton of stuff in a short time, so I really needed to write this stuff down, both for my own sanity and future self, as well as for others to benefit.

I originally posted my notes on the Framework community forum, but it seemed more appropriate to publish them on my own blog eventually (also because there's no 32,000 character limit here :-p).

There are still quite a few assumptions or unknowns below, so if you have any confirmations, corrections or additions, please let me know in a reply (either here, or in the Framework community forum topic).

Parts of this post are based on info and suggestions provided by others on the Framework community forum, so many thanks to them!

Getting started

First off, I can recommend this article with a bit of overview and history of the involved USB and Thunderbolt technolgies.

Then, if you're looking for a dock, like I was, the Framework community forum has a good list of docks (focused on Framework operability), and Dan S. Charlton published an overview of Thunderbolt 4 docks and an overview of USB-C DP-altmode docks (both posts with important specs summarized, and occasional updates too).

Then, into the details...


Lines, lanes, channels, duplex, encoding and bandwidth

  • The transmission of high-speed signals almost exclusively used "differential pairs" for encoding. This means two wires/pins are used to transmit a single signal in a single direction (at the same time). In this post, I will call one such pair a "line".
  • A single line can either be simplex (unidirectional, transmitting in the same direction all the time) or half-duplex (alternating between directions).
  • Two lines (one for each direction) can be combined to form a full-duplex channel, where data can be transmitted in both directions simultaneously. Sometimes (e.g. in USB, PCIe) such a pair of lines (so 4 wires in total) is referred to as a "(full duplex) lane", but since the term "lane" is also used to refer to a single unidirectional wire pair (e.g. in DisplayPort), I'll use "full duplex channel" for this instead.
  • I'll still sometimes use "lane" when referring to specific protocols, but I'll try to qualify what I mean exactly then.
  • A line will be operated at a specific bitrate, which is the same as the available bandwidth for that line. Multiple lines can be combined, leading to higher total bandwidth (but the same bitrate).
  • To really compare different technologies, it is often useful to look at the bitrate used on the individual lines, for which I'll use Gbps-per-line below. I'll use "total bandwidth" refer to the sum of all the lines used in either direction, and "full duplex bandwidth" means the bandwidth that can be used in both directions at the same time.
  • Most of the numbers shown here are for bits on the wire. However, that does not represent the amount of actual data being sent. For example, SATA, PCIe 2.0, USB 3.0, HDMI, and DisplayPort 1.4 transmit 10 bits on the wire for every 8 bits of data, otherwise known as 8b/10b encoding. This means USB 3.0 is more like 4 Gbps instead of 5 Gbps. This data includes protocol overhead, such as framing, headers, error correction, etc. so the effective data transfer rates (i.e. data written to disk) is lower still.

USB1/2/3

  • USB1/2: single half-duplex pair up to 480Mbs half-duplex.
  • USB1/2 link setup happens using pullups/pulldowns on the datalines, always starting up at USB1 speeds (low-speed/full-speed) and then using USB data messages to coordinate the upgrade to USB2 speeds (hi-speed).
  • When a device using USB1 speeds ("low speed" or "full speed") is connected to a USB2 hub, the difference in speed is translated using a "Transaction Translator". Most hubs have a single TT for the entire hub, meaning that when multiple such devices are connected, only one of them can be transmitting at the same time, and they have to share the 12Mbps (full speed) bandwidth amongst them. In practice, this can sometimes lead to issues, especially with devices that need guaranteed bandwidth, like USB audio devices. Note that this applies even to modern devices that support USB2, but only implement the slower transmission speed (like often audio devices, mice, keyboard, etc.). Some hubs have multiple TTs, meaning each downstream port has the full 12Mbps available, making multi-TT hubs a better choice (but unfortunately, manufacturers usually do not specify this, you can use lsusb -v to tell).
  • USB3.0: Adds two extra lines at 5Gbps-per-line (== 1 full-duplex channel, 10Gbps total bandwidth) using a new "USB3" connector with 4 extra data wires.
  • USB3 is a separate protocol from USB1/2 and works exclusively over the newly added lines. This means that a USB3 hub is essentially two fully separate devices in one (USB1/2 hub and USB3 hub) and USB3 traffic and USB1/2 traffic can happen at the same time over the same cable independently.
  • USB3.0 link setup uses "LFPS link training" pulses, where both sides just start to transmit on their TX line and listen on their RX line). [source].
  • USB3.0 over a USB-C connector only uses 2 of the 4 high-speed lines available in the connector and cable.
  • USB3.1 increases single line speed to 10Gbps (with appropriate cabling), for 20Gbps total bandwidth.
  • USB3.2 allows use of all 4 high-speed lines in a USB-C connection at the same 10Gbps-per-line speed of 3.1, for 40Gbps total bandwidth.

DisplayPort

[source]

  • Displayport is essentially a high-speed 4 line unidirectional connection (main link) for display and audio data, plus one lower speed half-duplex auxilliary channel.
  • The auxilliary channel is used for things like EDID (detecting supported resolution) and CEC (controlling playback and power state of devices). Deciding the link speed to use on the high-speed lines is done using "link training" on those lines themselves, not using the aux channel. [source]
  • Unlike VGA/DVI/HDMI, Displayport is based on data packets (similar to ethernet), rather than a continuous (clocked) stream of data on specific pins.
  • Displayport speed evolved over time: 4x1.62=6.48Gbps for DP1/RBR, 4x2.7=10.8Gbps for DP1/HBR, 4x5.4=21.6Gbps for DP1.2/HBR2, 4x8.1=32.4Gbps for DP1.3/HBR3, 4x10=40Gbps for DP2.0/UHBR10, 4x13.5=54Gbps for DP2.0/UHBR13.5, 4x20=80Gbps for DP2.0/UHBR20. All versions use the same 4-line connection, just driving more data over the same wires (needing better quality cables). [source]
  • Displayport 1.2 introduced optional Display Stream Compression (DSC), which is a "visually lossless" compression that can reduce bandwidth usage by a factor 3. Starting with DisplayPort 2.0, supporting DSC is mandatory.
  • Displayport 1.2 introduced Multi-Stream Transport (MST), which allow sending data for multiple displays (up to 63 in theory) over a single DP link (multiplexed inside the datastream, so without having to dedicate lines to displays). Such an MST signal can then be split by an MST hub (often a 2-port hub inside a display to allow daisy-chaining displays). MST is also possible for DP inside an USB-C or thunderbolt connection. It can also use DSC (compression) on the input and uncompressed on the output, to drive displays that do not support compression. [source]
  • Displayport dualmode (aka DP++) allows sending out HDMI/DVI signals from a displayport connector. This essentially just repurposes pins on the DP connector to transfer HDMI/DVI signals, so can be used with a passive adapter or cable (though the adapter does need to do voltage level shifting). This only works to connect a DP source to a HDMI/DVI sink (display), not the other way around. This also repurposes two additional CONFIG1/2 pins from being grounded to carry HDMI/DVI signals, so dual-mode is not available on an USB-C DP alt mode connection. [source]
  • Some Displayport devices support Adaptive Sync, where the display refresh rate is synchronized with the GPU framerate. This seems to still work when DisplayPort is tunneled through Thunderbolt or USB4, but often breaks when going through an MST hub (though some explicitly document support for it and probably do work). [source]

Thunderbolt 1/2/3

[source]

  • Thunderbolt 1/2 is an Apple/Intel specific protocol that reuses the mini-displayport connector to transfer both PCIe (for e.g. fast storage or external GPUs) and DP signals using four 10Gbps high-speed lines (40Gbps total bandwidth) configured as two 10Gbps full-duplex channels (TB1) or a single 20Gbps full-duplex channel (TB2). [source].
  • It is a bit unclear to me how these signals are multiplexed. Apparently TB2 combines all four lines into a single full-duplex channel, suggesting that on TB1 there are two separate channels, but does that mean that one channel carries PCIe and one carries DP on TB1? Or each device connected is assigned to (a part of) either channel?
  • TB1/2 are backwards compatible with DP, so you can plug in a non-TB DP display into a TB1/2 port. This uses a multiplexer to connect the port to either the TB controller or the GPU. This mux is controlled by snooping the signal lines and detecting if the signals look like DP or TB. In early Light Ridge controllers, this was external circuitry, from Cactus Ridge all this (including the mux) was integrated in the TB controller. [source: comments below]
  • TB1/2/3 ports can be daisychained with up to 6 devices in the chain. Hubs (i.e. a split in the connection) are not supported until TB4. The last device in the chain can be a non-TB DP display as well. [source]
  • USB (1/2/3) over thunderbolt is not supported directly, but is implemented by putting a full USB-host controller in a thunderbolt hub/dock that is then controlled by the host through PCIe-over-thunderbolt.
  • Thunderbolt 3 is an upgrade of TB2 that uses an USB-C connector instead of mini-displayport. It uses 4 high-speed lines at 20Gbps-per-line (double of TB2), to get 40Gbps full-duplex, or 80Gbps total bandwidth (and maybe also the sidechannel SBU line for link management or so, but I could not find this). [source]
  • Thunderbolt 4 seems to be essentially just USB4 with all optional items implemented (but USB4 is more like TB3 than USB3...).
  • TB1/2/3 are only implemented by Intel chips, and TB1/2 is used almost exclusively by Apple. TB4/USB4 can be implemented by other chip makers too, but this has not happened yet I believe.

USB-C

[source]

  • USB-C is a specification of a connector, intended to use for USB by default, but also for other protocols (usually in addition to USB) and not tied to any particular USB version.
  • USB-C has a huge number of pins (24 pins), consisting of:
    • 4 VBUS and 4 GND pins to allow delivering more current,
    • 4 high-speed lines (8 pins), typically used as two full-duplex channels,
    • USB2 D+/D- pins (duplicated, to allow reverse insertion without additional circuitry)
    • Two CC/VCONN pins for low-speed connection setup, USB-PD negotiation, altmode setup, querying the cable, etc.
    • Two SBU sidechannel pins as an additional lower speed line.
  • USB-C cables come in different flavors: [source]
    • The simplest cables have just power and USB2 pins, and do not support anything other than USB1/2. Cables that do connect all pins are apparently called "full-featured", though these might still not support the higher (10/20Gbps-per-line) speeds.
    • Cables can be passive (direct copper connections), or active (with electronics in the cable to repeat/amplify a signal to allow higher speeds and longer cables).
    • Active cables typically seem to be geared towards (and/or limited to) a particular signalling scheme and/or direction. This means that they are usually not usable for protocols they are not explicitly designed for. This partially comes down to the bitrates used, but also the exact low-level signalling protocol (e.g. USB3 Gen2, USB4 Gen2 and DP UHBR 10 all use 10Gbps-per-line, but use different encodings, error correction and/or direction, meaning active cables might support one, but not the other).
    • Passive cables can generally be used more flexibly, mainly limited to a certain bitrate based on cable quality and length.
    • Cables can also contain a "e-marker" chip that can be queried using USB-PD messages on the CC pin, containing info on the cable's spec. Such a chip is required for 5A operation (all other cables are expected to allow up to 3A) and for all USB4 (even the lower-speed 10Gbps-per-line USB4) operation.
    • Cables with "e-marker" chip but no signal conditioning are still considered passive cables.
    • USB4/TB4 requires an e-marker chip in the cable (falling back to USB3 or TB3 without it). Passive cables (including passive TB3 cables) can always be used for USB4 (with the exact speed depending on the cable rating), while active cables can only be used when they are rated for USB4 (so active USB3 cables cannot be used for USB4, but active USB4 cables can be used for USB3). [source]
  • USB-C uses the CC pins for detecting a connection and deciding how to set it up. A host (or dowstream facing port or sourcing port) has pullups on both CC pins (value depens on available current), a device (or upstream facing port or sinking port) has pulldowns on both pins (value is fixed). Together this forms a voltage divider that allows detecting a connection and maximum current. A cable connects only one of the CC pins (leaving the other floating, or pulling it down with a different resistor value) which allows detecting the orientation of the cable. [source]
  • After initial connection, the CC pin is also used for other management communication, such as USB-PD (including setting up alt modes and USB4), or vendor-specific extensions.
  • The other CC pin (pulled down inside the cable) is repurposed as VCONN and used to power any active circuitry (including the e-marker) inside the cable.
  • USB-C can provide up to 15W (5V×3A) on the VBUS pins with passive negotiation using resistors on the CC pins. USB-PD uses active negotiation using data messages on the CC pin and allows up to 100W (20V×5A) of power on the VBUS pins and allows reversing the power direction as well (i.e. allow a laptop USB-C port to receive power instead of supplying it).

USB-C alt modes

  • USB-C alt modes allow using the pins (in particular the 4 high-speed lines and the lower speed SBU line) for other purposes.
  • Alt modes are negotiated actively through the CC pin using USB-PD VDM messages. Once negotiated, this allows using these pins different physical signalling, including different directions. This means that e.g. a USB-C-to-DP adapter using DP alt mode can be mostly a passive device, though it does need some logic for the USB-PD negotiation and might need some switches to disconnect the wires until the alt mode is enabled.
  • Alt modes are single-hop only (hubs might connect upstream and downstream alt mode channels, but there is no protocol to configure this AFAICT).
  • Alt modes are technically separate from the USB protocol (version) supported (i.e. they can be configured without having to do USB communication), but the bandwidth capabilities does introduce some correlation (i.e. DP alt mode 2.0 supports DP 2.0 and might need up to 20Gbps-per-line, so effectively needs a USB4-capable controller, cable and hub/dock).
  • DP alt mode (and probably others) can use 1, 2, or all 4 of the high-speed lines (plus the SBU line for the aux channel). When using only 1 or 2 lines, the 2 other lines can still be used for USB3 traffic (just using 1 line for USB is not possible, USB3/4 always needs a full-duplex channel, so 2 or 4 lines). In all cases, the USB2 line can still be used for USB1/2.
  • Because in DP alt mode all four lines can be used unidirectionally (unlike USB, which is always full-duplex), this means the effective bandwidth for driving a display can be twice as much in alt mode than when tunneling DP over USB4 or using DisplayLink over USB3.

    In practice though, DP-altmode on devices usually supports only DP1.4 (HBR3 = 8.1Gbps-per-line) for 4x8.1 = 32.4Gbps of total and unidirectional bandwidth, which is less than TB3/4 with its 20Gbps-per-line for 2x20 = 40Gbps of full-duplex bandwidth. This will change once devices start support DP-altmode 2.0 (UHBR20 = 20Gbps-per-line) for 4x20=80Gbps of unidirectional bandwidth.

  • It seems that DP alt mode can only be combined with USB3, not TB3 or USB4 (USB4 and DP alt mode both need the SBU pins). Since DP can be tunneled over TB/USB4, there is not much need for DP alt mode when using TB/USB4, except that it could potentially have allowed more bandwidth (2x20Gbps unidirectional DP2.0 plus 1x20Gbps full-duplex TB/USB4).
  • HDMI alt mode exists, but is apparently not actually implemented by anyone. USB-C-to-HDMI adapters typically seem to use DP alt mode combined with an active DP-to-HDMI converter. I'm assuming this also holds for the Framework HDMI expansion cards. [source]
  • Thunderbolt 3 is also implemented as an alt mode.
  • Audio Adapter Accessory Mode allows sending analog audio data over an USB-C connector (using the USB2 and sideband pins). This looks like another alt mode, but instead of active negotiation over the CC pin, this mode is detected by just grounding both CC pins so adapters for this mode can be completely passive. [source]
  • Ethernet or PCIe are sometimes mentioned as alt modes, but apparently do not actually exist. [source]

USB4 / Thunderbolt 4

[source] and [source] and [source]

  • USB4 is again really different from USB3 and looks more like Thunderbolt 3 than USB3.
  • USB4 does not specify any device types by itself (like USB1/2/3), but is a generic bulk data transport that allows tunneling of other protocols, like PCIe, DisplayPort and USB3.
  • USB4 uses the 4 USB-C high-speed lines at up to 20Gbps-per-line for 40Gbps full-duplex and 80Gbps total bandwidth.
  • USB4 link setup happens using USB-PD messages over the CC pins, falling back to USB3 setup if no USB4 link is detected within a second or so (so also in that sense USB4 is also more like Thunderbolt 3 alt mode than USB3). [source]
  • USB4 also uses two additional pins (sideband channel) of the USB-C connector for further link initialization and management, so it needs USB-C and cannot be used over USB3-capable USB-A/USB-B/USB-micro connectors and cables. [source]
  • Like with TB3, this tunneling happens on a data level: The physical layer signalling is defined by USB4 and different tunneled protocols are mixed in the same bitstream (unlike USB-C alt mode, where individual lines are completely allocated to some alt mode, which also determines the physical signalling used). This also means that DP alt mode and USB4-tunneled-displayport are two different ways to transfer DP over a USB4-capable connection.
  • Unlike with TB3, USB2 is not tunneled but still ran in parallel over the USB2 pins.
  • Unlike with TB3, USB3 is tunneled directly over USB4 and uses the host USB3 controller with only USB3 hubs in docks (though I guess docks could still choose to include PCIe connected USB controllers). In some cases, this might result in lower total USB bandwidth (10 or 20Gbps shared by all devices) compared to the TB3 approach of (sometimes multiple) USB3 controllers in the dock that are accessed through tunneled PCIe.
  • USB4 end devices have most features optional. USB4 hosts require 10Gbps-per-line, tunneled 10Gbps USB3, tunneled DP, and DP altmode (unsure if this requires DP altmode 2.0), leaving 20Gbps-per-line, tunneled 20Gbps USB3, tunneled PCIe, TB3 and other altmodes optional. USB4 hubs require everything, except 20Gbps USB3 and other altmodes. [source]
  • Thunderbolt 4 is (unlike thunderbolt 3) not really a protocol by itself, but is essentially just USB4 with all or most optional features enabled, and with additional certification of compatibility and performance. In particular, TB4 ports must support 20Gbps-per-line, computer-to-computer networking, PCIe, charging through at least one port, wake-up from sleep, 2x4K@60 or 1x8K@60 display, up to 2m passive cables. I have not been able to find the actual specification to see what these things really mean technically (i.e. are these display resolutions required over USB4, or altmode (2.0), must they be uncompressed, from which states does this wakeup wake up, etc.). [source] and [source]
  • TB1/2/3 were propietary protocols, implemented only on Intel chips. USB4 (and also TB4) are (somewhat) more open and can be implemented by multiple vendors (though I could find the USB4 spec easily, but not the TB4 spec, or the TB3 spec which is required for compatibility, or the DP alt mode spec, which is required for USB4 ports).
  • USB4 hubs must also support the TB3 for compatibility, controllers and devices may support TB3. [source]
  • Different USB3/4 transfer modes have been renamed repeatedly (i.e. USB3.0 became USB3.1 gen1 and then USB3.2 gen1x1). Roughly, gen1 (aka SuperSpeed aka SuperSpeed USB) means 5Gbps-per-line, gen2 (aka SuperSpeed+ aka SuperSpeed USB 10Gbps, or SuperSpeed USB 20Gbps aka USB4 20Gbps for gen2x2) means 10Gbps-per-line, gen3 (aka USB4 40Gbps) means 20Gbps per line. No suffix or x1 means using only one pair of lines, x2 means two pairs of lines (so e.g. gen2x2 uses all four lines, at 10Gbps-per-line, for 20Gbps full-duplex and 40Gbps of total bandwidth). Also, you can mostly ignore the USB versions (i.e. USB3.1 gen1 and USB3.2 gen1x1 are the same transfer mode), except for USB4 gen2, which is also 10Gbps-per-line, but uses different encoding. [source]
  • A USB4 hub now effectively contains 3 separate USB hubs: A USB1/2 hub, a USB3 hub, and a USB4 hub (aka USB4 router). As with USB3 hubs, the USB1/2 hub is completely distinct, always connected to the USB-C connectors and its communication runs in parallel with the USB3/4 communication. The USB3 (superspeed) hub can either connect directly to the USB-C connectors directly (when an USB3 device or host is connected), or it can connect through the USB4 router to receive / send USB4-tunneled-USB3 traffic. USB3 traffic seems to be always tunneled over just one hop in the connection, so a host -> USB4 hub -> USB4 hub -> USB3 device connection is tunneled in USB4 from the host to the first hub, is then unpacked into the USB3 hub, then tunneled again for the trip to the second hub, unpacked again into the USB3 hub inside the second hub, and then to the USB3 device directly over the USB-C (or USB-A) connector without further tunneling. [source]
  • PCIe tunneling works similar to USB3 tunneling, where the tunneling is done over a single hop, unpacked into a PCIe switch, and then tunneled again for further hops is needed. [source]
  • For DP tunneling, this works differently, there a single tunnel can stretch over multiple hops, all the way from the DP source in the USB4 host towards the DP output in the last USB4 hub (or USB4-capable display). [source]

Thunderbolt / USB4 hardware

  • Thunderbolt 1/2/3 is exclusively implemented by Intel controllers. USB4/TB4 (including TB3 compatibility) is open for others to implement too (but currently Intel controllers are still used mostly).
  • Intel Thunderbolt and USB4 controller families are all named "Something Ridge". Initially there were quite a few families implementing TB1, then Falcon Ridge implemented TB2, Alpine and Titan ridge for TB3, Maple and Goshen ridge for TB4 [source].
  • Most of these controllers can be used in hosts, devices and hubs, but some of them are exclusively for devices/hubs (such as Goshen Ridge) or hosts.
  • Originally, thunderbolt was implemented using discrete thunderbolt controller chips, which were connected to the CPU using high-speed PCIe and DP traces on the mainboard, making thunderbolt support a complex and expensive thing. With Intel Ice Lake and Tiger Lake, the thunderbolt controller is integrated in the processor, making the implementation cheaper and more power efficient. [source] and [source]

Understanding tunneling and bandwidth limitations

[source]

  • Looking at how Thunderbolt/USB4 hardware is constructed, you typically have a controller chip (router in USB4 terminology) that handles the TB/USB4 traffic (and usually also TB3/USB3 backward compatibility). Both the host and the dock (or real TB device) have such a controller (often the same controller can be used in hosts, docks/hubs and devices).
  • These controllers have one or more TB outputs, connected to TB (USB-C) connectors, forming the TB link between then.
  • These controllers also have other interfaces (adapters in USB4 terminology), such as DP inputs or outputs, PCIe upstream/downstream interfaces, or USB upstream/downstream interfaces.
    • In the host these interfaces are usually connected to the CPU/GPU. USB links are usually internally connected to a USB controller integrated into the controller.
    • In a dock/hub DP interface(s) are usually connected to output connectors (DP, HDMI, TB or USB-C). These are often multiplexed, so there might be 2 DP interfaces available that can be routed to any two of the available outputs, or they can be connected through an internal MST hub.
    • In a dock/hub PCIe interface(s) are usually internally connected to devices (e.g. PCIe ethernet) or exposed (e.g. PCIe NVMe slot). If multiple PCIe lanes are available, they can be connected to the same device, or split among multiple.
    • In a dock/hub the USB interface(s) are usually internally connected to USB2/3 hubs (often integrated into the TB/USB4 controller).
    • Thunderbolt ports can usually be internally connected to the TB interface, a DP interface (for DP alt mode) or an USB3 controller (for non-thunderbolt USB compatibility).
  • Often these interfaces provide a transparent connection, where e.g. the GPU negotiates a certain DP bitrate with an attached display through the tunnel, just as if the display was directly connected.
  • On Linux, to get some basic info on present TB controllers and devices and the used link speed, you can use boltctl (e.g. boltctl list -a).
  • On Linux, to see what interfaces are present on e.g. a TB dock, run echo "module thunderbolt +p" > /sys/kernel/debug/dynamic_debug/control, plug in your dock and check dmesg (which will call the USB4 controller/router in the the dock a "USB4 switch" and its interfaces "Ports").
  • When looking at what limits / determines the bandwidth available when a tunneled (i.e. TB3 or TB4) link is involved, there are mainly two kinds of limit:
    1. The total bandwidth of (each hop of) the link itself. This bandwidth is shared between all tunneled connections and depends on the protocol and cable used (e.g 20/40Gbps for TB3/TB4/USB4 depending on cable and controller capabilities, TB3/TB4 controllers always support 40Gbps).
    2. The bandwidth (and number) of individual tunnels. This is mostly a limit are limited by the hardware interfaces present on the controllers/routers used (e.g. dual 4xHBR2 interfaces on Alpine Ridge, dual 4xHBR3 interfaces on Titan Ridge). These limitations are mostly linked to the hardware used, not so much limited by the protocol (though the protocols do seem to typically specify minimum bandwidth/features for each port).
  • The second limit is the most complex to figure out, because it involves limitations on both ends of the connection (and for USB3/PCIe also any intermediate steps).
  • The link bandwidth is shared between all tunnels on a single port, but the tunnel bandwidth is often shared between all ports on the same controller.
  • When multiple tunnels run over the same link, only the actual bandwidth usage counts (except DP does limit links based on max link bandwidth, see below). For example, an USB3 10Gbps tunnel with only a thumb drive connected will only use very little bandwidth, leaving the rest available for other tunnels. Only when the combined bandwidth usage of all tunnels exceeds the link bandwidth will bandwidth of each tunnel be limited.
  • The number of tunnels over a single link seems to be practically unlimited by the protocols, but only limited by the number of interfaces on the used controllers (so we could in theory see future controllers that support three or four DP interfaces, though that probably does not make too much sense given MST exists and total bandwidth is still limited). Maybe TB1/2/3 do have protocol limitations, but since it is unlikely that new controllers will be made for these, it won't matter anyway.
  • For DP:

    • TB1 controllers support one 4xHBR interface (DP1.0), TB2 controllers support one 4xHBR2 interface (DP1.2), Alpine Ridge (TB3) supports two 4xHBR2 interfaces (2xDP1.2), Titan Ridge (TB3), Goshen/Maple Ridge (TB4) and Tiger Lake (TB4) support two 4xHBR3 interfaces (2xDP1.4).
    • Each DP interface consists of four lanes, which cannot be split.
    • Each of the interfaces (four lanes each) on the host controller can be routed to a single dock or display (where they can still be split into multiple DP or DP-alt-mode outputs using MST, but they cannot be split into multiple TB-tunneled-DP-links).
    • Goshen Ridge seems to have no dedicated DP output pins, but it can still internally multiplex its two DP interfaces to all three of its downstream TB ports for DP altmode. Docks that have a dedicated DP connector usually sacrifice one of these TB ports (leaving only two downstream TB ports) which is then tied to the DP connector (and probably permanently configured in DP "altmode" in firmware).
    • DP links can be fully transparent (a single DP link between GPU and display), or the tunnel can be a "DP LTTPR" (like an active cable) resulting in two DP links (GPU to host TB controller, dock TB controller to display). [source, section 2.2.10.2]
    • Note that the full bandwidth on the newest controllers (2x4xHBR3 == 51.84Gbps after encoding) cannot be sent over a single TB3/TB4 port (40Gbps after encoding). But when using two TB ports on the same controller, both can transmit a full 4xHBR3 signal, allowing to use the full DP bandwidth of the controller.
    • Bandwidth allocation for DP links happens based on the maximum bandwidth for the negotiated link rate (e.g. HBR3) and seems to happen on a first-come first-served basis. For example, if the first display negotiates 4xHBR3, this takes up 25.92Gbs (after encoding) of bandwidth, leaving only 2xHBR3 or 4xHBR1 for a second display connected.

      This means that the order of connecting displays can be relevant to the supported resolutions on each (on multi-output hubs with displays already connected, it seems the hub typically initializes displays in a fixed order).

      If the actual bandwidth for the resolution used is less than the allocated bandwidth, the extra bandwidth is not lost, but can still be used for other traffic (like bulk USB traffic, which does not need allocated / reserved bandwidth). [source]

    • In some cases, it seems bandwidth can still be over-allocated if the OS knows that the full bandwidth will not be used (e.g. allocate 2x4xHDR3, knowing that with the resolutions in use, the total bandwidth will fit in the available 40Gbps). [source]
    • MST hubs often seem to allocate maximum supported bandwidth whenever the first display is connected (to prevent the need for renegotiation when another display is added), which might significantly over-allocate. [source]
  • For USB3:
    • When using USB4/TB4, the USB3 controller is in the host, with hubs in downstream devices. This means the total USB3 bandwidth is 5, 10 or 20Gbps, limited by the hosts's USB3 controller (often integrated into the TB/USB4 controller), as well as any hubs along the way. This bandwidth is probably also shared between all (both) ports connected to the same TB/USB4 controller.
    • USB3 tunnels are single-hop, so on a USB4/TB4 dock, the TB4/USB4 controller has an upstream USB interface (tunneled to the upstream host or hub) and one downstream interface for each downstream TB4 port (tunneled to the downstream devices or hubs) and an (integrated) USB3 hub that connects all these USB3 interfaces.
    • When using TB3, the USB3 controller (and usually also one or more hubs) is in the (each) dock, connected to the host through tunneled PCIe. This means that the USB3 bandwidth is limited by the controllers in the docks (each controller having its own chunk of 5/10/20Gbps). When there are multiple USB3 controllers involved (inside the same dock, in multiple daisy-chained docks, or connected to multiple ports using the same host TB controller), the total bandwidth can be higher but is still limited by the available PCIe bandwidth.

Bandwidths & Encoding

[source] and [source] and [source]

As mentioned, different protocols use different bitrates and different encodings. Here's an overview of these, with the effective transfer rate (additional protocol overhead like packet headers, error correction, etc. still needs to be accounted for, so i.e. effective transfer rate for a mass-storage device will be lower still). Again, these are single-line bandwidths, so total bandwidth is x4 (unidirectional) for DP and x2 (full-duplex) for TB/USB3.2/USB4.

  • PCIe 3.0 8 Gbps using 128b/130b = 7.877 Gbps
  • PCIe 6.0 64 Gbps using 242b/256b = 60.5 Gbps
  • DP 1.0/1.1 RBR 1.62Gbps using 8b/10b = 1.296Gbps
  • DP 1.0/1.1 HBR 2.70Gbps using 8b/10b = 2.16Gbps
  • DP 1.2 HBR2 5.40Gbps using 8b/10b = 4.32Gbps
  • DP 1.3/1.4 HBR3 8.10Gbps using 8b/10b = 6.48Gbps
  • DP 2.0 UHBR10 10Gbps using 128b/132b = 9.697Gbps
  • DP 2.0 UHBR13.5 13.5Gbps using 128/132b = 13.091Gbps
  • DP 2.0 UHBR20 20Gbps using 128/132b = 19.394Gbps
  • USB 3.x 5 Gbps (gen 1) using 8b/10b = 4 Gbps
  • USB 3.x 10 Gbps (gen 2) using 128b/132b = 9.697 Gbps
  • USB4 10 Gbps (gen 2) uses 64b/66b = 9.697 Gbps
  • USB4 20 Gbps (gen 3) uses 128b/132b = 19.39 Gbps
  • Thunderbolt 3 10.3125Gbps uses 64b/66b = 10Gbps
  • Thunderbolt 3 20.625Gbps uses 64b/66b = 20Gbps

Note that for TB3, the rounded 10/20Gbps speeds are after encoding, so the raw bitrate is actually slightly higher. Thunderbolt 4 is not listed as (AFAICT) this just uses USB4 transfer modes.

Displaylink

[source]

  • Displaylink is a technology (made by the company with the same name) that allows sending display data over USB2/3.
  • Some DisplayLink chips also support ethernet and analog audio, presumably by just presenting these as parts of a composite device.
  • Displaylink is used in docks as well as inside monitors directly, to allow sending video without needing any specific hardware support other than USB2/3. A driver for these devices is needed in the OS though.
  • It seems at least some DisplayLink devices are supported in Linux, though mostly using a closed-source binary blob. There is also basic support in open-source drivers, but these are apparently reverse-engineered, so I suspect DisplayLink is not forthcoming with documentation or code themselves. [source]
  • DisplayLink performance is limited, mainly because even though the laptop GPU can apparently be used for some acceleration, actually transferring the resulting display data is largely a software/driver affair, involving compression and wrapping data in USB packets. Performance can also be affected by other activity on the bus. DisplayLink seems to be mostly intended for office work, less for video and gaming (unlike other solutions, like DP alt mode or tunneling DP over Thunderbolt or USB4, where the GPU takes care of generating the DisplayPort data stream, which is delivered directly to the USB/Thunderbolt controller without involving software, and which allocates dedicated bus bandwidth for that stream). [source]

Docks / hubs

  • Docks come roughly in two flavors:
    • "USB-C docks", without thunderbolt, that connect to the host using USB2, USB3 and/or DP-altmode. These are typically cheaper, but also run at lower bandwidth.
    • "Thunderbolt 3/4" docks, which tunnel everything (including USB3) over Thunderbolt/USB4 (for TB3 even USB2.0 is tunneled, for TB4 it runs over its own wires). These typically have higher upstream bandwidth and more features.
    • USB4 (non-Thunderbolt) docks could technically exist too, but since TB4 is really just USB4 with most optional features made mandatory (and some additional certification), and most of these optional features are already mandatory for USB4 hubs (including even TB3 compatibility), it seems likely that such hubs will just be made TB4-compatible anyway.
  • Thunderbolt docks often also support a fallback mode where they work like a USB-C dock using USB2/3 and/or DP-alt-mode to connect to the host, at reduced bandwidth and features (i.e. PCIe connected devices are not supported).
  • USB-A ports on docks are fairly uncomplicated and will just support USB devices (using either USB2 or USB3), though note the caveat about single-TT vs multi-TT above.
  • Downstream USB-C ports are more tricky: I expect these will always support USB2 and/or 3, but it seems that often only a selection also supports TB3, USB4/TB4, or DP alt mode.
  • I suspect that if a dock is TB4-certified, it must probably support all TB4 features on all of its downstream USB4 ports too (which are probably going to be labeled "TB4" ports anyway). USB4 downstream ports are at least required to support DP alt mode.
  • Video on docks is complex, see below.
  • Other things like ethernet, SD-card readers and audio are typically just implemented using USB devices inside the dock, connected to an integrated USB2/3 hub.
  • Some docks have their ethernet (especially 2.5G/10G) connected through PCIe instead of USB. These often allow better performance (because these chips often offload more processing, and this moves bandwidth usage from the USB3 tunnel to the PCIe tunnel, which can be helpful depending on what else is connected), but does require specific drivers for the ethernet chip used (often shipped with OS's, though). See the source for a list of docks and the ethernet chips they use. [source]

Displays on docks / hubs

  • Here the situation becomes more complicated. There seem to be at least 6 ways that display data can be transferred over USB-C to a dock: DP alt mode 1.0, DP alt mode 2.0, tunneled through USB4, tunneled through TB3, using a PCIe-connected eGPU (where PCIe can run over TB3 or USB4), or using DisplayLink or a similar USB2/3 connected device.
  • AFAICS, DisplayLink has significant downsides (so I would avoid it), a PCIe-connected eGPU is quite expensive (so only used when you really need that extra GPU power), and the other options (altmode and tunneling DisplayPort) are largely equivalent (except for available bandwidth and thus resolution).
  • It seems DP-over-USB4 and DP-over-Thunderbolt (also TB3) can always be sent through multiple hops (i.e. when using multiple hubs or docks, or a single hub or dock with a thunderbolt or USB4 capable display connected using USB-C to the dock), provided the hubs or docks have downstream ports supporting Thunderbolt or USB4.
  • For DP altmode, multiple hops can be supported in theory, but this depends on how the dock is constructed. I'm not sure if docks actually implement this (and if they do, if they properly document how it works, on which downstream ports this works, etc.).
  • HDMI (or DP++) outputs on docks typically just take a DP signal and add an active DP-to-HDMI converter. These converters have been seen to cause problems with some displays, so it might be better to avoid HDMI and use DP exclusively. [source]
  • How video is routed inside docks can apparently also be complex. For example the HP Thunderbolt dock G2 has a TB3 input, which contains two tunneled DP signals, one of which is split using MST into three outputs, two available on DP++ connectors and one available either as DP alt mode on a downstream USB-C connector, or converted into a VGA connector. The second DP signal is forwarded entirely to the downstream TB3 port. [source]
  • Since MacOS does not support MST, any docks that document support for > 2 display outputs on MacOS likely use DisplayLink, while docks that document support for > 2 display outputs on Windows/PC, but only 2 on MacOS likely use MST.
  • I also recall reading something (on this forum?) about a dock that uses DisplayLink for the first two monitor outputs, and DP alt mode only when three outputs are used, but I cannot seem to find the posting again (also, it seems like the decision of how to send display data to the dock is made by the OS, the dock can only decide or limit how it outputs the signals sent, I guess?).
  • Displays with a USB-C connector are usually internally similar to a USB-C or Thunderbolt dock supporting video input through DP-altmode and/or tunneling over Thunderbolt combined with USB over the remaining wires or also tunneled. Displays with integrated USB ports can sometimes be configured in 2-line USB3.0 + 2-line DP-alt-mode, or USB2.0-only mode, leaving all 4 lines available for a full-bandwidth 4-lane DP-alt-mode connection.

Framework laptop

  • The framework laptop supports USB4, and is intended to be TB4 certified, so I'm assuming that it already has the right hardware and software to support the entire TB4 featureset (though there might still be problems in either hardware or software, of course).
  • The Framework laptop GPU supports driving 4 simultaneous display outputs, including the builtin display (which can be disabled by closing the lid to allow driving 4 external monitors). [source]
  • All four USB-C ports can be used to output video, and multiple displays can be driven through one USB-C port with a DisplayPort MST hub in the dock or by separate DP-over-TB/USB4 tunnels. [source]
  • The USB-C ports are divided into two pairs, each pair sharing a single USB4 router / TB controller, and with two DP 4xHBR3 interfaces on each controller. This means at most two DP streams divided between the two ports in the pair (which includes both TB/USB4 tunnels and DP-alt-mode). [source, section 10, Framework uses UP3]
  • Using MST, multiple displays can be driven from a single DP stream, so this probably allows driving more than two displays from a single port (or pair of ports), but probably still limited to four displays in total (it is not entirely clear where in the Tiger Lake chip the MST-encoding happens).
  • The DisplayPort expansion card (and also the Tiger Lake TB controller) supports DisplayPort 1.4 (so not 2.0), I assume it is mostly a passive unit with DP altmode 1.0 then. [source]
  • Looking at the maximum uncompressed resolution supported by the framework DP expansion card (5120x3200 60Hz), according to the wikipedia resolution list that needs 22.18Gbps total bandwidth, so HBR3 (4x8.1Gbps-per-line), so it seems the expansion card supports all DP 1.4 bitrates (HBR3 is the highest available).
  • The expansion card also supports "8192x4320 60Hz uncompressed dual-port", which needs 49.65Gbps total bandwidth, which is indeed too much for HBR3, so I guess this means you'd need two DP expansion cards and two DP cables to a single monitor. [source] and [source]

That's it for now. Kudos if you made it this far! As always, feel free to leave a comment if this has been helpful to you, or if you have any suggestions or questions!

Update 2022-03-29: Added "Understanding tunneling and bandwidth limitations" section and some other small updates.

Update 2022-06-10: Mention boltctl as a tool for getting device info.

Comments
Rob Fisher wrote at 2022-03-26 10:46

This has dramatically improved my understanding of USB, which is even more complicated than I thought. Excellent research!

Matthijs Kooijman wrote at 2022-03-26 23:40

> This has dramatically improved my understanding of USB, which is even more complicated than I thought. Excellent research!

Good to hear! And then I did not even dive into the details of how USB itself works, with all its descriptors, packet types, transfer modes, and all other complex stuff that is USB (at least 1/2, I haven't looked at USB3 in that much detail yet...).

Dirk Ballay wrote at 2022-04-03 19:59

Hi there are 4 corrupted links with a slipped ")". Just search for ")]".

Matthijs Kooijman wrote at 2022-04-03 20:44

Ah, I copied the markdown from my original post on Discourse, and it seems my own markdown renderer is slightly different when there are closing parenthesis inside a URL. Urlencoding them fixed it. Thanks!

Jens wrote at 2022-04-23 11:08

Thank you for sharing! Which dock (hub?) did you end up buying?

Matthijs Kooijman wrote at 2022-04-23 15:21

Hey Jens, good question!

I ended up getting the Caldigit TS4, which is probably one of the most expensive ones, but I wanted to get at least TB4 (since USB4 has public specifications, and supports wake-from-sleep, unlike TB3) and this one has a good selection of ports, the host port on the back side (which I think ruled out 70% of the available products or so - I used this great list to make the initial selection).

See also this post for my initial experiences with that dock.

Lukas wrote at 2022-09-04 16:27

> It is a bit unclear to me how these signals are multiplexed. Apparently TB2 combines all four lines into a single full-duplex channel, suggesting that on TB1 there are two separate channels, but does that mean that one channel carries PCIe and one carries DP on TB1? Or each device connected is assigned to (a part of) either channel?

No. Each Thunderbolt controller contains a switch. Protocol adapters for PCIe and DP attach to the switch and translate between that protocol and the low-level Thunderbolt transport protocol. The protocols are tunneled transparently over the switching fabric and packets for different protocols may be transmitted on the same lane in parallel.

Someone wrote at 2022-09-04 16:42

> TB1/2 are backwards compatible with DP, so you can plug in a non-TB DP display into a TB1/2 port. I suspect these ports start out as DP ports and there is some negotiation over the aux channel to switch into TB mode, but I could not find any documentation about this.

When Apple integrated the very first Copper-based Thunderbolt controller, Light Ridge, into their products, they put a mux and redrivers on the mainboard which switches the plug's signals between the Thunderbolt controller and the DP-port on the GPU. They also added a custom microcontroller which snoops on the signal lines, autodetects whether the signals are DP or Thunderbolt, and drives the mux accordingly.

The next-generation chip Cactus Ridge (still Thunderbolt 1) integrated this functionality into the Thunderbolt controller. So the signals go from the DP plug to the Thunderbolt controller, and if it detects DP signals, it routes them through to the GPU via its DP-in pins.

(Source: I've worked on Linux Thunderbolt bringup on Macs and studied their schematics extensively.)

MikeNap wrote at 2022-09-04 20:20

Amazing write up. I spent so long trying to patch together an understanding of TB docks for displays and this has helped TREMENDOUSLY. Thanks again

Omar wrote at 2022-09-04 21:25

I just "Waybaked*" your page. It's such a gem. :) Thank you for your writeup Matthijs!

*(archive.org)

cheater wrote at 2022-09-04 23:01

Interesting read. After reading it, though, I am still none the wiser on a situation I experienced recently. I have a Moto G6 phone. I bought a USB C dock for it, a "Kapok 11-in-1-USB-C Laptop Dockingstation Dual HDMI Hub" on Amazon. It would charge, but the video out didn't work, and the Ethernet didn't work either. It did work for my Steam Deck, though. How come? I understand some USB C hubs work for android phones, some don't, and I don't know how that works. How does one find a dock that will work with Android?

Matthijs Kooijman wrote at 2022-09-05 18:22

@Lukas, you replied:

> No. Each Thunderbolt controller contains a switch. Protocol adapters for PCIe and DP attach to the switch and translate between that protocol and the low-level Thunderbolt transport protocol. The protocols are tunneled transparently over the switching fabric and packets for different protocols may be transmitted on the same lane in parallel.

I understand this, but the questions in my post that you are responding to were prompted because the wikipedia page on TB2 ( en.wikipedia.org/wiki/Thunderbolt(interface)#Thunderbolt2 ) says:

> The data-rate of 20 Gbit/s is made possible by joining the two existing 10 Gbit/s-channels, which does not change the maximum bandwidth, but makes using it more flexible.

Which is confusing to me. I could imagine this refers to the TB controller having one adapter that supports 20Gbps DP or PCIe instead of 2 adapters that support 10Gbps each, but TB2 only supports DP1.2 (so only 4.32Gbps) and PCIe bandwidths seem to be multiples of 8Gbps/GTps (but I do see PCIe 2.0 that has 5GTps, so maybe TB1 supported two PCIe 2.0 x2 ports for 10Gbps each and TB2 (also?) supports once PCIe 2.0 x4 port for 20Gbps?

Matthijs Kooijman wrote at 2022-09-05 18:30

@Someone, thanks for clarifying how DP vs TB1/2 negotiation worked, I'll edit my post accordingly.

@MikeNap, I feel your pain, great to hear my post has been helpful :-D

@Omar, cool!

@cheater, my first thought about video is that your dock might be of the DisplayLink (or some other video-over-USB) kind and your Android phone might not have the needed support/drivers. A quick google suggests this might need an app from the Play Store.

Or maybe the reverse is true and the dock only supports DP alt mode (over USB C) to get the display signal and your Android does not support this (but given it is a dual HDMI dock without thunderbolt, this seems unlikely, since the only way to get multiple display outputs over DP alt mode is to use MST and I think not many docks support that - though if the dock specs state that dual display output is not supported on Mac, that might be a hint you have an MST dock after all, since MacOS does not support MST).

As for ethernet, that might also be a driver issue?

Laurent Lyaudet wrote at 2022-09-07 19:05

Hello,

Thanks for this instructive post :)

I spotted some typos if it may help:

"can be combied" -> "can be combined"

"USB3.2 allows use of a all" -> "USB3.2 allows use of all"

"but a also" -> " but also"

"up to 20Gps-per-line" -> "up to 20Gbps-per-line"

"an USB3 10Gpbs" -> "an USB3 10Gbps"

"leaving ony" -> "leaving only"

"reverse-engineerd" -> "reverse-engineered"

"a single hub or dock with and a thunderbolt or USB4" -> "a single hub or dock with a thunderbolt or USB4"

"certifified" -> "certified"

Best regards, Laurent Lyaudet

Matthijs Kooijman wrote at 2022-09-07 19:31

@Laurent, thanks for your comments, I've fixed all of the typos :-D Especially "certifified" is nice, that should have been the correct word, sounds much nicer ;-p

Laurent Lyaudet wrote at 2022-09-08 15:41

Hello :)

Google does understand (incomplete URL because of SPAM protection):

/search?q=certifified+meme&source=lnms&tbm=isch&sa=X

But there is not yet an exact match for "certifified meme" ;P

Name:
URL:
Comment:

 
Comment can contain markdown formatting
 

 
16 comments -:- permalink -:- 19:50
Copyright by Matthijs Kooijman - most content WTFPL