Mellanox (NVIDIA Mellanox) MCX623106AN-CDAT Server Network Card in Practice | RDMA/RoCE Low-Latency Transport

Mellanox (NVIDIA Mellanox) MCX623106AN-CDAT Server Network Card in Practice | RDMA/RoCE Low-Latency Transport and Server Throughput Improvement

Background and Challenges: The Traditional TCP/IP Stack as a Hidden Bottleneck in Distributed Clusters

In the daily operations of a large-scale internet infrastructure team, thousands of servers carry distributed storage systems and real-time recommendation workloads. As east-west traffic keeps growing, the CPU overhead of the traditional TCP/IP stack has become increasingly prominent. During peak hours, a single node handles thousands of concurrent connections, and network protocol processing consumes nearly 30% of CPU resources, while cross-node read/write latency in the distributed storage system frequently exceeds the 200-microsecond threshold. More critically, in cross-rack NVMe over Fabrics testing, legacy network cards without hardware offload could never approach local flash performance.

This is where the Mellanox (NVIDIA Mellanox) MCX623106AN-CDAT enters the picture. As a dual-port 100GbE server network card from the NVIDIA ConnectX-6 Dx family, it was selected to replace the previous generation of adapters in the storage and compute tiers. The team's goal was clear: move data-path processing off the host CPU, cut end-to-end latency, and raise per-server throughput without adding racks.

Solution and Deployment: RDMA/RoCE with Hardware Offload

The deployment centered on RoCEv2 as the transport for storage and messaging traffic. As an MCX623106AN-CDAT Ethernet adapter card, the device offloads RDMA processing to hardware, so kernel bypass becomes practical for both the distributed storage layer and the in-house RPC framework. Each server was fitted with the card in a PCIe 4.0 x16 slot, with dual 100GbE ports split across two leaf switches for redundancy and aggregate 200Gb/s per node.

For virtualization hosts, SR-IOV was enabled so that virtual machines could access RDMA queues directly, avoiding the overhead of a virtual switch in the data path. The team also used the programmable pipeline for fine-grained congestion control and telemetry, which proved valuable when mixed storage and business traffic shared the same links. Before rollout, engineers reviewed the MCX623106AN-CDAT specifications and checked the MCX623106AN-CDAT datasheet to confirm firmware support and OS driver compatibility across the fleet.

• RoCEv2 transport with hardware RDMA offload for storage and RPC traffic
• Dual 100GbE ports across two leaf switches for redundancy and 200Gb/s aggregate
• SR-IOV enabled on virtualization hosts for direct VM access to RDMA queues
• Programmable pipeline used for congestion control and telemetry
• Standard PCIe 4.0 x16 form factor for straightforward integration

Because the team needed the MCX623106AN-CDAT compatible with existing server platforms from multiple vendors, a staged pilot was run on a small cluster first. The MCX623106AN-CDAT Ethernet adapter card solution was validated against the storage stack, the container network plugin, and the monitoring pipeline before broader rollout. Procurement also compared MCX623106AN-CDAT price and lead time across channels, and confirmed MCX623106AN-CDAT for sale availability for the second-phase expansion.

Results and Benefits: Lower Latency, Higher Throughput, Less CPU Load

After the migration, cross-node storage latency dropped from over 200 microseconds to roughly 60–80 microseconds in the pilot cluster, while CPU utilization attributable to network processing fell by about half. Per-server throughput in the distributed storage tier improved noticeably, since RDMA transfers no longer competed with application threads for CPU cycles. The operations team reported fewer latency spikes during peak hours, and virtualized hosts could support more active VMs per node without saturating the network stack.

Telemetry data from the programmable pipeline also shortened troubleshooting time. When a congestion event occurred, engineers could trace it to a specific queue pair and adjust traffic shaping policies rather than guessing at the cause. For architects planning the next expansion phase, the NVIDIA Mellanox MCX623106AN-CDAT has become the default reference design for 100G east-west links, and the team is now standardizing on it across storage, compute, and virtualization tiers.

Summary and Outlook

This practice shows that RDMA/RoCE is not only about raw bandwidth but about removing CPU overhead and stabilizing tail latency in distributed systems. The MCX623106AN-CDAT ConnectX adapter PCIe network card delivered measurable gains in latency, throughput, and host efficiency without changing rack layout. As the team prepares for 200G and beyond, the experience gained with this adapter provides a solid baseline for future network refresh planning.