Oracle has introduced the Oracle Collective Communications Library (OCCL), a network plugin designed to optimize distributed artificial intelligence training efficiency. By integrating with standard network-plugin interfaces from Nvidia and AMD, OCCL enables remote direct memory access over converged Ethernet version 2 (RoCEv2) and InfiniBand. The plugin monitors and distributes traffic across multiple paths to minimize congestion, and it automatically redirects traffic around failed links or network interface cards, restoring connectivity when the hardware is repaired. This functionality helps maintain efficient data transfer and reduces the risk of interrupted training sessions.
Internal benchmarks demonstrate improved bandwidth
Oracle’s internal testing, which utilized Nvidia GB300 systems with 16 to 128 nodes and four GPUs per node, showed significant bandwidth gains when using OCCL. For AllReduce operations, which aggregate data across worker GPUs, OCCL increased average NCCL bus bandwidth by up to 10%. Additionally, the tests recorded a 14% increase for AllToAll operations, where each worker node exchanges data with every other node. These results highlight the potential benefits of OCCL in optimizing network efficiency for distributed AI training.
In Oracle Cloud Infrastructure (OCI) GPU clusters, RDMA networks are often deployed with a multiplanar architecture, providing network capacity through multiple independent planes. This design helps create path diversity and isolate failures, but it requires collective traffic to utilize capacity across available planes and respond to congestion or failures without interrupting the training job. The OCCL network plugin addresses this challenge by placing networking intelligence for congestion control and link/NIC failure recovery in the collective-communication transport layer. As a result, AI frameworks can continue to use NCCL or RCCL, while the OCCL network plugin manages network paths for inter-node data transfers using the standard network-plugin interface.
OCCL is preinstalled in supported Oracle HPC images within OCI, eliminating the need for separate control planes, agents, or application-side services. This release helps address the challenges of large-scale AI training, where distributed clusters are often prone to congested network paths or failed links and NICs. Meanwhile, other tech giants, such as OpenAI, are also working to enable networks for AI workloads. OpenAI has developed the Ethernet-based Multipath Reliable Connection (MRC) transport protocol, created with support from AMD, Broadcom, Intel, Microsoft, and Nvidia, which is currently deployed across all of OpenAI’s largest supercomputers, including the Oracle-built Zettascale10.
Leave a Reply