API reference

Except for Communicator, the functions and types of NCCL.jl are not exported: refer to them as NCCL.name.

Library

NCCL_jll.is_available — Function

NCCL.is_available() :: Bool

Is the NCCL library available?

source

NCCL.version — Function

NCCL.version() :: VersionNumber

Get the version of the current NCCL library.

source

Communicators

NCCL.Communicator — Type

NCCL.Communicator

An NCCL communicator: one rank of a group of ranks that perform collective and point-to-point operations together.

Communicators are not destroyed by the garbage collector. Destroying a communicator involves communication with the other ranks, which can block, or deadlock when the ranks don't do so at the same time, so it needs to happen at a well-defined point in the program: call NCCL.destroy (or NCCL.abort after an error) when you are done with it, or use the do-block form of the constructors. A communicator that is never destroyed keeps its resources until the process exits.

After a communicator has been destroyed or aborted, it is closed (see isopen), and using it throws an InvalidStateException. Its NCCL.rank, NCCL.nranks and NCCL.device remain available.

Communicators are always blocking; see NCCL.Config.

source

NCCL.Communicator — Method

NCCL.Communicator(nranks, rank, id::UniqueID;
                  device=CUDACore.device(), config=nothing) :: Communicator
NCCL.Communicator(f, nranks, rank, id::UniqueID; device, config)

Create the communicator for rank rank (0-based) of a communicator with nranks ranks, on device. All ranks have to call this with the same nranks and the same id, which one of them created with NCCL.UniqueID() and shared with the others, e.g. using MPI or a file. Each rank uses a different device. The call blocks until all ranks have joined, so the ranks have to be created concurrently: from different processes, or from different tasks running on different threads.

config can be an NCCL.Config to configure the communicator.

If initialization fails, the local communicator is aborted before the error is thrown. Inside an NCCL.group, initialization only happens when the group ends; if it fails there, the error is thrown by NCCL.group, and the communicator has to be aborted.

With the do-block form, the communicator is passed to f, and destroyed when f returns (see NCCL.destroy). If f throws, the communicator is aborted instead (see NCCL.abort) and the exception is rethrown. This only cleans up the local communicator: it does not recover from the error, and the other ranks need to abort their communicators as well, or they may block.

Examples

# on every rank, e.g. a process using a single GPUid = rank == 0 ? NCCL.UniqueID() : receive_id_from_rank0()Communicator(nranks, rank, id) do comm    NCCL.Allreduce!(buf, +, comm)end

External links

source

NCCL.Communicator — Method

NCCL.Communicator(mpicomm::MPI.Comm;
                  device=CUDACore.device(), config=nothing) :: Communicator
NCCL.Communicator(f, mpicomm::MPI.Comm; device, config)

Create a communicator with the same ranks as the MPI communicator mpicomm. Every rank of mpicomm has to call this, and becomes the rank of the new communicator with the same index: NCCL.rank(comm) == MPI.Comm_rank(mpicomm) and NCCL.nranks(comm) == MPI.Comm_size(mpicomm).

This method requires MPI.jl to be loaded (using MPI), and MPI to be initialized (MPI.Init()). Rank 0 of mpicomm creates the NCCL.UniqueID and broadcasts it to the other ranks over mpicomm, after which all ranks create the communicator as with NCCL.Communicator(nranks, rank, id), which also describes the keyword arguments and the do-block form.

Each rank uses device, the current device by default, and the ranks on one node have to use different devices. A common way to achieve this is to number the processes on each node with an MPI communicator of the processes that share memory, and to select a device based on that number before creating the communicator:

using MPI, CUDACore, NCCLMPI.Init()world_rank = MPI.Comm_rank(MPI.COMM_WORLD)node_comm = MPI.Comm_split_type(MPI.COMM_WORLD, MPI.COMM_TYPE_SHARED, world_rank)MPI.Comm_size(node_comm) <= ndevices() ||    error("more processes on this node than GPUs")CUDACore.device!(MPI.Comm_rank(node_comm))NCCL.Communicator(MPI.COMM_WORLD) do comm    NCCL.Allreduce!(buf, +, comm)end

Before initializing the communicator, the ranks check their arguments (e.g. that device is valid) and agree over mpicomm on whether all of them succeeded, so that an invalid argument on one rank makes all ranks throw, rather than leaving the others waiting for it. Errors during the initialization itself, e.g. a network failure, can still leave the other ranks blocked: as with any collective failure in an MPI program, handle them with a job-wide policy, e.g. by calling MPI.Abort.

source

NCCL.Communicators — Function

NCCL.Communicators(devices; config=nothing) :: Vector{Communicator}
NCCL.Communicators(f, devices; config=nothing)

Create a communicator with one rank per device in devices, all driven from the current task. Rank i-1 uses devices[i]; the devices must be distinct. devices can contain CuDevices or device ordinals.

config can be an NCCL.Config to configure the communicator. Without it, this uses ncclCommInitAll, otherwise one ncclCommInitRankConfig per device in a group.

Operations on several of these communicators have to be issued in an NCCL.group, or the first one blocks waiting for the others. Destroy the communicators together, with NCCL.destroy(comms), or use the do-block form, which destroys them when f returns, and aborts them if it throws (see NCCL.Communicator).

Examples

NCCL.Communicators(CUDACore.devices()) do comms    NCCL.group() do        for (comm, buf) in zip(comms, bufs)            NCCL.Allreduce!(buf, +, comm)        end    endend

External links

source

NCCL.UniqueID — Type

NCCL.UniqueID()
NCCL.UniqueID(bytes::AbstractVector{UInt8})

The identifier that all ranks of a communicator pass to Communicator, so that they find each other.

UniqueID() creates a new identifier (ncclGetUniqueId). This is done by one rank, which then has to send it to all other ranks by some other means, e.g. MPI, a socket, or a file. UniqueID is a plain bits type of 128 bytes: Vector{UInt8}(id) returns these bytes, and UniqueID(bytes) reconstructs the identifier from them. It can also be sent as-is with anything that handles bits types, e.g. Serialization or Distributed.

Creating an identifier also starts a bootstrap listener on the creating process, so the rank that creates it has to take part in the communicator.

Examples

if rank == 0    id = NCCL.UniqueID()    write(io, Vector{UInt8}(id))else    id = NCCL.UniqueID(read(io, 128))end

External links

ncclGetUniqueId

source

NCCL.Config — Type

NCCL.Config(; kwargs...)

Configuration for creating a communicator, passed as the config keyword argument of Communicator and Communicators. Every option defaults to nothing, which leaves the choice to NCCL (or the corresponding environment variable, which takes precedence over most options).

comm_name::String: a name for the communicator, used by NCCL in its logs and profiling.
net_name::String: the network module to use, e.g. "IB" or "Socket", or the name of an external network plugin (case-insensitive).
cga_cluster_size::Integer: the cooperative group array size of the kernels NCCL launches (0 to 8).
min_ctas::Integer, max_ctas::Integer: the minimal and maximal number of CTAs that NCCL should use for each kernel (1 to 32).
nvls_ctas::Integer: the total number of CTAs to use for NVLink SHARP kernels.
cta_policy::Integer: a bitmask of NCCL.LibNCCL.NCCL_CTA_POLICY_* flags.
split_share::Bool: whether communicators split from this one may share its resources.
traffic_class::Integer: the traffic class to use for network operations; its meaning depends on the network plugin.
collnet_enable::Bool: whether to use IB SHARP.
n_channels_per_net_peer::Integer: the number of network channels used for each peer in point-to-point communication.
max_p2p_peers::Integer: the maximal number of peers a rank communicates with at the same time using point-to-point communication.
graph_usage_mode::Integer: how the communicator is used with CUDA graphs: 0 (no graphs), 1 (one graph) or 2 (multiple graphs, or graphs mixed with regular launches; the default).

See the NCCL documentation for the details of each option. Options that the NCCL library in use doesn't know about are ignored.

Communicators are always created in blocking mode: nonblocking communicators, which require polling for the completion of every NCCL call, are not supported yet. NCCL lets the environment variable NCCL_COMM_BLOCKING override this, and also reads that variable from its configuration files (the file named by NCCL_CONF_FILE, or ~/.nccl.conf, and /etc/nccl.conf). Those files only set variables that are not in the environment yet, so when NCCL.jl is loaded, or creates a communicator, while NCCL_COMM_BLOCKING is unset, it sets it to 1, which NCCL's configuration files can then not override. Processes started afterwards inherit this setting. NCCL reads the variable only once per process, so this cannot take effect if another library already initialized NCCL in the process (e.g., by creating a communicator) before NCCL.jl was loaded. If NCCL_COMM_BLOCKING is set to a value that NCCL reads as 0, creating a communicator throws an error. NCCL only parses the leading number of the value (and older versions read a value without digits as 0), so this includes values like 0x0 or 0abc.

External links

ncclConfig_t

source

NCCL.rank — Function

NCCL.rank(comm::Communicator) :: Int

The 0-based index of this rank in the communicator.

source

NCCL.nranks — Function

NCCL.nranks(comm::Communicator) :: Int

The number of ranks in the communicator.

source

NCCL.device — Function

NCCL.device(comm::Communicator) :: CuDevice

The device of the communicator.

source

NCCL.split — Function

NCCL.split(comm::Communicator, color, key=NCCL.rank(comm);
           config=nothing) :: Union{Communicator,Nothing}
NCCL.split(comms::AbstractVector{Communicator}, colors, keys=NCCL.rank.(comms);
           config=nothing) :: Vector{Union{Communicator,Nothing}}
NCCL.split(f, comm, color, key; config)
NCCL.split(f, comms, colors, keys; config)

Split the communicator comm into new communicators: the ranks that pass the same color (a non-negative integer) form a new communicator, in which they are ordered by key (ranks with equal keys keep the order they have in comm). Ranks that pass nothing as color don't take part in any new communicator, and get nothing back. The new communicators use the same devices as comm.

This is a collective operation: every rank of comm has to call NCCL.split, even the ones that pass nothing, and the call blocks until all of them have. A task that drives several ranks of comm, e.g. the communicators returned by Communicators, has to split them together by passing vectors of communicators, colors and keys, which splits them in a single group; splitting them one by one deadlocks. NCCL.split cannot be called inside NCCL.group.

There must not be any outstanding operations on comm when it is split, or the split may deadlock: wait for the operations issued on comm to complete first (e.g. by synchronizing the streams they were issued on).

The new communicators use the configuration config (an NCCL.Config). Without it, they inherit the configuration of comm. If comm was created with split_share=true, the new communicators share resources with it: then comm and the new communicators must not be used concurrently, and aborting one of them can make the others unusable.

The new communicators are independent of comm, and need to be destroyed separately (see NCCL.destroy), before or after comm. With the do-block forms, they are passed to f, and destroyed when f returns, or aborted if it throws (see Communicator).

If the split fails, any new communicator that NCCL created is aborted before the error is thrown.

Examples

# split the ranks of `comm` in two halvesn = NCCL.nranks(comm)half = NCCL.split(comm, NCCL.rank(comm) < n ÷ 2 ? 0 : 1)# a communicator with only the even ranks of a cliqueNCCL.Communicators(CUDACore.devices()) do comms    NCCL.split(comms, [isodd(NCCL.rank(c)) ? nothing : 0 for c in comms]) do evens        ...    endend

External links

source

NCCL.destroy — Function

NCCL.destroy(comm::Communicator)
NCCL.destroy(comms::AbstractVector{Communicator})

Destroy the communicator comm (or all communicators in comms), after waiting for its outstanding operations to complete. Afterwards, the communicator is closed and can no longer be used. Destroying a closed communicator does nothing.

nothing, which NCCL.split returns for ranks that are not part of a new communicator, is ignored.

This first finalizes the communicator (ncclCommFinalize), which flushes its outstanding operations and synchronizes with the other ranks, and then frees its local resources (ncclCommDestroy). Finalization is an intra-node collective operation: all ranks on the same node need to destroy their communicator, or this call blocks. When a task drives several ranks of the same communicator, e.g. the communicators returned by Communicators, destroy them together by passing a vector: their finalization is then grouped, as NCCL requires. Destroying them one by one may deadlock.

If finalization fails, the communicator remains open; use NCCL.abort to free it.

This function must not be called inside NCCL.group.

External links

source

NCCL.abort — Function

NCCL.abort(comm::Communicator)
NCCL.abort(comms::AbstractVector{Communicator})

Abort the communicator comm (or all communicators in comms): cancel its uncompleted operations and free its resources. Afterwards, the communicator is closed and can no longer be used. Aborting a closed communicator does nothing. nothing is ignored, as with NCCL.destroy.

This is meant to release a communicator after an error. It does not wait for outstanding operations or synchronize with the other ranks, but NCCL expects every rank to abort its communicator, and peers that keep using theirs may block. Aborting a communicator is not guaranteed to be able to interrupt an operation that is blocked inside NCCL, because NCCL.jl only supports blocking communicators.

This function must not be called inside NCCL.group.

External links

source

Collective communication

NCCL.Allreduce! — Function

NCCL.Allreduce!(sendbuf, recvbuf, op, comm::Communicator;
                stream::Union{CuStream,Nothing} = nothing)
NCCL.Allreduce!(sendrecvbuf, op, comm::Communicator; stream)

Reduce sendbuf over all ranks using op (one of +, *, min, max, NCCL.avg, or an NCCL.PreMulSum), and write the result to recvbuf on all ranks. Returns recvbuf. Complex numbers can only be reduced with +, NCCL.avg and NCCL.PreMulSum, and Bools with max (logical or) and min (logical and).

sendbuf and recvbuf must have the same element type and length. With a single buffer sendrecvbuf, the operation is performed in place.

The buffers must be stored contiguously in memory of the device of comm: a CuArray, or a contiguous view or reshape of one. The operation is enqueued on stream, which defaults to the task's stream on that device, and is ordered like other work on that stream. Inside an NCCL.group, it is only enqueued when the group ends, and its buffers must not be used until then.

External links

ncclAllReduce

source

NCCL.Broadcast! — Function

NCCL.Broadcast!(sendbuf, recvbuf, comm::Communicator; root::Integer = 0,
                stream::Union{CuStream,Nothing} = nothing)
NCCL.Broadcast!(sendrecvbuf, comm::Communicator; root, stream)

Copy sendbuf on rank root to recvbuf on all ranks. Returns recvbuf.

sendbuf and recvbuf must have the same element type and length. sendbuf is only read on the root: other ranks can pass nothing instead. With a single buffer sendrecvbuf, the operation is performed in place.

External links

ncclBroadcast

source

NCCL.Reduce! — Function

NCCL.Reduce!(sendbuf, recvbuf, op, comm::Communicator; root::Integer = 0,
             stream::Union{CuStream,Nothing} = nothing)
NCCL.Reduce!(sendrecvbuf, op, comm::Communicator; root, stream)

Reduce sendbuf over all ranks using op (one of +, *, min, max, NCCL.avg, or an NCCL.PreMulSum), and write the result to recvbuf on rank root. Returns recvbuf. Complex numbers can only be reduced with +, NCCL.avg and NCCL.PreMulSum, and Bools with max (logical or) and min (logical and).

sendbuf and recvbuf must have the same element type and length. recvbuf is only written on the root: other ranks can pass nothing instead. With a single buffer sendrecvbuf, the operation is performed in place.

External links

ncclReduce

source

NCCL.Allgather! — Function

NCCL.Allgather!(sendbuf, recvbuf, comm::Communicator;
                stream::Union{CuStream,Nothing} = nothing)
NCCL.Allgather!(recvbuf, comm::Communicator; stream)

Concatenate sendbuf from each rank, in rank order, into recvbuf on all ranks. Returns recvbuf.

sendbuf and recvbuf must have the same element type, and recvbuf must have nranks(comm) times as many elements as sendbuf. The operation is performed in place when sendbuf is the part of recvbuf that receives the data of this rank. With only recvbuf, that part is sent: recvbuf then consists of nranks(comm) blocks of equal length, and on rank i (counting from 0) the i-th block holds the data to send.

External links

ncclAllGather

source

NCCL.ReduceScatter! — Function

NCCL.ReduceScatter!(sendbuf, recvbuf, op, comm::Communicator;
                    stream::Union{CuStream,Nothing} = nothing)
NCCL.ReduceScatter!(sendrecvbuf, op, comm::Communicator; stream) -> view

Reduce sendbuf over all ranks using op (one of +, *, min, max, NCCL.avg, or an NCCL.PreMulSum), and scatter the result over the ranks: recvbuf on rank i receives the i-th block of the result (counting from 0). Returns recvbuf. Complex numbers can only be reduced with +, NCCL.avg and NCCL.PreMulSum, and Bools with max (logical or) and min (logical and).

sendbuf and recvbuf must have the same element type, and sendbuf must have nranks(comm) times as many elements as recvbuf. The operation is performed in place when recvbuf is the block of sendbuf for this rank. With a single buffer sendrecvbuf, which consists of nranks(comm) blocks of equal length, the result for this rank is written to its own block, and a view of that block is returned; the other blocks are left in an undefined state.

External links

ncclReduceScatter

source

NCCL.Alltoall! — Function

NCCL.Alltoall!(sendbuf, recvbuf, comm::Communicator;
               stream::Union{CuStream,Nothing} = nothing)

Exchange blocks of data between all ranks: sendbuf and recvbuf are divided into nranks(comm) blocks of equal length, and the j-th block of sendbuf on rank i is copied to the i-th block of recvbuf on rank j (counting from 0). Returns recvbuf.

sendbuf and recvbuf must have the same element type and length, which must be a multiple of nranks(comm). The operation cannot be performed in place: the buffers must not overlap.

External links

ncclAlltoAll

source

NCCL.Gather! — Function

NCCL.Gather!(sendbuf, recvbuf, comm::Communicator; root::Integer = 0,
             stream::Union{CuStream,Nothing} = nothing)

Concatenate sendbuf from each rank, in rank order, into recvbuf on rank root. Returns recvbuf.

sendbuf and recvbuf must have the same element type, and recvbuf must have nranks(comm) times as many elements as sendbuf. recvbuf is only written on the root: other ranks can pass nothing instead. The operation is performed in place when sendbuf is the part of recvbuf that receives the data of the root, e.g. view(recvbuf, root*n+1:(root+1)*n) for buffers of n elements.

External links

ncclGather

source

NCCL.Scatter! — Function

NCCL.Scatter!(sendbuf, recvbuf, comm::Communicator; root::Integer = 0,
              stream::Union{CuStream,Nothing} = nothing)

Divide sendbuf on rank root into nranks(comm) blocks of equal length, and copy the i-th block to recvbuf on rank i (counting from 0). Returns recvbuf.

sendbuf and recvbuf must have the same element type, and sendbuf must have nranks(comm) times as many elements as recvbuf. sendbuf is only read on the root: other ranks can pass nothing instead. The operation is performed in place when recvbuf is the block of sendbuf for the root, e.g. view(sendbuf, root*n+1:(root+1)*n) for blocks of n elements.

External links

ncclScatter

source

Point-to-point communication

NCCL.Send — Function

NCCL.Send(sendbuf, comm::Communicator; dest::Integer,
          stream::Union{CuStream,Nothing} = nothing)

Send sendbuf to rank dest, which must call NCCL.Recv! with a buffer of the same element type and length.

The buffer must be stored contiguously in memory of the device of comm: a CuArray, or a contiguous view or reshape of one. The operation is enqueued on stream, which defaults to the task's stream on that device, and is ordered like other work on that stream. Inside an NCCL.group, it is only enqueued when the group ends, and its buffer must not be used until then.

External links

ncclSend

source

NCCL.Recv! — Function

NCCL.Recv!(recvbuf, comm::Communicator; source::Integer,
           stream::Union{CuStream,Nothing} = nothing)

Receive the data that rank source sends with NCCL.Send into recvbuf. Returns recvbuf.

External links

ncclRecv

source

Reduction operators

NCCL.avg — Constant

NCCL.avg

Perform an average operation, i.e. a sum across all ranks, divided by the number of ranks. For integers, the sum wraps around on overflow, and the division truncates towards zero.

source

NCCL.PreMulSum — Type

NCCL.PreMulSum(scalar::Real)

Reduction operator that multiplies the values of every rank by scalar before summing them: reducing x over the ranks results in scalar*x₀ + scalar*x₁ + .... It can be used with every reduction (Allreduce!, Reduce! and ReduceScatter!), e.g. NCCL.Allreduce!(buf, NCCL.PreMulSum(0.5f0), comm).

The multiplication happens in the element type of the buffers: scalar is converted to it with convert when the operation is issued. For floating-point buffers, it is rounded to the nearest representable value. For integer buffers, it has to be representable exactly (e.g. NCCL.PreMulSum(0.5) throws an InexactError with an Int32 buffer), and the products and sums wrap around on overflow, as Julia's integer arithmetic does. Complex buffers are scaled by a real scalar, which is converted to the type of their real and imaginary parts. Bool buffers cannot be reduced with PreMulSum.

Only real scalars are supported: NCCL multiplies the real and imaginary parts of complex numbers separately, so it cannot multiply them by a complex number.

External links

ncclRedOpCreatePreMulSum

source

Group calls

NCCL.group — Function

NCCL.group(f)

Evaluate f() as an NCCL group: operations issued by f are only submitted when it returns, which makes it possible to issue a collective on several communicators from a single task, or to combine multiple point-to-point operations without deadlocking.

Groups can be nested; operations are submitted when the outermost group ends.

Because operations in a group are only enqueued on their streams when the outermost group ends, the buffers passed to them must not be read, written, freed, or handed to another task or stream until then. NCCL.jl keeps these buffers, as well as the streams and communicators of the operations, alive until the group ends, so they do not need to be referenced elsewhere. Once the group has ended, the operations are enqueued on their streams, and are ordered like any other work on those streams: the usual CUDA.jl rules for using arrays from other tasks and streams apply.

NCCL keeps track of groups per OS thread, so the following rules apply:

The current task is pinned to its thread, and stays pinned after the group ends (the same happens when a task uses @async; tasks created by f may still be running afterwards and rely on it). All operations of the group must be issued from this task; operations issued from other tasks, e.g. Threads.@spawn, are not part of the group.
f must not yield to another task that uses NCCL on the same thread. If that happens, the other task's NCCL calls throw a ConcurrencyViolationError rather than silently joining the group.

If f throws, the group is still ended, and the exception thrown by f is rethrown, even if ending the group fails as well. What happens to the operations issued before the exception depends on whether NCCL recorded an error in its group state:

NCCL records the errors of operations issued in the group, e.g. when it rejects their arguments. Then the outermost group end discards all queued operations, and submits nothing.
Other exceptions don't affect the group, and ending it submits the operations that were issued. This includes Julia exceptions, e.g. from the argument checks of NCCL.jl, which run before NCCL is called, and NCCLErrors from NCCL calls that don't record their errors in the group, e.g. when creating an NCCL.UniqueID. If peers wait for operations that were never issued, this may deadlock.

Communicators are blocking, so ending the group can block, e.g. until the peers have issued their matching operations, before the exception reaches any handler in the caller. Recovering from such a failure, e.g. by having all ranks call NCCL.abort, therefore has to be arranged out of band for the whole job, and can't rely on the failing task. Aborting a blocking communicator is not guaranteed to interrupt a call that is blocked inside NCCL, so terminating the job is the only reliable way out of a hang.

Examples

NCCL.group() do    for (comm, sendbuf, recvbuf) in zip(comms, sendbufs, recvbufs)        NCCL.Allreduce!(sendbuf, recvbuf, +, comm)    endend

External links

source

Errors

NCCL.LibNCCL.NCCLError — Type

NCCLError(code::ncclResult_t, [msg::String])

An error returned by a NCCL function. code is the result code, and msg the error message NCCL reported for it (see ncclGetLastError), or an empty string if there is none.

NCCL keeps a single, process-wide last error message, which it only updates when it logs a warning and never clears. msg is therefore only set when that message changed since NCCL.jl last read it, so that an error which NCCL didn't log doesn't repeat the message of an earlier one. As a consequence, msg is empty when NCCL reports the same message twice in a row, and may describe an error raised concurrently by another thread. Set the environment variable NCCL_DEBUG=WARN to see all of NCCL's messages.

source