API

@kernel function f(args) end

Takes a function definition and generates a Kernel constructor from it. The enclosed function is allowed to contain kernel language constructs. In order to call it the kernel has first to be specialized on the backend and then invoked on the arguments.

Kernel language

• @Const
• @index
• @groupsize
• @ndrange
• @localmem
• @private
• @uniform
• @synchronize
• @print

Kernel constructor

After defining a kernel function f, call f(backend[, workgroupsize[, ndrange]]) to obtain a Kernel specialized for that backend. Workgroup size and ndrange can be fixed at construction time (enabling size-specific compile-time optimizations and fewer runtime checks, at the cost of recompilation when the sizes change) or supplied at launch:

f(backend)                    # dynamic workgroup size and ndrange
f(backend, 64)                # static workgroup size of 64
f(backend, 64, 1024)          # static workgroup size and ndrange
f(backend, 64, (128, 128))    # multi-dimensional ndrange

Example

using KernelAbstractions

@kernel function vecadd(A, @Const(B))
    I = @index(Global)
    @inbounds A[I] += B[I]
end

dev = CPU()
A = ones(1024)
B = rand(1024)
vecadd(dev, 64)(A, B, ndrange=length(A))
synchronize(dev)

@kernel config function f(args) end

This allows for two different configurations:

1. cpu={true, false}: Disables code-generation of the CPU function. This relaxes semantics such that KernelAbstractions primitives can be used in non-kernel functions.
2. inbounds={false, true}: Enables a forced @inbounds macro around the function definition in the case the user is using too many @inbounds already in their kernel. Note that this can lead to incorrect results, crashes, etc and is fundamentally unsafe. Be careful!
3. unsafe_indices={false, true}: Disables the implicit validation of indices, users must avoid @index(Global).

• @context

@Const(A)

@Const is an argument annotiation that asserts that the memory reference by A is both not written to as part of the kernel and that it does not alias any other memory in the kernel.

@index

The @index macro can be used to give you the index of a workitem within a kernel function. It supports both the production of a linear index or a cartesian index. A cartesian index is a general N-dimensional index that is derived from the iteration space.

Index granularity

• Global: Used to access global memory.
• Group: The index of the workgroup.
• Local: The within workgroup index.

Index kind

• Linear: Produces an Int64 that can be used to linearly index into memory.
• Cartesian: Produces a CartesianIndex{N} that can be used to index into memory.
• NTuple: Produces a NTuple{N} that can be used to index into memory.

If the index kind is not provided it defaults to Linear, this is subject to change.

Examples

@index(Global, Linear)
@index(Global, Cartesian)
@index(Local, Cartesian)
@index(Group, Linear)
@index(Local, NTuple)
@index(Global)

@localmem T dims

Declare storage that is local to a workgroup.

@private T dims

Declare storage that is local to each item in the workgroup. This can be safely used across @synchronize statements. On a CPU, this will allocate additional implicit dimensions to ensure correct localization.

For storage that only persists between @synchronize statements, an MArray can be used instead.

See also @uniform.

@private mem = 1

Creates a private local of mem per item in the workgroup. This can be safely used across @synchronize statements.

@synchronize()

After a @synchronize statement all read and writes to global and local memory from each thread in the workgroup are visible in from all other threads in the workgroup.

@synchronize(cond)

After a @synchronize statement all read and writes to global and local memory from each thread in the workgroup are visible in from all other threads in the workgroup. cond is not allowed to have any visible sideffects.

Platform differences

• GPU: This synchronization will only occur if the cond evaluates.
• CPU: This synchronization will always occur.

@print(items...)

This is a unified print statement.

Platform differences

• GPU: This will reorganize the items to print via @cuprintf
• CPU: This will call print(items...)

@uniform expr

expr is evaluated outside the workitem scope. This is useful for variable declarations that span workitems, or are reused across @synchronize statements.

@groupsize()

Query the workgroupsize on the backend. This function returns a tuple corresponding to kernel configuration. In order to get the total size you can use prod(@groupsize()).

@ndrange()

Query the ndrange on the backend. This function returns a tuple corresponding to kernel configuration.

Backend

Abstract supertype for all KernelAbstractions backends.

Concrete backends (for example CUDABackend from CUDA.jl or CPU from KernelAbstractions) determine where arrays are allocated and where kernels execute. Use get_backend to obtain the backend for an array and allocate to create storage on a backend.

Example

backend = get_backend(A)
kernel = my_kernel(backend, 256)
kernel(A, ndrange=length(A))
synchronize(backend)

Abstract type for all GPU based KernelAbstractions backends.

CPU

Type alias for POCLBackend, the CPU execution backend.

Construct with CPU() (equivalent to POCLBackend()). Kernels run on the host via POCL/OpenCL using the same programming model as GPU backends, which is useful for debugging and for running kernel code without a GPU.

Example

A = ones(Float32, 1024)
mul2_kernel(CPU(), 64)(A, ndrange=length(A))
synchronize(CPU())

POCLBackend()

CPU backend that compiles kernels to OpenCL via POCL and executes them on the host. This is the concrete type behind the CPU alias.

get_backend(A::AbstractArray)::Backend

Get a Backend instance suitable for array A.

adapt(backend::Backend, x)

Convert x such that its array storage lives on backend. This is an extension of Adapt.jl, and lets code move data to a backend without knowing the backend's array type:

using Adapt
x = adapt(CUDABackend(), rand(Float32, 8))  # a CuArray
y = adapt(CPU(), x)                         # an Array again

Adapt.adapt_storage(::CUDABackend, x) = adapt(CuArray, x)

allocate(::Backend, Type, dims...; unified=false)::AbstractArray

Allocate a storage array appropriate for the computational backend. unified=true allocates an array using unified memory if the backend supports it and throws otherwise. Use supports_unified to determine whether it is supported by a backend.

zeros(::Backend, Type, dims...; unified=false)::AbstractArray

Allocate a storage array appropriate for the computational backend filled with zeros. unified=true allocates an array using unified memory if the backend supports it and throws otherwise.

ones(::Backend, Type, dims...; unified=false)::AbstractArray

Allocate a storage array appropriate for the computational backend filled with ones. unified=true allocates an array using unified memory if the backend supports it and throws otherwise.

copyto!(::Backend, dest::AbstractArray, src::AbstractArray)

Perform an asynchronous copyto! operation that is execution ordered with respect to the back-end.

For most users, Base.copyto! should suffice, performance a simple, synchronous copy. Only when you know you need asynchronicity w.r.t. the host, you should consider using this asynchronous version, which requires additional lifetime guarantees as documented below.

arr = zeros(64)
GC.@preserve arr begin
    copyto!(backend, arr, ...)
    # other operations
    synchronize(backend)
end

pagelock!(::Backend, dest::AbstractArray)::Union{Nothing, Missing}

Pagelock (pin) a host memory buffer for a backend device. This may be necessary for copyto! to perform asynchronously w.r.t to the host/

This function should return nothing; or missing if not implemented.

unsafe_free!(x::AbstractArray)

Release the memory of an array for reuse by future allocations and reduce pressure on the allocator. After releasing the memory of an array, it should no longer be accessed.

functional(::Backend)::Union{Bool, Missing}

Queries if the provided backend is functional. This may mean different things for different backends, but generally should mean that the necessary drivers and a compute device are available.

This function should return a Bool or missing if not implemented.

versioninfo(io::IO=stdout, backend::Backend)::Nothing

Print information about backend to io. It is up to the backends to determine what is relevant.

supports_unified(::Backend)::Bool

Returns whether unified memory arrays are supported by the backend.

supports_atomics(::Backend)::Bool

Returns whether @atomic operations are supported by the backend.

supports_float64(::Backend)::Bool

Returns whether Float64 values are supported by the backend.

synchronize(::Backend)

Synchronize the current backend: block the calling task until all work it has queued on backend has completed.

@spawn [threadpool] backend [device=id] expr

Run expr on a new Julia task, like Threads.@spawn, and return the Task. Use it in place of Threads.@spawn to launch kernels from a task. It guarantees that

• the task runs on the device that was active in the spawning task, or on device when that argument is given;
• the work the task queues on backend runs after the work the spawning task had queued on backend before calling @spawn;
• once wait(task) or fetch(task) returns, all work the task queued on backend has completed, so its results may be used from any task. fetch(task) returns the value of expr. If expr throws, the task is not synchronized: its queued work may still be running when the exception surfaces.

Everything else works as for Threads.@spawn: the optional threadpool argument (:default or :interactive) is forwarded, $x captures the value of x at spawn time, and an enclosing @sync waits for the task.

Example

A = KernelAbstractions.ones(backend, Float32, 1024)
mul2_kernel(backend, 64)(A, ndrange = length(A))   # queued by the current task

task = KernelAbstractions.@spawn backend begin
    mul2_kernel(backend, 64)(A, ndrange = length(A))  # ordered after the launch above
    sum(A)
end
fetch(task) == 4 * length(A)

Choosing the device

Backends keep the active device in task-local state, and Julia does not copy that state into a child task. A task started with plain Threads.@spawn therefore runs on the backend's default device, whichever device the spawning task was using. @spawn selects the device explicitly instead: by default the one active in the spawning task, or the one named by device, a 1-based index into 1:ndevices(backend):

task = KernelAbstractions.@spawn backend device=2 begin
    mul2_kernel(backend, 64)(B, ndrange = length(B))
end

The ordering guarantee holds across that switch: the task's work on device is still ordered after the work the spawning task had queued on its device. Backends that support more than one device implement this with a cross-device wait_event.

Backend authors: see the notes for backend implementations for the protocol behind these guarantees, and for how to support it without a full synchronize.

record_event(backend::Backend)

Capture the work the calling task has queued on backend's currently active device so far, and return a handle that wait_event can use to order later work after it, either from another task or from the same task after switching devices.

The handle is only meaningful for the pair record_event/wait_event; do not use it for anything else.

wait_event(backend::Backend, event)

Order the work the calling task subsequently queues on backend's currently active device after the work captured by event, which was returned by record_event.

The dependency is queue-ordered rather than task-ordered: it applies to the device that is active when wait_event is called, and a later device! leaves the newly selected device unordered with respect to event. Select the device first and wait afterwards:

event = record_event(backend)   # captures work on the current device
device!(backend, 2)
wait_event(backend, event)      # device 2 now waits for that work

device(backend::Backend)::Int

Return the 1-based index of the currently active device for backend.

ndevices(backend::Backend)::Int

Return the number of devices available to backend.

device!(backend::Backend, id::Int)::Nothing

Select the active device for backend. id is a 1-based device index and must satisfy 1 <= id <= ndevices(backend).

device! is not a synchronization point: work queued before the switch is not ordered with respect to work queued after it. To order across a switch, either synchronize beforehand, or bracket the switch with record_event and wait_event.

Example

device!(CUDABackend(), 2)  # use the second CUDA device

priority!(::Backend, prio::Symbol)::Nothing

Set the priority for the backend stream/queue. This is an optional feature that backends may or may not implement. If a backend shall support priorities it must accept :high, :normal, :low. Where :normal is the default.

Kernel{Backend, WorkgroupSize, NDRange, Func}

Host-side handle for a kernel specialized on a backend, workgroup size, and ndrange.

Kernels are created by calling a @kernel function on a backend, for example my_kernel(CUDABackend(), 256). The returned object is callable:

kernel = my_kernel(backend, 64)
kernel(A, B, ndrange=length(A))   # launch asynchronously
synchronize(backend)

Use workgroupsize, ndrange, and backend to inspect a kernel's static configuration.

(kernel::Kernel{<:NewBackend})(args...; ndrange=nothing, workgroupsize=nothing)

workgroupsize(kernel::Kernel)

Return the static workgroup size type parameter of kernel (StaticSize or DynamicSize).

ndrange(ctx)

Return the launch ndrange as a tuple.

backend(kernel::Kernel)

Return the Backend that kernel was constructed for.

KernelAbstractions.@device_code_lowered [kwargs...] ex

Evaluate ex and, for every device kernel compiled along the way, show the lowered IR.

This is GPUCompiler.@device_code_lowered, re-exposed for convenience; see its documentation for the supported keyword arguments. It applies to any GPUCompiler-based backend, so wrapping a kernel launch works on the CPU backend and on GPU backends alike. Note that ex is really evaluated: the kernels it launches are compiled and run.

Examples

KernelAbstractions.@device_code_lowered my_kernel(backend, 64)(A, ndrange=length(A))

KernelAbstractions.@device_code_typed [kwargs...] ex

Evaluate ex and, for every device kernel compiled along the way, show the type-inferred IR.

This is GPUCompiler.@device_code_typed, re-exposed for convenience; see its documentation for the supported keyword arguments. It applies to any GPUCompiler-based backend, so wrapping a kernel launch works on the CPU backend and on GPU backends alike. Note that ex is really evaluated: the kernels it launches are compiled and run.

Examples

KernelAbstractions.@device_code_typed my_kernel(backend, 64)(A, ndrange=length(A))

KernelAbstractions.@device_code_warntype [kwargs...] ex

Evaluate ex and, for every device kernel compiled along the way, show the type-inferred IR, highlighting type instabilities.

This is GPUCompiler.@device_code_warntype, re-exposed for convenience; see its documentation for the supported keyword arguments. It applies to any GPUCompiler-based backend, so wrapping a kernel launch works on the CPU backend and on GPU backends alike. Note that ex is really evaluated: the kernels it launches are compiled and run.

Examples

KernelAbstractions.@device_code_warntype my_kernel(backend, 64)(A, ndrange=length(A))

KernelAbstractions.@device_code_llvm [kwargs...] ex

Evaluate ex and, for every device kernel compiled along the way, show the generated LLVM IR.

This is GPUCompiler.@device_code_llvm, re-exposed for convenience; see its documentation for the supported keyword arguments. It applies to any GPUCompiler-based backend, so wrapping a kernel launch works on the CPU backend and on GPU backends alike. Note that ex is really evaluated: the kernels it launches are compiled and run.

Examples

KernelAbstractions.@device_code_llvm my_kernel(backend, 64)(A, ndrange=length(A))

KernelAbstractions.@device_code_native [kwargs...] ex

Evaluate ex and, for every device kernel compiled along the way, show the generated machine code.

This is GPUCompiler.@device_code_native, re-exposed for convenience; see its documentation for the supported keyword arguments. It applies to any GPUCompiler-based backend, so wrapping a kernel launch works on the CPU backend and on GPU backends alike. Note that ex is really evaluated: the kernels it launches are compiled and run.

Examples

KernelAbstractions.@device_code_native my_kernel(backend, 64)(A, ndrange=length(A))

KernelAbstractions.@device_code [dir=...] [...] ex

Evaluate ex and dump all forms of code generated for the device kernels it compiles to the directory dir, or to a temporary directory if none is given.

This is GPUCompiler.@device_code, re-exposed for convenience; see its documentation for the supported keyword arguments. Like the other @device_code_* macros it applies to any GPUCompiler-based backend, and really evaluates ex.

partition(kernel, ndrange, workgroupsize)

Partition the iteration space of kernel into workgroups.

Returns the blocked iteration space and whether dynamic bounds-checking is required for the last (possibly partial) workgroup. Primarily used by backend implementations and tests.

@context()

Access the hidden context object used by KernelAbstractions.

function f(@context, a)
    I = @index(Global, Linear)
    a[I]
end

@kernel cpu=false function my_kernel(a)
    f(@context, a)
end

argconvert(kernel::Kernel, arg)

Convert arg to the device-side representation expected by kernel's backend.

Backend implementations define methods for their array and scalar types. This is called automatically when a kernel is launched.

DynamicSize

Marker type indicating that a kernel's workgroup size or ndrange is chosen at launch time.

StaticSize{S}

Marker type encoding a compile-time workgroup size or ndrange as a tuple S. Each entry of S is an Int extent or, for an ndrange axis whose indices do not start at 1, a UnitRange{Int}.

NDRange

Encodes a blocked iteration space. The mapping field relates blocked indices to ndrange indices: nothing for the identity, or a StaticOffset/DynamicOffset for an ndrange whose indices do not start at 1.

Example

ndrange = NDRange{2, DynamicSize, DynamicSize}(CartesianIndices((256, 256)), CartesianIndices((32, 32)))
for block in ndrange
    for items in workitems(ndrange)
        I = expand(ndrange, block, items)
        checkbounds(Bool, A, I) || continue
        @inbounds A[I] = 2*A[I]
    end
end

StaticOffset{O}

Compile-time offset O::NTuple{N, Int} added to the indices produced by an NDRange.

DynamicOffset{N}

Runtime offset added to the indices produced by an NDRange.

extents(ndrange)

Number of indices along each axis of ndrange, given as a tuple of extents and/or ranges, a CartesianIndices, a single range, or an integer.

offsets(ndrange)

Offset of the first index along each axis of ndrange relative to 1.

linear_index(ndrange::CartesianIndices, I::CartesianIndex)

Column-major position of I within ndrange, counted from 1.