Interface

To extend the above functionality to a new array type, you should use the types and implement the interfaces listed on this page. GPUArrays is designed around having two different array types to represent a GPU array: one that exists only on the host, and one that actually can be instantiated on the device (i.e. in kernels). Device functionality is then handled by KernelAbstractions.jl.

Host abstractions

You should provide an array type that builds on the AbstractGPUArray supertype, such as:

mutable struct CustomArray{T, N} <: AbstractGPUArray{T, N}
    data::DataRef{Vector{UInt8}}
    offset::Int
    dims::Dims{N}
    ...
end

This will allow your defined type (in this case JLArray) to use the GPUArrays interface where available. To be able to actually use the functionality that is defined for AbstractGPUArrays, you need to define the backend, like so:

import KernelAbstractions: Backend
struct CustomBackend <: KernelAbstractions.GPU
KernelAbstractions.get_backend(a::CA) where CA <: CustomArray = CustomBackend()

There are numerous examples of potential interfaces for GPUArrays, such as with JLArrays, CuArrays, and ROCArrays.

Alias detection

Without the hook below, all views of one buffer are treated as aliasing each other, so Base and GPUArrays copy one of them before operations that write to the other (e.g. copyto!, broadcasting, accumulate!). Report where an array's elements are stored to make that check precise:

GPUArrays.memory_location — Function

GPUArrays.memory_location(A::AbstractGPUArray) -> Union{Nothing, Tuple{UInt, Int}}

Where the elements of A are stored: a coordinate base that identifies the allocation holding them (normally its address), and the byte offset of the first element from it. GPUArrays defines Base.dataids, Base.mightalias and the check Base uses to compare two SubArrays of the same parent for every AbstractGPUArray; with this hook, they compare the byte ranges arrays cover instead of the allocations they share.

Two arrays are taken to share memory when their byte ranges, sizeof(A) bytes from base + offset, overlap. For that to be sound:

Every array derived from an allocation (a view, reshape or reinterpretation, which are built with derive) must report the same base, and a nonnegative offset into it. Memory wrapped from a pointer into the middle of an allocation (unsafe_wrap) reports a base of its own: it is still compared by byte range against arrays that implement this hook, but against wrapped arrays (e.g. a SubArray) and arrays of other types (e.g. an Array), only when both start at the same address.
Arrays that share memory must report it in the same coordinates. That holds for addresses in a single address space, but not for memory that is reachable at two addresses (such as a host buffer and its device mapping) or that is wrapped twice with separate identities (such as two handles for one buffer): the hook cannot express such aliases, so a back-end that creates them has to detect them itself.
Memory without an address may be identified by any other value that is unique to its allocation, such as a handle. Overlap with the coordinates of other memory only causes false positives.
For valid, live arrays it must not have side effects (taking stream ownership, synchronizing) and must not throw, as Base checks for aliasing in many operations, including copyto! and broadcasting.

The default, nothing, keeps alias detection at the level of allocations: arrays alias when they share the DataRef returned by GPUArrays.storage.

source

Caching Allocator

GPUArrays.@cached — Macro

@cached cache expr

Evaluate expr using allocations cache cache.

When GPU memory is allocated during the execution of expr, cache will first be checked. If no memory is available in the cache, a new allocation will be requested.

After the execution of expr, all allocations made under the scope of @cached will be cached within cache for future use. This is useful to avoid relying on GC to free GPU memory in time.

Once cache goes out scope, or when the user calls unsafe_free! on it, all cached allocations will be freed.

Example

In the following example, each iteration of the for-loop requires 8 GiB of GPU memory. Without caching those allocations, significant pressure would be put on the GC, resulting in high memory usage and latency. By using the allocator cache, the memory usage is stable:

cache = GPUArrays.AllocCache()
for i in 1:1000
    GPUArrays.@cached cache begin
        sin.(CUDA.rand(Float32, 1024^3))
    end
end

# optionally: free the memory now, instead of waiting for the GC to collect `cache`
GPUArrays.unsafe_free!(cache)

See @uncached.

source

GPUArrays.@uncached — Macro

@uncached expr

Evaluate expression expr without using the allocation. This is useful to call from within @cached to avoid caching some allocations, e.g., because they can be returned out of the @cached scope.

source