Memory Management

This page documents memory management in oneAPI.jl.

Memory Operations

Base.unsafe_copyto!(ctx::ZeContext, dev::ZeDevice, dst, src, N)

Low-level memory copy operation on the GPU. Copies N elements from src to dst using the specified context and device. Both src and dst can be either host pointers (Ptr) or device pointers (ZePtr).

unsafe_fill!(ctx::ZeContext, dev::ZeDevice, ptr, pattern, N)

Low-level memory fill operation on the GPU. Fills N elements at ptr with the given pattern using the specified context and device.

Memory Types

oneAPI supports three types of memory through Unified Shared Memory (USM):

Device Memory (Default)

Fastest GPU access, not directly accessible from CPU.

# Create array in device memory (default)a = oneArray{Float32}(undef, 1000)@assert is_device(a)# Or explicitly specifyb = oneArray{Float32,1,oneL0.DeviceBuffer}(undef, 1000)

Advantages:

• Fastest GPU access
• Best for compute-intensive operations

Disadvantages:

• Cannot directly access from CPU
• Requires explicit copy to/from CPU

Use when: Data stays on GPU for multiple operations

Shared Memory

Accessible from both CPU and GPU with automatic migration.

# Create array in shared memorya = oneArray{Float32,1,oneL0.SharedBuffer}(undef, 1000)@assert is_shared(a)# Can access from CPUa[1] = 42.0f0  # Automatic migration to CPUprintln(a[1])  # Read from CPU# Can use in GPU kernels@oneapi groups=1 items=1000 kernel(a)  # Automatic migration to GPU

Advantages:

• Accessible from both CPU and GPU
• Unified virtual addressing
• Automatic migration

Disadvantages:

• Migration overhead
• Slower than device memory for pure GPU work

Use when: Frequent CPU-GPU data exchange needed

Host Memory

CPU memory that's pinned and visible to GPU.

# Create array in host memorya = oneArray{Float32,1,oneL0.HostBuffer}(undef, 1000)@assert is_host(a)# Direct CPU accessa[1] = 42.0f0# Can be used by GPU (but slower than device memory)@oneapi groups=1 items=1000 kernel(a)

Advantages:

• Direct CPU access
• Pinned memory (faster PCIe transfers)
• Good for staging

Disadvantages:

• Slower GPU access than device memory
• Uses pinned system memory (limited resource)

Use when: Staging data for transfer, or CPU needs to write while GPU reads

Memory Type Comparison

Memory Allocation and Deallocation

Automatic Management

Julia's garbage collector automatically manages oneArray memory:

function allocate_and_compute()    a = oneArray(rand(Float32, 1000))    b = oneArray(rand(Float32, 1000))    c = a .+ b    return Array(c)  # Only c is copied back    # a and b will be garbage collectedendresult = allocate_and_compute()# GPU memory for a and b is freed eventually

Manual Garbage Collection

Force garbage collection to free GPU memory:

# Allocate large arraysa = oneArray(rand(Float32, 10_000_000))b = oneArray(rand(Float32, 10_000_000))# Clear referencesa = nothingb = nothing# Force GC to reclaim GPU memoryGC.gc()

Explicit Freeing

Immediately free GPU memory (use with caution):

a = oneArray(rand(Float32, 1000))# ... use a ...# Explicitly free (dangerous if still in use!)unsafe_free!(a)# a is now invalid - do not use!

Do-Block Pattern

Use do-blocks for automatic cleanup:

result = oneArray{Float32}(1000) do temp    # temp is automatically freed when block exits    temp .= 1.0f0    sum(temp)  # Result is returnedend

Memory Pooling

oneAPI.jl uses memory pooling to reduce allocation overhead:

using oneAPI# Allocations are pooledfor i in 1:100    a = oneArray(rand(Float32, 1000))    # ... use a ...    # Memory is returned to pool, not freedend

The pool automatically manages memory reuse, reducing allocation costs.

Checking Memory Usage

Query GPU memory info:

using oneAPI.oneL0dev = device()props = memory_properties(dev)for prop in props    println("Memory size: ", prop.totalSize ÷ (1024^3), " GB")end

Out of Memory Errors

If you encounter out-of-memory errors:

1. Reduce Batch Size

# Instead of processing all at onceresult = process(oneArray(huge_data))# Process in smaller batchesfor batch in batches(huge_data, size=1000)    result = process(oneArray(batch))    # Process result...end

2. Free Unused Arrays

a = oneArray(rand(Float32, 1_000_000))b = compute(a)# If 'a' is no longer neededunsafe_free!(a)# Continue with 'b'result = process(b)

3. Use Shared or Host Memory

# Instead of device memorya = oneArray{Float32}(undef, huge_size)# Use shared memory (can swap to system RAM)a = oneArray{Float32,1,oneL0.SharedBuffer}(undef, huge_size)

4. Force Garbage Collection

# After freeing referenceslarge_array = nothingGC.gc()  # Immediately reclaim GPU memory

5. Use Multiple Devices

# Distribute work across devicesfor (i, dev_id) in enumerate(1:length(devices()))    Threads.@spawn begin        device!(dev_id)        partition = data_partitions[i]        a = oneArray(partition)        result = compute(a)        # ...    endend

Low-Level Memory Operations

For advanced users, oneL0 provides direct memory management:

using oneAPI.oneL0ctx = context()dev = device()# Allocate device memoryptr = device_alloc(ctx, dev, 1024, 8)  # 1024 bytes, 8-byte aligned# Copy datadata = rand(Float32, 256)GC.@preserve data begin    unsafe_copyto!(ctx, dev, ptr, pointer(data), 256)end# Free memoryfree(ctx, ptr)

Memory Advise and Prefetch

Hint to the runtime about memory usage (shared memory only):

using oneAPI.oneL0a = oneArray{Float32,1,oneL0.SharedBuffer}(undef, 1000)# Advise that this will be read-only on the device# (Implementation depends on Level Zero driver support)# Prefetch to devicectx = context()dev = device()stream = global_stream(ctx, dev)# appended to the task's stream, in order with its kernels and copiesexecute!(stream) do list    append_prefetch!(list, pointer(a), sizeof(a))end

Best Practices

1. Use device memory by default for best GPU performance
2. Use shared memory when you need CPU access without explicit copies
3. Use host memory for staging data or when CPU writes frequently
4. Let GC handle cleanup unless you have specific memory pressure
5. Reuse allocations within loops when possible
6. Profile memory usage to identify bottlenecks
7. Be cautious with unsafe_free! - use only when you're certain it's safe

Example: Efficient Memory Usage

using oneAPIfunction efficient_pipeline(data_batches)    # Allocate output buffer once    result = oneArray{Float32}(undef, 1000)    results = Float32[]    for batch in data_batches        # Reuse input buffer by copying        input = oneArray(batch)        # Compute in-place when possible        @oneapi groups=4 items=250 process_kernel!(result, input)        # Copy result back        push!(results, Array(result)...)        # Input is freed when loop continues    end    return resultsend