API Reference

This documents the public C ABI. Every function maps Vulkan concepts to columnar data operations. Opaque handles keep driver internals out of the public surface.

Core Types

enum VcDtype

Column data type enumeration. Corresponds to GPU-side uint / float types in GLSL shaders.

enumerator VC_DTYPE_U8 = 0

enumerator VC_DTYPE_U16 = 1

enumerator VC_DTYPE_U32 = 2

enumerator VC_DTYPE_U64 = 3

enumerator VC_DTYPE_I8 = 4

enumerator VC_DTYPE_I16 = 5

enumerator VC_DTYPE_I32 = 6

enumerator VC_DTYPE_I64 = 7

enumerator VC_DTYPE_F32 = 8

enumerator VC_DTYPE_F64 = 9

type VcContext: Opaque handle to a GPU context. Internally owns a VkInstance, VkPhysicalDevice, VkDevice, one compute VkQueue, a VkCommandPool, and a VmaAllocator.

type VcColumn: Opaque handle to a typed column buffer on the GPU. Backed by a VkBuffer allocated through VMA with VK_BUFFER_USAGE_STORAGE_BUFFER_BIT so compute shaders can read and write it directly.

Context Lifecycle

VcContext *vc_create_context(void)

Status: fully implemented.

Creates a Vulkan instance (VK_API_VERSION_1_3), selects a physical device (preferring AMD vendorID == 0x1002, falling back to any compute-queue GPU), creates a logical device with one compute queue, a VMA allocator, and a command pool with VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT.

In debug builds (_DEBUG defined), attempts to enable VK_LAYER_KHRONOS_validation and VK_EXT_DEBUG_UTILS_EXTENSION_NAME. Gracefully falls back if the layer is not installed.

Returns:: A new context, or NULL on failure.

void vc_destroy_context(VcContext *ctx)

Status: fully implemented.

Tears down the command pool, VMA allocator, logical device, and instance in reverse creation order. Passing NULL is a no-op.

Column Management

VcColumn *vc_create_column(VcContext *ctx, VcDtype dtype, uint64_t length, const void *host_data)

Status: fully implemented.

Allocates a VkBuffer through VMA with VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT (shader access + transfer source/target for host staging). Uses VMA_MEMORY_USAGE_AUTO so the driver picks device-local or host-visible memory as appropriate.

If host_data is provided, creates a transient staging buffer (TRANSFER_SRC, HOST_ACCESS_SEQUENTIAL_WRITE | CREATE_MAPPED), memcpy’s the data into it, issues vkCmdCopyBuffer from staging to the device-local buffer, inserts a TRANSFER_WRITE → SHADER_READ pipeline barrier, submits, and waits.

Edge case: zero-length columns allocate a minimal 4-byte buffer to keep Vulkan happy. Large no-data allocations (>64 MiB) also use the minimal allocation — the logical length is stored but the backing buffer is placeholder-sized.

Parameters:

ctx – GPU context.
dtype – Element data type.
length – Number of elements.
host_data – Host data to upload, or NULL to leave uninitialized.

Returns:

A new column, or NULL on failure.

void vc_destroy_column(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Calls vmaDestroyBuffer and frees the column struct. Passing NULL for either argument is a no-op.

int vc_read_column(VcContext *ctx, VcColumn *col, void *host_buffer, uint64_t host_buffer_size)

Status: fully implemented.

Reads GPU column data back to the host through a staging buffer:

Allocates a temporary staging buffer (TRANSFER_DST, HOST_ACCESS_RANDOM | CREATE_MAPPED).
Inserts a SHADER_WRITE → TRANSFER_READ barrier on the column buffer (ensures any prior compute writes are visible to the transfer engine).
Issues vkCmdCopyBuffer from the column buffer to the staging buffer.
Inserts a TRANSFER_WRITE → HOST_READ barrier on the staging buffer.
Submits, waits, memcpy’s from staging mapped memory to host_buffer.

Zero-length columns return success immediately.

Parameters:

ctx – GPU context.
col – Column to read.
host_buffer – Destination buffer on the host.
host_buffer_size – Size of the destination buffer in bytes.

Returns:

0 on success, -1 on error (e.g., buffer too small).

uint64_t vc_column_length(VcColumn *col): Status: fully implemented. Returns the number of elements. Returns 0 if col is NULL.

VcDtype vc_column_dtype(VcColumn *col): Status: fully implemented. Returns the element data type. Returns VC_DTYPE_U8 if col is NULL.

Null Support (Arrow Validity Bitmaps)

The library supports nullable columns through Arrow-compatible validity bitmaps. A validity bitmap is a packed array of uint32_t words, one bit per element, LSB-first per word. A set bit means the element is valid; a cleared bit means null. The bitmap length is ceil(N/32) words for a column with N elements.

VcColumn maps 1:1 to an Arrow PrimitiveArray: validity_buffer is Arrow buffers[0] and buffer is Arrow buffers[1]. This enables zero-copy wrappers in Polars, DataFusion, or PySpark bindings.

VcColumn *vc_create_column_nullable(VcContext *ctx, VcDtype dtype, uint64_t length, const void *host_data, const uint32_t *validity, uint64_t null_count)

Status: fully implemented.

Creates a nullable column. Behaves identically to vc_create_column() for the data buffer, then additionally allocates and uploads the validity bitmap if validity is non-NULL and null_count > 0.

Vulkan primitives: allocates a STORAGE_BUFFER | TRANSFER_DST buffer for the validity bitmap through VMA, copies the host-side bitmap via a transient staging buffer and vkCmdCopyBuffer, then inserts a TRANSFER_WRITE -> SHADER_READ barrier.

Edge case: if validity is NULL or null_count == 0, the column is created without a validity buffer (i.e., all elements are considered valid).

Parameters:

ctx – GPU context.
dtype – Element data type.
length – Number of elements.
host_data – Host data to upload, or NULL.
validity – Arrow-format validity bitmap (packed uint32), or NULL.
null_count – Number of null elements.

Returns:

A new nullable column, or NULL on failure.

int vc_column_has_validity(VcColumn *col)

Status: fully implemented.

Returns 1 if the column has a validity buffer, 0 otherwise. Returns 0 if col is NULL.

uint64_t vc_column_null_count(VcColumn *col)

Status: fully implemented.

Returns the null count set at column creation time. Note: operations that remove elements (vc_filter) do not recompute the null count; the value is preserved from the input column. A future compute-shader pass can recompute it from the output validity buffer.

Returns 0 if col is NULL.

int vc_read_column_validity(VcContext *ctx, VcColumn *col, void *host_buffer, uint64_t host_buffer_size, uint32_t *validity_out)

Status: fully implemented.

Reads both column data and validity bitmap back to the host. First calls vc_read_column() for the data, then performs a vkCmdCopyBuffer from the validity buffer to a transient staging buffer and memcpy’s the bitmap to validity_out.

If the column has no validity buffer, validity_out is left untouched. The caller must provide a validity_out buffer of at least ceil(col->length / 32) uint32_t words.

Parameters:

ctx – GPU context.
col – Column to read.
host_buffer – Destination buffer for data (passed to vc_read_column).
host_buffer_size – Size of the data destination buffer.
validity_out – Destination buffer for validity bitmap.

Returns:

0 on success, -1 on error.

Arrow C Data Interface

The library implements the Apache Arrow C Data Interface for zero-copy interop with Polars, DataFusion, Spark, DuckDB, Velox, and any other framework that speaks Arrow. VcColumn maps 1:1 to an Arrow PrimitiveArray (buffers[0] = validity, buffers[1] = data).

VcColumn *vc_import_arrow(VcContext *ctx, VcArrowArray *array, VcArrowSchema *schema)

Status: fully implemented.

Import an Arrow array into a VcColumn. The data and validity buffers are copied from host to GPU via staging buffers and vkCmdCopyBuffer. The Arrow structs must remain valid until the call returns.

Constraints: array->n_buffers >= 2, array->offset == 0 (sliced arrays not supported in v0.1). Only primitive numeric types are supported (C/S/I/L/c/s/i/l/f/g format strings).

VcArrowArray aa;
VcArrowSchema as;
// ... framework populates aa and as ...
VcColumn* col = vc_import_arrow(ctx, &aa, &as);
// ... run GPU operations on col ...
vc_destroy_column(ctx, col);

Parameters:

ctx – GPU context.
array – Arrow array to import.
schema – Arrow schema describing the data type.

Returns:

A new VcColumn on GPU, or NULL on failure.

int vc_export_arrow(VcContext *ctx, VcColumn *col, VcArrowArray *out_array, VcArrowSchema *out_schema)

Status: fully implemented.

Export a VcColumn to Arrow-compatible host buffers. Reads GPU data and validity (if present) back to host via staging buffers. Fills out_array and out_schema with buffer pointers and type metadata.

The caller must call out_array->release(out_array) and out_schema->release(out_schema) to free the exported host buffers when done. The structs themselves are not freed — the caller owns them (typically stack-allocated).

VcArrowArray aa;
VcArrowSchema as;
vc_export_arrow(ctx, col, &aa, &as);
// aa.private_data -> buffers[0]=validity, buffers[1]=data
// ... consume the Arrow array ...
aa.release(&aa);
as.release(&as);

Parameters:

ctx – GPU context.
col – Column to export.
out_array – [out] Populated Arrow array struct.
out_schema – [out] Populated Arrow schema struct.

Returns:

0 on success, -1 on error.

Integration Example: Polars in Python

import ctypes
import polars as pl
from pyarrow.cffi import ffi as arrow_ffi

lib = ctypes.CDLL("libvulkan_columnar.so")
ctx = lib.vc_create_context()

# Export Polars Series as Arrow C Data Interface structs.
series = pl.Series("vals", [1, 2, None, 4, 5], dtype=pl.UInt32)
arrow_array = series.to_arrow()
c_array_ptr, c_schema_ptr = arrow_ffi.export_array_and_schema(
    arrow_array, arrow_array.type)

# Import to GPU.
col = lib.vc_import_arrow(ctx, c_array_ptr, c_schema_ptr)

# Run GPU operations.
mask_col = lib.vc_create_column(ctx, 0, ...)  # U8 mask
filtered = lib.vc_filter(ctx, col, mask_col, byref(out_len))

# Export back.
result_array = ffi.new("VcArrowArray*")
result_schema = ffi.new("VcArrowSchema*")
lib.vc_export_arrow(ctx, filtered, result_array, result_schema)

# Import into Polars.
result_series = pl.Series.from_arrow(
    arrow_ffi.import_array_and_schema(result_array, result_schema))

lib.vc_destroy_context(ctx)

Operations

VcColumn *vc_filter(VcContext *ctx, VcColumn *data_col, VcColumn *mask_col, uint64_t *out_length)

Status: fully implemented.

Stream compaction (data[mask != 0]) via a 3-pass algorithm:

Pass 1 — GPU: bitmap + intra-workgroup Blelloch scan (filter_pass1.comp). 256 threads per workgroup. Each thread evaluates its mask element (0 or 1), then the workgroup runs a Blelloch exclusive prefix sum in 256 elements of shared memory (upsweep + downsweep). Output: per-element exclusive prefix (prefixes buffer) and per-workgroup total (totals buffer). Pipeline barrier ensures totals are host-readable after dispatch.

Pass 2 — CPU: workgroup-total exclusive scan. Maps the totals buffer, reads per-workgroup sums, computes an exclusive prefix sum serially, writes the per-workgroup base offsets to the offsets buffer. This is O(num_workgroups) — typically a few thousand iterations even for billion-element columns. The last running total is the output column length.

Pass 3 — GPU: scatter (filter_pass3.comp). Each thread reads its mask value and element index. If mask is non-zero, it computes scatter_pos = prefixes[gid] + offsets[wg] and writes data[gid] to output[scatter_pos].

Vulkan pipeline construction: one VkPipelineLayout with a 7-binding descriptor set layout (data, mask, prefixes, offsets, output, input validity, output validity) and a single push constant (element count + has_validity flag). Pass 1 uses bindings 0-3; pass 3 uses all 7 bindings. Both passes share the same layout.

Validity propagation: when the input column is nullable, pass3 scatters validity bits to the output using atomicOr. Each surviving element writes its validity bit to the corresponding output position — no contention since each output position is unique. The output validity buffer is zeroed via vkCmdFillBuffer before scatter.

Output: a new VcColumn whose buffer is the direct scatter output (device-local, STORAGE_BUFFER | TRANSFER_SRC). No host round-trip needed before chaining further GPU operations.

Edge cases

Zero-length input: returns NULL with *out_length = 0.
Mask type must be VC_DTYPE_U8.
Data and mask must have equal length.

Parameters:

ctx – GPU context.
data_col – Input data column.
mask_col – U8 mask column (non-zero = keep).
out_length – [out] number of surviving elements.

Returns:

A new filtered column, or NULL on failure.

VcColumn *vc_cast(VcContext *ctx, VcColumn *col, VcDtype target_dtype)

Status: fully implemented.

Element-wise type conversion between numeric data types. Supports all 10 VcDtype values as both source and target. Each element is processed independently — an element-wise parallel map.

Shader: type_cast.comp — a single GLSL compute shader that packs all type-pair logic with dispatch-time branching via push constants (src_dtype, dst_dtype). All data is read and written as raw uint arrays with custom pack/unpack helpers for sub-32-bit types. Intermediate conversions pass through float64 to preserve precision and avoid double rounding for integer-to-integer casts.

Vulkan pipeline construction: one VkPipelineLayout with a 2-binding descriptor set layout (source storage buffer at binding 0, destination at binding 1) and a single push constant block (element_count, src_dtype, dst_dtype).

Output: a new VcColumn with target_dtype, allocated via VMA with host access for readback. Data stays in VRAM for chaining further GPU operations.

Edge cases

Zero-length input: returns an empty column of the target dtype.
NaN input when casting to integer: clamped to 0.
Negative float to unsigned: clamped to 0.
Out-of-range float to integer: saturated to the target type’s min/max.
Identity cast (same source and target dtype): produces a copy with identical values.
Null context or column: returns NULL.

Parameters:

ctx – GPU context.
col – Input column.
target_dtype – Desired output data type.

Returns:

A new cast column, or NULL on failure.

int vc_reduce(VcContext *ctx, VcColumn *col, VcReduceOp op, uint32_t flags, VcScalar *out)

Status: fully implemented.

Reduce an entire column to a single scalar via a two-level parallel reduction: sum, min, or max.

Shader: reduce.comp — shared-memory binary tree reduction. 256 threads per workgroup. Each thread loops over stride elements, accumulating into a register. The tree reduces 256 values to 1 per WG. If num_workgroups > 1, a second dispatch reduces the intermediate buffer (using 64-bit promoted dtype since each intermediate value is stored as 2 uint32_t). Only the final 8-byte scalar crosses PCIe.

Vulkan pipeline construction: one VkPipelineLayout with a 3-binding descriptor set layout (input buffer, validity bitmap, output buffer) and a single push constant block (element count, op selector, dtype, skip-nulls flag, stride).

Return types by operation

Edge cases

Zero-length input: returns identity value (0 for sum, +inf or max integer for min).
VC_REDUCE_SKIP_NULLS: skips invalid elements. All-null input returns the identity value.
Single-workgroup columns: pass 2 is skipped; pass 1 output is the final scalar.
Multi-workgroup columns: pass 2 reduces the intermediate buffer with stride computed to cover all intermediate values.

Parameters:

ctx – GPU context.
col – Input column.
op – VC_REDUCE_SUM, VC_REDUCE_MIN, or VC_REDUCE_MAX.
flags – 0 or VC_REDUCE_SKIP_NULLS.
out – [out] Scalar result (dtype + union).

Returns:

0 on success, -1 on error.

Chained Execution

Operations are designed to chain without host round-trips. vc_filter returns a VcColumn* whose buffer is device-local with STORAGE_BUFFER | TRANSFER_SRC usage — immediately consumable by vc_cast or any future operation. The column handle carries metadata (length, dtype) that downstream operations read without GPU-to-host synchronization.

Chaining avoids the PCIe tax on intermediate results. Benchmark results on an RX 9070 (Debug build):

Scenario	64 KB	4 MB	64 MB
GPU chained (VRAM)	0.87 ms	3.35 ms	88.9 ms
GPU separate ops	—	5.70 ms	154 ms
CPU reference	1.38 ms	22.0 ms	331 ms

Scatter (filter) output feeds directly into an element-wise map (cast) without ever touching host RAM. The GPU beats CPU by 1.6–6.6x on chained operations.

If a vc_filter returns zero surviving elements, the returned column has length 0 and can still be passed to downstream operations (they will produce zero-length output). The caller is responsible for freeing every VcColumn* returned by an operation.

Utilities

uint32_t vc_dtype_size(VcDtype dtype)

Status: fully implemented.

Returns the size in bytes of the given data type (U8 = 1, U32 = 4, F64 = 8, etc.). Does not touch Vulkan — a pure lookup table.

VcColumn *vc_sort(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Sorts a column in ascending order using LSD radix sort.

Vulkan primitives:

Histogram pass: dispatches num_wgs workgroups (256 threads each). Each WG computes a private 257-bin shared-memory histogram (0-255 digit bins + bin 256 for nulls), then writes the per-WG histogram to a host-visible STORAGE_BUFFER. The host reads the per-WG histogram and computes per-WG exclusive-scan offsets for each digit.
Reorder pass: dispatches num_wgs workgroups. Each WG processes its slice of the column (wg * 256 * stride elements), computes per-digit local ranks via an O(256) shared-memory scan, and writes each element to per_wg_base_offset[wg][digit] + local_rank in the output buffer.
Ping-pong buffers: two STORAGE_BUFFER allocations swapped each digit pass. No host round-trip during the sort loop.
Sub-u32 types (U8, I8, U16, I16) are widened to U32/I32 via a pre-processing compute pass before the radix sort, then repacked to the original type at the end.

Algorithm: LSD (least significant digit) radix sort. 4 passes for 32-bit dtypes (U8–F32), 8 passes for 64-bit (U64–F64). Each pass sorts by one byte. Signed types XOR-flip MSB at extraction; floats convert to total order via sign-bit complement.

Null handling: nulls are counted in bin 256. During reorder, null elements are placed after all valid elements. The output null count is the sum of ‘bin 256’ counts across all WGs.

Edge cases:

Zero-length input: returns an empty column of the same dtype.
Null context or column: returns NULL.
Sub-u32 types: transparently widened and repacked.
Already-sorted / reverse-sorted / duplicates: handled correctly.
Single-element / prime-length columns: handled correctly.
Nullable columns with mixed valid/null elements: nulls placed at end, valid elements sorted ascending.

Parameters:

ctx – GPU context.
col – Input column.

Returns:

A sorted column (same length and dtype), or NULL on error.