API Reference ============= This documents the public C ABI. Every function maps Vulkan concepts to columnar data operations. Opaque handles keep driver internals out of the public surface. ---- Core Types ---------- .. c:enum:: VcDtype Column data type enumeration. Corresponds to GPU-side ``uint`` / ``float`` types in GLSL shaders. .. c:enumerator:: VC_DTYPE_U8 = 0 .. c:enumerator:: VC_DTYPE_U16 = 1 .. c:enumerator:: VC_DTYPE_U32 = 2 .. c:enumerator:: VC_DTYPE_U64 = 3 .. c:enumerator:: VC_DTYPE_I8 = 4 .. c:enumerator:: VC_DTYPE_I16 = 5 .. c:enumerator:: VC_DTYPE_I32 = 6 .. c:enumerator:: VC_DTYPE_I64 = 7 .. c:enumerator:: VC_DTYPE_F32 = 8 .. c:enumerator:: VC_DTYPE_F64 = 9 .. c:type:: VcContext Opaque handle to a GPU context. Internally owns a :c:type:`VkInstance`, :c:type:`VkPhysicalDevice`, :c:type:`VkDevice`, one compute :c:type:`VkQueue`, a :c:type:`VkCommandPool`, and a :c:type:`VmaAllocator`. .. c:type:: VcColumn Opaque handle to a typed column buffer on the GPU. Backed by a :c:type:`VkBuffer` allocated through VMA with ``VK_BUFFER_USAGE_STORAGE_BUFFER_BIT`` so compute shaders can read and write it directly. ---- Context Lifecycle ----------------- .. c:function:: VcContext* vc_create_context(void) **Status:** fully implemented. Creates a Vulkan instance (``VK_API_VERSION_1_3``), selects a physical device (preferring AMD ``vendorID == 0x1002``, falling back to any compute-queue GPU), creates a logical device with one compute queue, a VMA allocator, and a command pool with ``VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT``. In debug builds (``_DEBUG`` defined), attempts to enable ``VK_LAYER_KHRONOS_validation`` and ``VK_EXT_DEBUG_UTILS_EXTENSION_NAME``. Gracefully falls back if the layer is not installed. :returns: A new context, or ``NULL`` on failure. .. c:function:: void vc_destroy_context(VcContext* ctx) **Status:** fully implemented. Tears down the command pool, VMA allocator, logical device, and instance in reverse creation order. Passing ``NULL`` is a no-op. ---- Column Management ----------------- .. c:function:: VcColumn* vc_create_column(VcContext* ctx, VcDtype dtype, uint64_t length, const void* host_data) **Status:** fully implemented. Allocates a :c:type:`VkBuffer` through VMA with ``VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT`` (shader access + transfer source/target for host staging). Uses ``VMA_MEMORY_USAGE_AUTO`` so the driver picks device-local or host-visible memory as appropriate. If ``host_data`` is provided, creates a transient staging buffer (``TRANSFER_SRC``, ``HOST_ACCESS_SEQUENTIAL_WRITE | CREATE_MAPPED``), memcpy's the data into it, issues ``vkCmdCopyBuffer`` from staging to the device-local buffer, inserts a ``TRANSFER_WRITE → SHADER_READ`` pipeline barrier, submits, and waits. Edge case: zero-length columns allocate a minimal 4-byte buffer to keep Vulkan happy. Large no-data allocations (>64 MiB) also use the minimal allocation — the logical length is stored but the backing buffer is placeholder-sized. :param ctx: GPU context. :param dtype: Element data type. :param length: Number of elements. :param host_data: Host data to upload, or ``NULL`` to leave uninitialized. :returns: A new column, or ``NULL`` on failure. .. c:function:: void vc_destroy_column(VcContext* ctx, VcColumn* col) **Status:** fully implemented. Calls ``vmaDestroyBuffer`` and frees the column struct. Passing ``NULL`` for either argument is a no-op. .. c:function:: int vc_read_column(VcContext* ctx, VcColumn* col, void* host_buffer, uint64_t host_buffer_size) **Status:** fully implemented. Reads GPU column data back to the host through a staging buffer: 1. Allocates a temporary staging buffer (``TRANSFER_DST``, ``HOST_ACCESS_RANDOM | CREATE_MAPPED``). 2. Inserts a ``SHADER_WRITE → TRANSFER_READ`` barrier on the column buffer (ensures any prior compute writes are visible to the transfer engine). 3. Issues ``vkCmdCopyBuffer`` from the column buffer to the staging buffer. 4. Inserts a ``TRANSFER_WRITE → HOST_READ`` barrier on the staging buffer. 5. Submits, waits, memcpy's from staging mapped memory to ``host_buffer``. Zero-length columns return success immediately. :param ctx: GPU context. :param col: Column to read. :param host_buffer: Destination buffer on the host. :param host_buffer_size: Size of the destination buffer in bytes. :returns: ``0`` on success, ``-1`` on error (e.g., buffer too small). .. c:function:: uint64_t vc_column_length(VcColumn* col) **Status:** fully implemented. Returns the number of elements. Returns ``0`` if ``col`` is ``NULL``. .. c:function:: VcDtype vc_column_dtype(VcColumn* col) **Status:** fully implemented. Returns the element data type. Returns ``VC_DTYPE_U8`` if ``col`` is ``NULL``. ---- Null Support (Arrow Validity Bitmaps) -------------------------------------- The library supports nullable columns through Arrow-compatible validity bitmaps. A validity bitmap is a packed array of ``uint32_t`` words, one bit per element, LSB-first per word. A set bit means the element is valid; a cleared bit means null. The bitmap length is ``ceil(N/32)`` words for a column with ``N`` elements. ``VcColumn`` maps 1:1 to an Arrow ``PrimitiveArray``: ``validity_buffer`` is Arrow buffers[0] and ``buffer`` is Arrow buffers[1]. This enables zero-copy wrappers in Polars, DataFusion, or PySpark bindings. .. c:function:: VcColumn* vc_create_column_nullable(VcContext* ctx, VcDtype dtype, uint64_t length, const void* host_data, const uint32_t* validity, uint64_t null_count) **Status:** fully implemented. Creates a nullable column. Behaves identically to :c:func:`vc_create_column` for the data buffer, then additionally allocates and uploads the validity bitmap if ``validity`` is non-NULL and ``null_count > 0``. **Vulkan primitives:** allocates a ``STORAGE_BUFFER | TRANSFER_DST`` buffer for the validity bitmap through VMA, copies the host-side bitmap via a transient staging buffer and ``vkCmdCopyBuffer``, then inserts a ``TRANSFER_WRITE -> SHADER_READ`` barrier. Edge case: if ``validity`` is ``NULL`` or ``null_count == 0``, the column is created without a validity buffer (i.e., all elements are considered valid). :param ctx: GPU context. :param dtype: Element data type. :param length: Number of elements. :param host_data: Host data to upload, or ``NULL``. :param validity: Arrow-format validity bitmap (packed uint32), or ``NULL``. :param null_count: Number of null elements. :returns: A new nullable column, or ``NULL`` on failure. .. c:function:: int vc_column_has_validity(VcColumn* col) **Status:** fully implemented. Returns ``1`` if the column has a validity buffer, ``0`` otherwise. Returns ``0`` if ``col`` is ``NULL``. .. c:function:: uint64_t vc_column_null_count(VcColumn* col) **Status:** fully implemented. Returns the null count set at column creation time. Note: operations that remove elements (``vc_filter``) do not recompute the null count; the value is preserved from the input column. A future compute-shader pass can recompute it from the output validity buffer. Returns ``0`` if ``col`` is ``NULL``. .. c:function:: int vc_read_column_validity(VcContext* ctx, VcColumn* col, void* host_buffer, uint64_t host_buffer_size, uint32_t* validity_out) **Status:** fully implemented. Reads both column data and validity bitmap back to the host. First calls :c:func:`vc_read_column` for the data, then performs a ``vkCmdCopyBuffer`` from the validity buffer to a transient staging buffer and memcpy's the bitmap to ``validity_out``. If the column has no validity buffer, ``validity_out`` is left untouched. The caller must provide a ``validity_out`` buffer of at least ``ceil(col->length / 32)`` ``uint32_t`` words. :param ctx: GPU context. :param col: Column to read. :param host_buffer: Destination buffer for data (passed to ``vc_read_column``). :param host_buffer_size: Size of the data destination buffer. :param validity_out: Destination buffer for validity bitmap. :returns: ``0`` on success, ``-1`` on error. ---- Arrow C Data Interface ----------------------- The library implements the `Apache Arrow C Data Interface `_ for zero-copy interop with Polars, DataFusion, Spark, DuckDB, Velox, and any other framework that speaks Arrow. ``VcColumn`` maps 1:1 to an Arrow ``PrimitiveArray`` (buffers[0] = validity, buffers[1] = data). .. c:function:: VcColumn* vc_import_arrow(VcContext* ctx, VcArrowArray* array, VcArrowSchema* schema) **Status:** fully implemented. Import an Arrow array into a ``VcColumn``. The data and validity buffers are copied from host to GPU via staging buffers and ``vkCmdCopyBuffer``. The Arrow structs must remain valid until the call returns. Constraints: ``array->n_buffers >= 2``, ``array->offset == 0`` (sliced arrays not supported in v0.1). Only primitive numeric types are supported (``C/S/I/L/c/s/i/l/f/g`` format strings). .. code-block:: c VcArrowArray aa; VcArrowSchema as; // ... framework populates aa and as ... VcColumn* col = vc_import_arrow(ctx, &aa, &as); // ... run GPU operations on col ... vc_destroy_column(ctx, col); :param ctx: GPU context. :param array: Arrow array to import. :param schema: Arrow schema describing the data type. :returns: A new ``VcColumn`` on GPU, or ``NULL`` on failure. .. c:function:: int vc_export_arrow(VcContext* ctx, VcColumn* col, VcArrowArray* out_array, VcArrowSchema* out_schema) **Status:** fully implemented. Export a ``VcColumn`` to Arrow-compatible host buffers. Reads GPU data and validity (if present) back to host via staging buffers. Fills ``out_array`` and ``out_schema`` with buffer pointers and type metadata. The caller must call ``out_array->release(out_array)`` and ``out_schema->release(out_schema)`` to free the exported host buffers when done. The structs themselves are not freed — the caller owns them (typically stack-allocated). .. code-block:: c VcArrowArray aa; VcArrowSchema as; vc_export_arrow(ctx, col, &aa, &as); // aa.private_data -> buffers[0]=validity, buffers[1]=data // ... consume the Arrow array ... aa.release(&aa); as.release(&as); :param ctx: GPU context. :param col: Column to export. :param out_array: [out] Populated Arrow array struct. :param out_schema: [out] Populated Arrow schema struct. :returns: ``0`` on success, ``-1`` on error. Integration Example: Polars in Python ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python import ctypes import polars as pl from pyarrow.cffi import ffi as arrow_ffi lib = ctypes.CDLL("libvulkan_columnar.so") ctx = lib.vc_create_context() # Export Polars Series as Arrow C Data Interface structs. series = pl.Series("vals", [1, 2, None, 4, 5], dtype=pl.UInt32) arrow_array = series.to_arrow() c_array_ptr, c_schema_ptr = arrow_ffi.export_array_and_schema( arrow_array, arrow_array.type) # Import to GPU. col = lib.vc_import_arrow(ctx, c_array_ptr, c_schema_ptr) # Run GPU operations. mask_col = lib.vc_create_column(ctx, 0, ...) # U8 mask filtered = lib.vc_filter(ctx, col, mask_col, byref(out_len)) # Export back. result_array = ffi.new("VcArrowArray*") result_schema = ffi.new("VcArrowSchema*") lib.vc_export_arrow(ctx, filtered, result_array, result_schema) # Import into Polars. result_series = pl.Series.from_arrow( arrow_ffi.import_array_and_schema(result_array, result_schema)) lib.vc_destroy_context(ctx) ---- Operations ---------- .. c:function:: VcColumn* vc_filter(VcContext* ctx, VcColumn* data_col, VcColumn* mask_col, uint64_t* out_length) **Status:** fully implemented. Stream compaction (``data[mask != 0]``) via a 3-pass algorithm: **Pass 1 — GPU: bitmap + intra-workgroup Blelloch scan** (``filter_pass1.comp``). 256 threads per workgroup. Each thread evaluates its mask element (``0`` or ``1``), then the workgroup runs a Blelloch exclusive prefix sum in 256 elements of shared memory (upsweep + downsweep). Output: per-element exclusive prefix (``prefixes`` buffer) and per-workgroup total (``totals`` buffer). Pipeline barrier ensures totals are host-readable after dispatch. **Pass 2 — CPU: workgroup-total exclusive scan.** Maps the ``totals`` buffer, reads per-workgroup sums, computes an exclusive prefix sum serially, writes the per-workgroup base offsets to the ``offsets`` buffer. This is ``O(num_workgroups)`` — typically a few thousand iterations even for billion-element columns. The last running total is the output column length. **Pass 3 — GPU: scatter** (``filter_pass3.comp``). Each thread reads its mask value and element index. If mask is non-zero, it computes ``scatter_pos = prefixes[gid] + offsets[wg]`` and writes ``data[gid]`` to ``output[scatter_pos]``. **Vulkan pipeline construction:** one ``VkPipelineLayout`` with a 7-binding descriptor set layout (data, mask, prefixes, offsets, output, input validity, output validity) and a single push constant (element count + has_validity flag). Pass 1 uses bindings 0-3; pass 3 uses all 7 bindings. Both passes share the same layout. **Validity propagation:** when the input column is nullable, pass3 scatters validity bits to the output using ``atomicOr``. Each surviving element writes its validity bit to the corresponding output position — no contention since each output position is unique. The output validity buffer is zeroed via ``vkCmdFillBuffer`` before scatter. **Output:** a new ``VcColumn`` whose buffer is the direct scatter output (device-local, ``STORAGE_BUFFER | TRANSFER_SRC``). No host round-trip needed before chaining further GPU operations. .. rubric:: Edge cases - Zero-length input: returns ``NULL`` with ``*out_length = 0``. - Mask type must be ``VC_DTYPE_U8``. - Data and mask must have equal length. :param ctx: GPU context. :param data_col: Input data column. :param mask_col: ``U8`` mask column (non-zero = keep). :param out_length: [out] number of surviving elements. :returns: A new filtered column, or ``NULL`` on failure. .. c:function:: VcColumn* vc_cast(VcContext* ctx, VcColumn* col, VcDtype target_dtype) **Status:** fully implemented. Element-wise type conversion between numeric data types. Supports all 10 ``VcDtype`` values as both source and target. Each element is processed independently — an element-wise parallel map. **Shader:** ``type_cast.comp`` — a single GLSL compute shader that packs all type-pair logic with dispatch-time branching via push constants (``src_dtype``, ``dst_dtype``). All data is read and written as raw ``uint`` arrays with custom pack/unpack helpers for sub-32-bit types. Intermediate conversions pass through ``float64`` to preserve precision and avoid double rounding for integer-to-integer casts. **Vulkan pipeline construction:** one ``VkPipelineLayout`` with a 2-binding descriptor set layout (source storage buffer at binding 0, destination at binding 1) and a single push constant block (``element_count``, ``src_dtype``, ``dst_dtype``). **Output:** a new ``VcColumn`` with ``target_dtype``, allocated via VMA with host access for readback. Data stays in VRAM for chaining further GPU operations. .. rubric:: Edge cases - Zero-length input: returns an empty column of the target dtype. - NaN input when casting to integer: clamped to 0. - Negative float to unsigned: clamped to 0. - Out-of-range float to integer: saturated to the target type's min/max. - Identity cast (same source and target dtype): produces a copy with identical values. - Null context or column: returns ``NULL``. :param ctx: GPU context. :param col: Input column. :param target_dtype: Desired output data type. :returns: A new cast column, or ``NULL`` on failure. .. c:function:: int vc_reduce(VcContext* ctx, VcColumn* col, VcReduceOp op, uint32_t flags, VcScalar* out) **Status:** fully implemented. Reduce an entire column to a single scalar via a two-level parallel reduction: sum, min, or max. **Shader:** ``reduce.comp`` — shared-memory binary tree reduction. 256 threads per workgroup. Each thread loops over ``stride`` elements, accumulating into a register. The tree reduces 256 values to 1 per WG. If ``num_workgroups > 1``, a second dispatch reduces the intermediate buffer (using 64-bit promoted dtype since each intermediate value is stored as 2 ``uint32_t``). Only the final 8-byte scalar crosses PCIe. **Vulkan pipeline construction:** one ``VkPipelineLayout`` with a 3-binding descriptor set layout (input buffer, validity bitmap, output buffer) and a single push constant block (element count, op selector, dtype, skip-nulls flag, stride). .. rubric:: Return types by operation +---------+-----------------------------------------+ | Op | ``VcScalar.dtype`` | +=========+=========================================+ | ``SUM`` | ``U64`` (unsigned), ``I64`` (signed), | | | ``F64`` (float). Overflow documented. | +---------+-----------------------------------------+ | ``MIN`` | Same as input column dtype. | +---------+-----------------------------------------+ | ``MAX`` | Same as input column dtype. | +---------+-----------------------------------------+ .. rubric:: Edge cases - Zero-length input: returns identity value (0 for sum, ``+inf`` or max integer for min). - ``VC_REDUCE_SKIP_NULLS``: skips invalid elements. All-null input returns the identity value. - Single-workgroup columns: pass 2 is skipped; pass 1 output is the final scalar. - Multi-workgroup columns: pass 2 reduces the intermediate buffer with stride computed to cover all intermediate values. :param ctx: GPU context. :param col: Input column. :param op: ``VC_REDUCE_SUM``, ``VC_REDUCE_MIN``, or ``VC_REDUCE_MAX``. :param flags: ``0`` or ``VC_REDUCE_SKIP_NULLS``. :param out: [out] Scalar result (dtype + union). :returns: ``0`` on success, ``-1`` on error. ---- Chained Execution ----------------- Operations are designed to chain without host round-trips. ``vc_filter`` returns a ``VcColumn*`` whose buffer is device-local with ``STORAGE_BUFFER | TRANSFER_SRC`` usage — immediately consumable by ``vc_cast`` or any future operation. The column handle carries metadata (length, dtype) that downstream operations read without GPU-to-host synchronization. Chaining avoids the PCIe tax on intermediate results. Benchmark results on an RX 9070 (Debug build): +---------------------+----------+-----------+-----------+ | Scenario | 64 KB | 4 MB | 64 MB | +=====================+==========+===========+===========+ | GPU chained (VRAM) | 0.87 ms | 3.35 ms | 88.9 ms | +---------------------+----------+-----------+-----------+ | GPU separate ops | — | 5.70 ms | 154 ms | +---------------------+----------+-----------+-----------+ | CPU reference | 1.38 ms | 22.0 ms | 331 ms | +---------------------+----------+-----------+-----------+ Scatter (filter) output feeds directly into an element-wise map (cast) without ever touching host RAM. The GPU beats CPU by 1.6--6.6x on chained operations. If a ``vc_filter`` returns zero surviving elements, the returned column has length 0 and can still be passed to downstream operations (they will produce zero-length output). The caller is responsible for freeing every ``VcColumn*`` returned by an operation. ---- Utilities --------- .. c:function:: uint32_t vc_dtype_size(VcDtype dtype) **Status:** fully implemented. Returns the size in bytes of the given data type (``U8`` = 1, ``U32`` = 4, ``F64`` = 8, etc.). Does not touch Vulkan — a pure lookup table. .. c:function:: VcColumn* vc_sort(VcContext* ctx, VcColumn* col) **Status:** fully implemented. Sorts a column in ascending order using LSD radix sort. **Vulkan primitives:** - **Histogram pass:** dispatches ``num_wgs`` workgroups (256 threads each). Each WG computes a private 257-bin shared-memory histogram (0-255 digit bins + bin 256 for nulls), then writes the per-WG histogram to a host-visible ``STORAGE_BUFFER``. The host reads the per-WG histogram and computes per-WG exclusive-scan offsets for each digit. - **Reorder pass:** dispatches ``num_wgs`` workgroups. Each WG processes its slice of the column (``wg * 256 * stride`` elements), computes per-digit local ranks via an O(256) shared-memory scan, and writes each element to ``per_wg_base_offset[wg][digit] + local_rank`` in the output buffer. - **Ping-pong buffers:** two ``STORAGE_BUFFER`` allocations swapped each digit pass. No host round-trip during the sort loop. - **Sub-u32 types** (U8, I8, U16, I16) are widened to U32/I32 via a pre-processing compute pass before the radix sort, then repacked to the original type at the end. **Algorithm:** LSD (least significant digit) radix sort. 4 passes for 32-bit dtypes (U8--F32), 8 passes for 64-bit (U64--F64). Each pass sorts by one byte. Signed types XOR-flip MSB at extraction; floats convert to total order via sign-bit complement. **Null handling:** nulls are counted in bin 256. During reorder, null elements are placed after all valid elements. The output null count is the sum of 'bin 256' counts across all WGs. **Edge cases:** - Zero-length input: returns an empty column of the same dtype. - Null context or column: returns ``NULL``. - Sub-u32 types: transparently widened and repacked. - Already-sorted / reverse-sorted / duplicates: handled correctly. - Single-element / prime-length columns: handled correctly. - Nullable columns with mixed valid/null elements: nulls placed at end, valid elements sorted ascending. :param ctx: GPU context. :param col: Input column. :returns: A sorted column (same length and dtype), or ``NULL`` on error.