Blob Types in Geneva UDFs

Geneva UDFs can consume and produce Lance Blob columns — large binary objects (roughly 1 MB to 100s of MB per value) such as images, audio clips, or documents. Blob columns keep large payloads out of the regular column data path and provide lazy, file-like access on read. Lance has two blob encodings, and Geneva supports both:

	Blob v1	Blob v2
Declaration	`pa.large_binary()` + `lance-encoding:blob` field metadata	`lance.blob_field(...)` (the `lance.blob.v2` extension type)
Table requirement	data storage version ≥ 2.0	data storage version ≥ 2.2
Storage	payload bytes inline in the column	payloads packed into sidecar blob files, referenced by descriptors
Use when	the table already exists on an older storage version	creating new tables — cheaper checkpoints and bulk writes at scale

For Blob v2 output columns on data storage version 2.2+, Geneva checkpoints compact descriptors instead of the payload bytes and writes payloads through PyLance’s bulk blob writer, which significantly reduces checkpoint I/O for large backfills. This requires pylance>=9.0.0b24.

Writing blob columns from UDFs

Blob v1

For a scalar UDF, return bytes, set data_type to pa.large_binary(), and add the field_metadata that marks the column as blob-encoded:

import pyarrow as pa
from geneva import udf

@udf(data_type=pa.large_binary(), field_metadata={"lance-encoding:blob": "true"})
def generate_blob(text: str, multiplier: int) -> bytes:
    """Generate blob data by repeating text."""
    return (text * multiplier).encode("utf-8")

tbl.add_columns({"blob_output": generate_blob})
tbl.backfill("blob_output")

A batched (pa.RecordBatch) UDF is similar — return a pa.large_binary() array:

@udf(data_type=pa.large_binary(), field_metadata={"lance-encoding:blob": "true"})
def batch_to_blob(batch: pa.RecordBatch) -> pa.Array:
    """Convert each row to blob data."""
    blobs = []
    for i in range(batch.num_rows):
        text = batch.column("text")[i].as_py()
        blobs.append(text.encode("utf-8"))
    return pa.array(blobs, type=pa.large_binary())

The destination table must use data storage version 2.0 or newer:

tbl = db.create_table(
    "images",
    data,
    storage_options={"new_table_data_storage_version": "2.0"},
)

Blob v2

The simplest form is a scalar UDF whose data_type is BlobType() — return the payload as bytes (or None):

from lance.blob import BlobType
from geneva import udf

@udf(data_type=BlobType())
def render_thumbnail(image_url: str) -> bytes:
    return make_thumbnail(image_url)

tbl.add_columns({"thumbnail": render_thumbnail})
tbl.backfill("thumbnail")

For production pipelines, declare the output as a struct with a lance.blob_field(...) child. This is the pattern Geneva’s large-scale image pipelines use — the struct carries the payload plus any per-row metadata (an error string, dimensions, a content hash, and so on):

import lance
import pyarrow as pa
from geneva import udf

result_type = pa.struct(
    [
        lance.blob_field("image_bytes"),
        pa.field("error", pa.string()),
    ]
)

@udf(data_type=result_type)
def download_image(url: str) -> dict:
    try:
        return {"image_bytes": fetch(url), "error": None}
    except Exception as exc:
        return {"image_bytes": None, "error": str(exc)}

tbl.add_columns({"download": download_image})
tbl.backfill("download")

Scalar UDFs return the blob child as inline bytes (or None); URI-style descriptors like {"uri": ...} are not accepted. A batched UDF builds the struct explicitly, using lance.blob_array(...) for the blob child:

@udf(data_type=result_type)
def download_images(batch: pa.RecordBatch) -> pa.Array:
    payloads, errors = [], []
    for url in batch.column("url"):
        payloads.append(fetch(url.as_py()))
        errors.append(None)
    return pa.StructArray.from_arrays(
        [lance.blob_array(payloads), pa.array(errors, type=pa.string())],
        fields=list(result_type),
    )

The destination table must use data storage version 2.2 or newer:

tbl = db.create_table(
    "images",
    data,
    storage_options={"new_table_data_storage_version": "2.2"},
)

Reading blob columns in UDFs

Blob v1

Blob v1 columns arrive in scalar UDFs as lazy BlobFile objects — call .read() to fetch the payload:

from lance.blob import BlobFile

@udf(data_type=pa.int64())
def blob_len(blob_output: BlobFile) -> int:
    data = blob_output.read()
    return len(data)

tbl.add_columns({"blob_len": blob_len})
tbl.backfill("blob_len")

Blob v2

Blob v2 payloads arrive as bytes. Reference a nested blob with its dotted path via the (udf, input_columns) tuple form:

@udf(data_type=pa.int64())
def payload_len(image: bytes) -> int:
    return len(image)

tbl.add_columns({"payload_len": (payload_len, ["download.image_bytes"])})
tbl.backfill("payload_len")

Reading blobs back

Table.take_blobs returns lazy BlobFile handles for both encodings. Use the dotted path for a blob nested in a struct:

# Blob v1 column
blobs = tbl.take_blobs([0, 1, 2], column="blob_output")
payloads = [b.read() for b in blobs]

# Blob v2 column nested in a struct
blobs = tbl.take_blobs([0, 1, 2], column="download.image_bytes")
payloads = [b.read() for b in blobs]

For bulk reads of Blob v2 payloads keyed by physical row address, drop down to the Lance dataset:

ds = tbl.to_lance()
pairs = ds.read_blobs("download.image_bytes", addresses=[0, 1, 2], preserve_order=True)
payloads = [bytes(payload) for _addr, payload in pairs]

API Reference

UDF — @udf decorator for defining blob-producing and blob-consuming functions
Table — add_columns(), backfill(), take_blobs()
Lance blob guide — the underlying Blob v1/v2 storage model

Get started

Model training

Guides

Feature Engineering (Geneva)

Support

Blob Types in Geneva UDFs

Writing blob columns from UDFs

Blob v1

Blob v2

Reading blob columns in UDFs

Blob v1

Blob v2

Reading blobs back

API Reference

​Writing blob columns from UDFs

​Blob v1

​Blob v2

​Reading blob columns in UDFs

​Blob v1

​Blob v2

​Reading blobs back

​API Reference

Writing blob columns from UDFs

Blob v1

Blob v2

Reading blob columns in UDFs

Blob v1

Blob v2

Reading blobs back

API Reference