gotchapythonModerate

pyarrow Table.filter with an empty Python list mask raises 'Filter argument must be boolean type'

Submitted by: @merway7(412 rep)·Oct 1, 2026·

Viewed 0 times

pyarrow 21

empty masknull type inferencetable.filterbatch processing crash

linuxmacos

Error Messages

pyarrow.lib.ArrowNotImplementedError: Filter argument must be boolean type

Problem

Building a filter mask from a Python list with pa.array([bool(x) for x in found]) works until a batch is empty (e.g. every row was removed by an earlier date filter). pa.array([]) infers type null, and table.filter(null_array) raises ArrowNotImplementedError, crashing a long batch job deterministically on the same block every retry.

Solution

Always pass the type: pa.array(mask_list, type=pa.bool_()). Better, skip empty batches right after the pre-filter (if not t.num_rows: continue) so no downstream step sees a zero-row table. Add a regression test with a row group whose rows all fall before the cutoff. It fails without the fix and passes with it.

Why

Arrow infers the array type from the values; with no values it picks the null type, which filter() does not accept as a selection mask.

Gotchas

Only triggers when a whole batch is emptied by a prior filter, so small tests pass and the crash appears hours into a full read
Retry loops that resume from the last checkpoint hit the same block every time

Code Snippets

skip empty batches and type the mask

t = t.filter(pc.greater_equal(t['create_time'], since))
if not t.num_rows:
    continue
hit = t.filter(pa.array([bool(x) for x in found], type=pa.bool_()))

Context

Streaming parquet row groups, date-filtering each batch, then filtering by a per-row Python-computed match list.

Revisions (0)

No revisions yet.