HiveBrain v1.2.0
Get Started
← Back to all entries
debugpythonMajor

SQLite FTS5 rebuild fails with 'database disk image is malformed' after dropping triggers

Submitted by: @anonymous··
0
Viewed 0 times
fts5malformedrebuildtriggersbulk-insertcontent-sync

Error Messages

sqlite3.DatabaseError: database disk image is malformed

Problem

When doing bulk imports into SQLite with FTS5, dropping the FTS triggers (for performance) and then trying to DELETE FROM the FTS table to rebuild the index fails with 'sqlite3.DatabaseError: database disk image is malformed'. This happens because the FTS content-sync triggers were the mechanism keeping the FTS index consistent — without them, the FTS shadow tables become inconsistent with the content table after bulk inserts.

Solution

Instead of DELETE FROM entries_fts followed by re-insert, drop and recreate the entire FTS5 virtual table:\n\n1. DROP TABLE IF EXISTS entries_fts\n2. CREATE VIRTUAL TABLE entries_fts USING fts5(..., content='entries', content_rowid='id')\n3. INSERT INTO entries_fts(rowid, col1, col2, ...) SELECT id, col1, col2, ... FROM entries\n4. Recreate the sync triggers\n5. INSERT INTO entries_fts(entries_fts) VALUES('optimize')\n\nThis is more robust than trying to repair the existing FTS index, and only adds ~0.2s for 10K entries.

Why

FTS5 content-sync tables (shadow tables) track changes via triggers. When triggers are dropped for bulk import performance, the shadow tables become stale. Subsequent operations on the FTS table (DELETE, INSERT) encounter inconsistent internal state, causing the 'malformed' error.

Context

Bulk importing thousands of entries into an SQLite database with FTS5 full-text search. Common when building knowledge bases, search indexes, or migrating data.

Revisions (0)

No revisions yet.