HiveBrain v1.2.0
Get Started
← Back to all entries
patternphpMinor

Cassandra "Operation timed out - received only 0 responses." during write

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
timedduringresponsesoperationwritereceivedonlycassandraout

Problem

I'm using Datastax PHP Cassandra Driver and Cassandra 2.2.5 on Windows 10. This is a single-node cluster, and the keyspace replication of

{
    'class': 'SimpleStrategy',
    'replication_factor': 1
}


This is the table I'm inserting to:

CREATE TABLE MyDocuments (
    "label" varchar,
    "fileName" varchar,
    "blob" blob,
    "createdTime" bigint,
    "lastUpdateTime" bigint,
    PRIMARY KEY("label", "fileName")
);


When I tried to insert a row with a blob that is 16419KB large, it gave me this error message Operation timed out - received only 0 responses.
The row is not inserted at all.

Inserting a row with a blob that is 16379KB worked fine.

The log files don't show anything. When the operation fails, nothing is appended to any log files inside Datastax / Cassandra folders.

After a few hours of Googling, I still have no clue as to why this is happening. I have tried increasing write_request_timeout_in_ms to 20000 in the settings (cassandra.yaml), but it does not help.

Does anyone know why a write seems to fail in this way when the size exceeds ~16MB?

Update:

I updated Cassandra to v3.7, same observation, but the error message now says Operation failed - received 0 responses and 1 failures.

Solution

You will need to increase commitlog_segment_size_in_mb to a higher value. Try 64MB instead.

It increases the size of the commit record segment. This property determines the maximum mutation size (max_mutation_size_in_kb), defined as half of the segment size. By default is 32MB, which made your record were overflowing the maximum mutation size.

From the docs:


(Default: 32MB) Sets the size of the individual commitlog file
segments. A commitlog segment may be archived, deleted, or recycled
after all its data has been flushed to SSTables. This amount of data
can potentially include commitlog segments from every table in the
system. The default size is usually suitable for most commitlog
archiving, but if you want a finer granularity, 8 or 16 MB is
reasonable.


This property determines the maximum mutation size, defined as half
the segment size. If a mutation's size exceeds the maximum mutation
size, the mutation is rejected. Before increasing the commitlog
segment size of the commitlog segments, investigate why the mutations
are larger than expected. Look for underlying issues with access
patterns and data model, because increasing the commitlog segment size
is a limited fix.

Context

StackExchange Database Administrators Q#145069, answer score: 5

Revisions (0)

No revisions yet.