patternphpMinor
Cassandra "Operation timed out - received only 0 responses." during write
Viewed 0 times
timedduringresponsesoperationwritereceivedonlycassandraout
Problem
I'm using Datastax PHP Cassandra Driver and Cassandra 2.2.5 on Windows 10. This is a single-node cluster, and the keyspace replication of
This is the table I'm inserting to:
When I tried to insert a row with a blob that is 16419KB large, it gave me this error message
The row is not inserted at all.
Inserting a row with a blob that is 16379KB worked fine.
The log files don't show anything. When the operation fails, nothing is appended to any log files inside Datastax / Cassandra folders.
After a few hours of Googling, I still have no clue as to why this is happening. I have tried increasing
Does anyone know why a write seems to fail in this way when the size exceeds ~16MB?
Update:
I updated Cassandra to v3.7, same observation, but the error message now says
{
'class': 'SimpleStrategy',
'replication_factor': 1
}This is the table I'm inserting to:
CREATE TABLE MyDocuments (
"label" varchar,
"fileName" varchar,
"blob" blob,
"createdTime" bigint,
"lastUpdateTime" bigint,
PRIMARY KEY("label", "fileName")
);When I tried to insert a row with a blob that is 16419KB large, it gave me this error message
Operation timed out - received only 0 responses.The row is not inserted at all.
Inserting a row with a blob that is 16379KB worked fine.
The log files don't show anything. When the operation fails, nothing is appended to any log files inside Datastax / Cassandra folders.
After a few hours of Googling, I still have no clue as to why this is happening. I have tried increasing
write_request_timeout_in_ms to 20000 in the settings (cassandra.yaml), but it does not help.Does anyone know why a write seems to fail in this way when the size exceeds ~16MB?
Update:
I updated Cassandra to v3.7, same observation, but the error message now says
Operation failed - received 0 responses and 1 failures.Solution
You will need to increase
It increases the size of the commit record segment. This property determines the maximum mutation size (
From the docs:
(Default: 32MB) Sets the size of the individual commitlog file
segments. A commitlog segment may be archived, deleted, or recycled
after all its data has been flushed to SSTables. This amount of data
can potentially include commitlog segments from every table in the
system. The default size is usually suitable for most commitlog
archiving, but if you want a finer granularity, 8 or 16 MB is
reasonable.
This property determines the maximum mutation size, defined as half
the segment size. If a mutation's size exceeds the maximum mutation
size, the mutation is rejected. Before increasing the commitlog
segment size of the commitlog segments, investigate why the mutations
are larger than expected. Look for underlying issues with access
patterns and data model, because increasing the commitlog segment size
is a limited fix.
commitlog_segment_size_in_mb to a higher value. Try 64MB instead.It increases the size of the commit record segment. This property determines the maximum mutation size (
max_mutation_size_in_kb), defined as half of the segment size. By default is 32MB, which made your record were overflowing the maximum mutation size.From the docs:
(Default: 32MB) Sets the size of the individual commitlog file
segments. A commitlog segment may be archived, deleted, or recycled
after all its data has been flushed to SSTables. This amount of data
can potentially include commitlog segments from every table in the
system. The default size is usually suitable for most commitlog
archiving, but if you want a finer granularity, 8 or 16 MB is
reasonable.
This property determines the maximum mutation size, defined as half
the segment size. If a mutation's size exceeds the maximum mutation
size, the mutation is rejected. Before increasing the commitlog
segment size of the commitlog segments, investigate why the mutations
are larger than expected. Look for underlying issues with access
patterns and data model, because increasing the commitlog segment size
is a limited fix.
Context
StackExchange Database Administrators Q#145069, answer score: 5
Revisions (0)
No revisions yet.