Concurrent UPDATE deadlock (updating 2 fields, selecting by PK)

I have a strange deadlock case, where simple concurrent updates are causing deadlocks on MySQL (5.6.27).

The setup is with multiple workers (several different servers) processing queues and sometimes (not always) we end up with a deadlock.

Queries are fairly straightforward:

START TRANSACTION
UPDATE transactions SET
        suspended_reason = 'PoF is required',
        updated_at = '2017-08-23 06:27:10'
    WHERE id = 1711933
COMMIT

Output from SHOW ENGINE INNODB STATUS

``
------------------------
LATEST DETECTED DEADLOCK
------------------------
2017-08-23 06:27:10 2b4f5ff03700
*** (1) TRANSACTION:
TRANSACTION 24880629, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 91461, OS thread handle 0x2b4f772c2700, query id 3579786 10.2.10.156 staging updating
UPDATE transactions SET suspended_reason = 'PoF is required', updated_at = '2017-08-23 06:27:10' WHERE id = 1711933
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 934 page no 26924 n bits 128 index PRIMARY of table maindb.transactions trx id 24880629 lock_mode X locks rec but not gap waiting
Record lock, heap no 44 PHYSICAL RECORD: n_fields 64; compact format; info bits 0

*** (2) TRANSACTION:
TRANSACTION 24880628, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 91345, OS thread handle 0x2b4f5ff03700, query id 3579787 10.2.11.209 staging updating
UPDATE transactions SET suspended_reason = 'PoF is required', updated_at = '2017-08-23 06:27:10' WHERE id = 1711933
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 934 page no 26924 n bits 128 index PRIMARY of table maindb.transactions` trx id 24880628 lock mode S locks rec but not gap
Record lock, heap no 44 PHYSICAL RECORD: n_fields 64; compact format; info bits 0

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 934 page

Why would workers be touching the same record?? Perhaps the real problem is back where a worker grabs a task and locks it for itself??

The deadlock is valid if two workers processed the same transaction. What is your mechanism for preventing such??

StackExchange Database Administrators Q#193002, answer score: 4