Two nodes share tokens Cassandra 3.11

After getting some inconsistent query results we noticed that two nodes are claiming ownership of two tokens (-5930834448474026108 and 1653676441334745669). When a node comes up we flip between consistent and inconsistent query results and we see this in the logs.

TokenMetadata.java:226 - Token -5930834448474026108 changing ownership from /172.17.32.251 to /172.17.33.62
TokenMetadata.java:226 - Token 1653676441334745669 changing ownership from /172.17.32.251 to /172.17.33.62

Or the reverse

TokenMetadata.java:226 - Token -5930834448474026108 changing ownership from /172.17.33.62 to /172.17.32.251
TokenMetadata.java:226 - Token 1653676441334745669 changing ownership from /172.17.33.62 to /172.17.32.251

This is the view of peers from a seed node on the cluster.

```
cqlsh> select peer, tokens from system.peers where tokens contains '-5930834448474026108' and tokens contains '1653676441334745669' ALLOW FILTERING;

peer | tokens
---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
172.17.32.251 | {'-3586663459313641829', '-4235041598543029843', '-5390242841269357826', '-5930834448474026108', '-8760593756478543069', '1653676441334745669', '166599935335281141', '2082376249768721299', '2737611629421780150', '3505320720095561779', '5463123014169021365', '6208992122760287054', '753682649578647219', '7718045837568216134', '8541991172278593633', '9138457739984968252'}
172.17.33.62 | {'-1187962964387570247', '-1890723219453312357', '-3797875287284457587', '-4763268132960963434', '-5264847899607678948', '-5930834448474026108', '-6918838058130066386', '-7528285457674952405', '-7572391200230517', '-8154

Did you happen to bootstrap nodes in parallel? I've seen this happen when attempting to bootstrap nodes too quickly.
The best way to fix it now is to decommission one of the nodes from the cluster, assassinate the node once decommission has finished, then clear the data and bootstrap it back to the cluster. With your data load, that's not a fast operation to complete, but it's the best way to fix the problem.

StackExchange Database Administrators Q#320449, answer score: 3