Why is the Mersenne Twister regarded as good?

The Mersenne Twister is widely regarded as good. Heck, the CPython source says that it "is one of the most extensively tested generators in existence." But what does this mean? When asked to list properties of this generator, most of what I can offer is bad:

• It's massive and inflexible (eg. no seeking or multiple streams),

• It fails standard statistical tests despite its massive state size,

• It has serious problems around 0, suggesting that it randomizes itself pretty poorly,

• It's hardly fast

and so on. Compared to simple RNGs like XorShift*, it's also hopelessly complicated.

So I looked for some information about why this was ever thought to be good. The original paper makes lots of comments on the "super astronomical" period and 623-dimensional equidistribution, saying


Among many known measures, the tests based on the higher dimensional
uniformity, such as the spectral test (c.f., Knuth [1981]) and the k-distribution test, described below, are considered to be strongest.

But, for this property, the generator is beaten by a counter of sufficient length! This makes no commentary of local distributions, which is what you actually care about in a generator (although "local" can mean various things). And even CSPRNGs don't care for such large periods, since it's just not remotely important.

There's a lot of maths in the paper, but as far as I can tell little of this is actually about randomness quality. Pretty much every mention of that quickly jumps back to these original, largely useless claims.

It seems like people jumped onto this bandwagon at the expense of older, more reliable technologies. For example, if you just up the number of words in an LCG to 3 (much less than the "only 624" of a Mersenne Twister) and output the top word each pass, it passes BigCrush (the harder part of the TestU01 test suite), despite the Twister failing it (PCG paper, fig. 2). Given this, and the weak evidence I was able to find in support of the Mersenne Twister, w

The initial Mersenne-Twister (MT) was regarded as good for some years, until it was found out to be pretty bad with the more advanced TestU01 BigCrush tests and better PRNGs.

This page lists the Mersenne-Twister features in detail:
Positive Qualities

• Produces 32-bit or 64-bit numbers (thus usable as source of random bits)

• Passes most statistical tests

Neutral Qualities

• Inordinately huge period of $2^{19937} - 1$

• 623-dimensionally equidistributed

• Period can be partitioned to emulate multiple streams

Negative Qualities

• Fails some statistical tests, with as few as 45,000 numbers. Fails LinearComp Test of the TestU01 Crush and BigCrush batteries.

• Predictable — after 624 outputs, we can completely predict its output.

• Generator state occupies 2504 bytes of RAM — in contrast, an extremely usable generator with a huger-than-anyone-can-ever-use period can fit in 8 bytes of RAM.

• Not particularly fast.

• Not particularly space efficient. The generator uses 20000 bits to store its internal state (20032 bits on 64-bit machines), but has a period of only $2^{19937}$, a factor of $2^{63}$ (or $2^{95}$) fewer than an ideal generator of the same size.

• Uneven in its output; the generator can get into “bad states” that are slow to recover from.

• Seedings that only differ slightly take a long time to diverge from each other; seeding must be done carefully to avoid bad states.

• While jump-ahead is possible, algorithms to do so are slow to compute (i.e., require several seconds) and rarely provided by implementations.

Summary: Mersenne Twister is not good enough anymore, but most applications and libraries are not there yet.

StackExchange Computer Science Q#50059, answer score: 23