What are GPUs bad at?

I understand that GPUs are generally used to do LOTS of calculations in parallel. I understand why we would want to parallelize processes in order to speed things up. However, GPUs aren't always better than CPUs, as far as I know.

What kinds of tasks are GPUs bad at? When would we prefer CPU over GPU for processing?

GPUs are bad at doing one thing at a time. A modern high-end GPU may have several thousand cores, but these are organized into SIMD blocks of 16 or 32. If you want to compute 2+2, you might have 32 cores each compute an addition operation, and then discard 31 of the results.

GPUs are bad at doing individual things fast. GPUs only recently topped the one-gigahertz mark, something that CPUs did more than twenty years ago. If your task involves doing many things to one piece of data, rather than one thing to many pieces of data, a CPU is far better.

GPUs are bad at dealing with data non-locality. The hardware is optimized for working on contiguous blocks of data. If your task involves picking up individual pieces of data scattered around your data set, the GPU's incredible memory bandwidth is mostly wasted.

StackExchange Computer Science Q#121080, answer score: 143