PostgreSQL Sequential Scan instead of Index Scan Why?

Hi All I've got a problem with my PostgreSQL database query and wondering if anyone can help. In some scenarios my query seems to ignore the index that I've created which is used for joining the two tables data and data_area. When this happens it uses a sequential scan and results in a much slower query.

Sequential Scan (~5 minutes)

```
Unique (cost=15368261.82..15369053.96 rows=200 width=1942) (actual time=301266.832..301346.936 rows=153812 loops=1)
CTE data
-> Bitmap Heap Scan on data (cost=6086.77..610089.54 rows=321976 width=297) (actual time=26.286..197.625 rows=335130 loops=1)
Recheck Cond: (datasetid = 1)
Filter: ((readingdatetime >= '1920-01-01 00:00:00'::timestamp without time zone) AND (readingdatetime = 0::double precision) AND (depth Bitmap Index Scan on data_datasetid_index (cost=0.00..6006.27 rows=324789 width=0) (actual time=25.462..25.462 rows=335130 loops=1)
Index Cond: (datasetid = 1)
-> Sort (cost=15368261.82..15368657.89 rows=158427 width=1942) (actual time=301266.829..301287.110 rows=155194 loops=1)
Sort Key: data.id
Sort Method: quicksort Memory: 81999kB
-> Hash Left Join (cost=15174943.29..15354578.91 rows=158427 width=1942) (actual time=300068.588..301052.832 rows=155194 loops=1)
Hash Cond: (data_area.area_id = area.id)
-> Hash Join (cost=15174792.93..15351854.12 rows=158427 width=684) (actual time=300066.288..300971.644 rows=155194 loops=1)
Hash Cond: (data.id = data_area.data_id)
-> CTE Scan on data (cost=0.00..6439.52 rows=321976 width=676) (actual time=26.290..313.842 rows=335130 loops=1)
-> Hash (cost=14857017.62..14857017.62 rows=25422025 width=8) (actual time=300028.260..300028.260 rows=26709939 loops=1)
Buckets: 4194304 Batches: 1 Memory Usage: 1043357kB
-> Seq Scan on data_area (cost=0.

Notice this line:

->  Index Scan using data_area_pkey on data_area  (cost=0.00..52.13 rows=1 width=8) 
    (actual time=0.006..0.008 rows=0 loops=335130)

If you compute the total cost, considering loops, it is 52.13 * 335130 = 17470326.9. This is larger than 14857017.62 for the seq_scan alternative. That is why it does not use the index.

So the optimizer is overestimating the cost of the index scan. I'd guess that your data is sorted on the index (either due to a clustered index or to how it was loaded) and/or you have plenty of cache memory and/or a nice fast disk. Hence there is little random I/O going on.

You should also check the correlation in pg_stats, that is used by the optimizer to assess clustering when computing the index cost, and finally try changing random_page_cost and cpu_index_tuple_cost, to match your system.

StackExchange Database Administrators Q#36374, answer score: 10