HiveBrain v1.2.0
Get Started
← Back to all entries
snippetsqlMajor

How to efficiently copy millions of rows from one table to another in Postgresql?

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
postgresqlrowsefficientlymillionsoneanotherhowfromtablecopy

Problem

I have two database tables. One contains hundreds of millions of records. Lets call that one history. The other one is calculated on daily basis and I want to copy all of its records into the history one.

What I did was to run:


INSERT INTO history SELECT * FROM daily


And it did the trick for a while, but it started to get slower and slower as the number of records kept growing. Now I have around 2 million records that need to be copied from daily to history in single operation and it takes too long to complete.

Is there another, more efficient way of copying data from one table to another?

Solution

The problem was with indexes. The history table had 160M indexed rows. By running either COPY FROM or INSERT INTO .. SELECT it was taking a lot of time not to insert rows, but to update indexes. When i disabled indexes, it imported 3M rows in 10 seconds. Now i need to find faster way of reindexing the big table.

Context

StackExchange Database Administrators Q#55568, answer score: 21

Revisions (0)

No revisions yet.