snippetsqlMajor
How do I find out how far along my PostgreSQL query is?
Viewed 0 times
postgresqlhowqueryfarfindalongout
Problem
I have a pretty decent idea of how many rows my SELECT...INTO query will actually process (e.g. I know how many will materialize).
I understand Postgres won't tell me percentage completeness, is there a way (buried deep in logs, system tables, or otherwise) that I can find out how many rows have been pumped into the destination table or have been read by the SELECT query?
I understand Postgres won't tell me percentage completeness, is there a way (buried deep in logs, system tables, or otherwise) that I can find out how many rows have been pumped into the destination table or have been read by the SELECT query?
Solution
As Daniel Vérité mentioned there doesn't seem to be a generic solution. When loading data into a table from a file the following technique can be used to get the progress of the load.
COPY command console progress bar
Create an empty table.
Create a data file with 10 million lines for loading into the table.
Load data from file into the table and display a progress bar.
Demo
How this works
By using the copy commands STDIN option we can feed in the data for the copy operation from another process. The pv command will output a file and track it's progress displaying a progress bar, ETA, total time elapsed and the rate of data transfer.
COPY command graphical progress bar
Using the same general technique we could display a progress bar in a graphical application or a web-based application. Using python for example the psycopg2 module lets you call the copy command with a file object of your choosing. You could then track how much of your file object has been read and display a progress bar.
COPY command console progress bar
Create an empty table.
CREATE TABLE mytest (n int);Create a data file with 10 million lines for loading into the table.
$ seq 10000000 > /tmp/data.txtLoad data from file into the table and display a progress bar.
$ pv /tmp/data.txt | psql -c "COPY mytest FROM STDIN;"Demo
How this works
By using the copy commands STDIN option we can feed in the data for the copy operation from another process. The pv command will output a file and track it's progress displaying a progress bar, ETA, total time elapsed and the rate of data transfer.
COPY command graphical progress bar
Using the same general technique we could display a progress bar in a graphical application or a web-based application. Using python for example the psycopg2 module lets you call the copy command with a file object of your choosing. You could then track how much of your file object has been read and display a progress bar.
Code Snippets
CREATE TABLE mytest (n int);$ seq 10000000 > /tmp/data.txt$ pv /tmp/data.txt | psql -c "COPY mytest FROM STDIN;"Context
StackExchange Database Administrators Q#50602, answer score: 41
Revisions (0)
No revisions yet.