Enhancing speed of looping cycle using Freepascal

I use Lazarus 1.2.4 and Freepascal 2.6.4.

I have created a program that reads a disk in buffers of 64Kb (tried various buffer sizes) using a repeat...until loop. Each buffer is hashed using the SHA1 unit, specifically, SHA1Init, SHA1Update and SHA1Final.

The trouble is, is that although it works and the hashes always match that computed by other tools that do the same job, my program is not as fast. On a specific workstation with an 80Gb disk attached, it reads and hashes at about 1.8Gb per minute, and this is as a result of some enhanced compiler directives (and using specific optimisations offered by the Lazarus\FPC compiler). Before those tweaks, it was just 1.22Gb p\min as an average. The other tools do it at about 2.5Gb+ a minute (around 45Mb a second) and some are faster than that.

If I remove the hashing element and just do the disk reading, it reads at about 4Gb per minute, so I am fairly sure my loop structure is actually fairly quick. So I'm almost certain the bottleneck is the hashing aspect and this has been discussed at the Lazarus forum, here where it has been suggested that maybe the library needs to be improved a little for better speed. One poster suggested I re-write the three functions in assembly but I am not that good.

There is a related post HERE regarding SHA256, where the gentlemen concerned experienced similar issues, though with a different language. His implementation was very similar to mine - Init, Update, Final. One suggestion was to use a buffer of 16Mb in that post. I have tried 4Kb, 8Kb, 64Kb, 256Kb, 512Kb and 1Mb. I haven't gone to 16Mb or anywhere near that - might that prove to be worthwhile? I read that once you go above about 1Mb programs usually go backward?

Is there an obvious way to improve speed?

I have included only the relevant parts in the hope it will make the task easier to read.

```
// Main parts of my code responsible for loop.
// The SHA1 functions from the SHA1 Freepascal unit follow
hSelectedDisk := Crea

If you're processing 1.8Gb per minute using 64Kb buffers, that's (1800000 / 64 =) 28000 buffers per minute i.e. (28000 / 60 =) 470 buffers / second.

I don't know what you're doing elsewhere with these statements ...

ProgressCounter := ProgressCounter + 1; // We use this update the progress display occasionally, instead of every buffer read
TimeStartRead   := Now;

... but you should probably NOT try to update a GUI progress bar 500 times/second!

Also, avoid calling this 500 times times/second:

lblBytesLeftToHashB.Caption := IntToStr(ExactDiskSize - NewPos) + ' bytes, ' + FormatByteSize(ExactDiskSize - NewPos);

Try disabling/removing that GUI-updating code completely, to see whether that improves performance. If it does improve performance then partially re-add the GUI-updating code: for example, update the GUI once every 100 buffers (instead of once every buffer as you're doing now).

Also you should get better performance if you move the I/O to a separate thread.

Alternatively this would be a time to use 'overlapped I/O' (e.g. passing a non-null LPOVERLAPPED parameter to the Win32 ReadFile function) but I don't know how to do that with the Freepascal run-time library.

StackExchange Code Review Q#56284, answer score: 4