patterncppModerate
Writing a large file of numbers
Viewed 0 times
filenumberswritinglarge
Problem
Basically, I want to write a huge file of numbers. Each line in that file should contain one number. This program below is extremely slow (obviously) but it works. It writes one line after line and always one number until the limit of an
This program wrote in 2 minutes on my desktop computer 12GB. On my laptop only 117 MB in 10 minutes. On both computers I had to stop the program because it was making my computer slow and still wasn't done. Any way to optimize it so it'll be faster and maybe run more smooth so my other programs will still run perfectly fine?
unsigned long has been reached.#include
#include
#include
int main() {
unsigned long max = std::numeric_limits::max();
unsigned long i = 0;
std::ofstream myFile;
myFile.open( "/home/awesome/Desktop/large_numbers.txt" );
while( i < max ) {
myFile << i << "\n";
i++;
}
myFile.flush();
myFile.close();
}This program wrote in 2 minutes on my desktop computer 12GB. On my laptop only 117 MB in 10 minutes. On both computers I had to stop the program because it was making my computer slow and still wasn't done. Any way to optimize it so it'll be faster and maybe run more smooth so my other programs will still run perfectly fine?
Solution
Few things.
This number:
Is
If we assume that the average size of a number is half this length (so 10 characters). Then the size of the file is:
Unless you have some sort of massive raid. I don't think you have storage space foe all those numbers.
Even if your system uses a smaller number of bits for an unsigned long than mine (say 32 bit). This is still a large file (though doable on modern systems).
But the problem is not the speed of the program. Its is the speed that the OS can copy data to the disk. That will be your limiting factor. If you want to see how fast it could write to an optimal disk then use the special file "/dev/null". You can write to this and it uses no disk space.
12G in 2 min is a write rate of:
A quick Google reveals that 500 M/s is achievable for SSD drives while normal HD achieve approx 100 M/s. So it looks like your desk top has normal spinning platter hard disk and it was going at the top speed.
Laptops are known to slower. Especially when they get hot. The hardware is designed to physically throttle the system to make sure it does not overheat.
Why not create and open the file in one line?
I would do this:
Simpler to use a
No need to flush or close the stream manually.
When the object goes out of scope the destructor is called. This will close the file which will flush the content of the buffers first.
This number:
unsigned long max = std::numeric_limits::max();Is
18446744073709551615If we assume that the average size of a number is half this length (so 10 characters). Then the size of the file is:
184467440737095516150 bytes.
=> 180143985094819830 Kbytes
=> 175921860444150 MBytes
=> 171798691830 GBytes
=> 167772150 TBytes
=> 163830 PBytes
=> 150 XBytesUnless you have some sort of massive raid. I don't think you have storage space foe all those numbers.
Even if your system uses a smaller number of bits for an unsigned long than mine (say 32 bit). This is still a large file (though doable on modern systems).
=> 15 GBytes (approx)But the problem is not the speed of the program. Its is the speed that the OS can copy data to the disk. That will be your limiting factor. If you want to see how fast it could write to an optimal disk then use the special file "/dev/null". You can write to this and it uses no disk space.
12G in 2 min is a write rate of:
107374182 bytes a sec or
104857 K/s
102 M/sA quick Google reveals that 500 M/s is achievable for SSD drives while normal HD achieve approx 100 M/s. So it looks like your desk top has normal spinning platter hard disk and it was going at the top speed.
Laptops are known to slower. Especially when they get hot. The hardware is designed to physically throttle the system to make sure it does not overheat.
Why not create and open the file in one line?
std::ofstream myFile;
myFile.open( "/home/awesome/Desktop/large_numbers.txt" );I would do this:
std::ofstream myFile( "/home/awesome/Desktop/large_numbers.txt" );Simpler to use a
for(;;) loop than a while().for(unsigned long i = 0;i <= max; ++i)
myFile << i << "\n"; // Note because you opened the file in
// text mode. The '\n' character will
// translated into the platform end of
// of line sequence.
}No need to flush or close the stream manually.
myFile.flush();
myFile.close();When the object goes out of scope the destructor is called. This will close the file which will flush the content of the buffers first.
#include
#include
#include
int main() {
unsigned long max = std::numeric_limits::max();
std::ofstream myFile( "/home/awesome/Desktop/large_numbers.txt" );
for(unsigned long i = 0; i < max; ++i) {
myFile << i << "\n";
}
}Code Snippets
unsigned long max = std::numeric_limits<unsigned long>::max();18446744073709551615184467440737095516150 bytes.
=> 180143985094819830 Kbytes
=> 175921860444150 MBytes
=> 171798691830 GBytes
=> 167772150 TBytes
=> 163830 PBytes
=> 150 XBytes=> 15 GBytes (approx)107374182 bytes a sec or
104857 K/s
102 M/sContext
StackExchange Code Review Q#94352, answer score: 15
Revisions (0)
No revisions yet.