patternMinor
Writing a log file with threads
Viewed 0 times
filelogwiththreadswriting
Problem
Here are all the threads write the log file. I want to know if the below script is fine, if it needs any modifications, and whether I am utilising threads properly to execute this code quickly.
open(LOG,"+>","log.txt")or die "cant open the file";
my $file_total= scalar @listOfFiles;
my $div = $file_total/$thread_count;
$div = ceil($div);
my $start = 0;
my $end = $div;
for ($count = 1; $count new(\®ister, $start, $end );
push(@threads,$thread);
$start = $end;
$end = $end + $div;
if ($end > $file_total)
{
$end = $file_total;
}
}
foreach (@threads)
{
$_->join;
}
close(LOG);
sub register
{
my $lstart = shift;
my $lend = shift;
for (my $index = $lstart; $index < $lend; ++$index)
{
print LOG $listOfFiles[$index];
process......
..........
.......
}
}process... placeholder: I have a huge number of documents, say, 1 million. I am collecting all the 1 million filenames, storing them in the listOfFiles array, and dividing the array index based on the number of threads the user wants. I am passing every thread and the start and end indices to fetch the document name, process the document, and store the status of the output to a log file.Solution
You can atomically write to a file if you are working with a threadsafe queue.
The output of /tmp/atomic.log looks like:
#!/usr/bin/env perl
use warnings FATAL => 'all';
use strict;
use Time::HiRes qw(time nanosleep);
use threads;
use Thread::Queue;
my $logQ = Thread::Queue->new();
my $consumer = threads->new(\&_logConsumer);
my @producer;
push(@producer, threads->new(\&_logProducer)) for (1..10);
$_->join() foreach @producer;
$logQ->enqueue('STOP');
$consumer->join();
sub _logProducer {
for (1..10) {
$logQ->enqueue(sprintf('[%-17s:%04i]', time(), threads->tid()));
nanosleep(int(rand(10_000)));
}
return;
}
sub _logConsumer {
open(my $fh, '>', '/tmp/atomic.log')
or die 'Cannot open log file, stopped';
my $running = 1;
while($running) {
if (defined(my $item = $logQ->dequeue())) {
$running = undef if $item eq 'STOP';
print $fh "$item\n";
}
}
close($fh);
return;
}The output of /tmp/atomic.log looks like:
[1526568957.83079 :0011]
[1526568957.8308 :0006]
[1526568957.83081 :0009]
[1526568957.83082 :0010]
[1526568957.83087 :0008]Code Snippets
#!/usr/bin/env perl
use warnings FATAL => 'all';
use strict;
use Time::HiRes qw(time nanosleep);
use threads;
use Thread::Queue;
my $logQ = Thread::Queue->new();
my $consumer = threads->new(\&_logConsumer);
my @producer;
push(@producer, threads->new(\&_logProducer)) for (1..10);
$_->join() foreach @producer;
$logQ->enqueue('STOP');
$consumer->join();
sub _logProducer {
for (1..10) {
$logQ->enqueue(sprintf('[%-17s:%04i]', time(), threads->tid()));
nanosleep(int(rand(10_000)));
}
return;
}
sub _logConsumer {
open(my $fh, '>', '/tmp/atomic.log')
or die 'Cannot open log file, stopped';
my $running = 1;
while($running) {
if (defined(my $item = $logQ->dequeue())) {
$running = undef if $item eq 'STOP';
print $fh "$item\n";
}
}
close($fh);
return;
}[1526568957.83079 :0011]
[1526568957.8308 :0006]
[1526568957.83081 :0009]
[1526568957.83082 :0010]
[1526568957.83087 :0008]Context
StackExchange Code Review Q#49708, answer score: 2
Revisions (0)
No revisions yet.