HiveBrain v1.2.0
Get Started
← Back to all entries
snippetcsharpMinor

Multiprocess, multithreaded read write on a single file

Submitted by: @import:stackexchange-codereview··
0
Viewed 0 times
filereadwritesinglemultiprocessmultithreaded

Problem

I've got a job to create a library that would be able to support multithreaded and multi process read and write to a single file. On that single file we would store C# models in an array in JSON format. It would basically work as a mini file system JSON DB.

When the application that uses this lib starts, it would load existing models (if they do exist) in memory, so that we can query the data in memory. Adding, updating and removing of objects would require to flush the data to that single file. I assume that the hardest thing here to achieve would be synchronization between threads and processes. I actually managed to do that with mutex. Currently, I only have two methods, one for writing, and one for reading:

```
public void Write (string text)
{
var appGuid = "{FBE403EA-0D59-48D6-941D-E74E0F20B482}";
var mutexId = $"Global\\{{{appGuid}}}";

bool createdNew;

var allowEveryoneRule = new MutexAccessRule(new SecurityIdentifier(WellKnownSidType.WorldSid, null), MutexRights.FullControl, AccessControlType.Allow);
var securitySettings = new MutexSecurity();
securitySettings.AddAccessRule(allowEveryoneRule);

using ( var mutex = new Mutex(false, mutexId, out createdNew, securitySettings) )
{
var hasHandle = false;
try
{
hasHandle = mutex.WaitOne(Timeout.Infinite, false);
using ( var fileStream = new FileStream("test.txt", FileMode.Truncate, FileAccess.Write, FileShare.Read) )
{
var buffer = Encoding.UTF8.GetBytes(text);
fileStream.Write(buffer, 0, buffer.Length);
fileStream.Close();
}
}
finally
{
if ( hasHandle )
mutex.ReleaseMutex();
}
}
}

public string Read ( )
{
var appGuid = "{FBE403EA-0D59-48D6-941D-E74E0F20B482}";
var mutexId = $"Global\\{{{appGuid}}}";

bool createdNew;

var allowEveryoneRule = new MutexAccessRule(new SecurityIdentif

Solution

Is there anything i can do in these methods to get any boost in performance?

Yes, You can reimplement them, using the 'memory-mapped files' (MMf) to solve both problems of parallel concurrency and access sharing.

using (MemoryMappedFile file = MemoryMappedFile.CreateFromFile("test.bin"))
using (MemoryMappedViewStream stream = file.CreateViewStream())
{
    Read(stream);
}


There are two ways of accessing data on a memory mapped file – using a MemoryMappedViewStream and an MemoryMappedViewAccessor.
The former is optimised to provide sequential access (e.g. when reading or writing a block of data) and the later is optimised for random access reading and writing.

Also, for 'production code' consider the following:

  • Performance can be greatly improved by always writing data sequentially and in large chunks



  • The chunk size is less important for small amounts of data (less than a kilobyte)



  • There is a cost to creating and disposing accessors, this cost becomes acceptable for reading files below 32K.



  • There is benefit of keeping hold of a stream, wich will remain significant.



  • When reading a large stream the performance is equivalent to raw memory access

Code Snippets

using (MemoryMappedFile file = MemoryMappedFile.CreateFromFile("test.bin"))
using (MemoryMappedViewStream stream = file.CreateViewStream())
{
    Read(stream);
}

Context

StackExchange Code Review Q#122918, answer score: 4

Revisions (0)

No revisions yet.