HiveBrain v1.2.0
Get Started
← Back to all entries
patterncsharpMinor

Hex file converter

Submitted by: @import:stackexchange-codereview··
0
Viewed 0 times
hexfileconverter

Problem

I have written a C# file reader that reads a file converts the bytes of the file to hex and writes it out to another file. It works fine but it takes 7.2GB of memory when converting a 300MB file.

What can I do to this code to make it take less memory?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Hex__Reader
{
    class Program
    {
        static void Main(string[] args)
        {
            OpenFile();
        }

        private static void OpenFile()
        {
            uint j = 0;
            byte[] bytes = System.IO.File.ReadAllBytes(filename);
            char[] hex = BitConverter.ToString(bytes).Replace("-", " ").ToCharArray();
            //char[] hexchar = hex.ToCharArray();
            System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false);
            Console.WriteLine("Writing Bytes");
            for (uint i = 0; i < hex.Length; i++)
            {
                if (j == 50)
                {
                    writer.WriteLine();
                    writer.WriteLine();
                    j = 0;
                }
                else
                {
                    writer.Write(hex[i]);
                    j++;
                }
            }
        }
    }
}

Solution

Your problem is that you load the whole file into memory, convert each byte to three characters (that's the file size times 6), going through Replace and ToCharArray.

To avoid holding all in memory, you should stream the data, converting and writing each chunk. This way, you'll only hold a fraction of the file at each time:

using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
    using (FileStream fs = File.OpenRead(filename)) 
    {
        byte[] b = new byte[50];

        while (fs.Read(b,0,b.Length) > 0) 
        {
            writer.WriteLine(BitConverter.ToString(bytes).Replace("-", " "));
        }
    }
}


If (even when the buffer size is more optimal at 1024) you still have memory issues, it might be because of BitConverter.ToString(bytes).Replace("-", " ")... you could use some other methods to make to conversion, as suggested here, for example:

int itemsInLineCounter = 0;
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
    using (FileStream fs = File.OpenRead(filename)) {
        byte[] b = new byte[1024];
        int read;

        while ((read = fs.Read(b,0,b.Length)) > 0) {
            for (int i = 0; i  NUM_OF_ITEMS_IN_LINE) {
                    writer.WriteLine();
                    itemsInLineCounter = 0;
                } else {
                    writer.Write(' ');
                    itemsInLineCounter++;
                }
            }
        }
    }
}

Code Snippets

using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
    using (FileStream fs = File.OpenRead(filename)) 
    {
        byte[] b = new byte[50];

        while (fs.Read(b,0,b.Length) > 0) 
        {
            writer.WriteLine(BitConverter.ToString(bytes).Replace("-", " "));
        }
    }
}
int itemsInLineCounter = 0;
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
    using (FileStream fs = File.OpenRead(filename)) {
        byte[] b = new byte[1024];
        int read;

        while ((read = fs.Read(b,0,b.Length)) > 0) {
            for (int i = 0; i < read; i++) {
                writer.Write(bytes[i].ToString("X2"));
                if (itemsInLineCounter > NUM_OF_ITEMS_IN_LINE) {
                    writer.WriteLine();
                    itemsInLineCounter = 0;
                } else {
                    writer.Write(' ');
                    itemsInLineCounter++;
                }
            }
        }
    }
}

Context

StackExchange Code Review Q#44457, answer score: 9

Revisions (0)

No revisions yet.