patterncsharpMinor
Hex file converter
Viewed 0 times
hexfileconverter
Problem
I have written a C# file reader that reads a file converts the bytes of the file to hex and writes it out to another file. It works fine but it takes 7.2GB of memory when converting a 300MB file.
What can I do to this code to make it take less memory?
What can I do to this code to make it take less memory?
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace Hex__Reader
{
class Program
{
static void Main(string[] args)
{
OpenFile();
}
private static void OpenFile()
{
uint j = 0;
byte[] bytes = System.IO.File.ReadAllBytes(filename);
char[] hex = BitConverter.ToString(bytes).Replace("-", " ").ToCharArray();
//char[] hexchar = hex.ToCharArray();
System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false);
Console.WriteLine("Writing Bytes");
for (uint i = 0; i < hex.Length; i++)
{
if (j == 50)
{
writer.WriteLine();
writer.WriteLine();
j = 0;
}
else
{
writer.Write(hex[i]);
j++;
}
}
}
}
}Solution
Your problem is that you load the whole file into memory, convert each byte to three characters (that's the file size times 6), going through
To avoid holding all in memory, you should stream the data, converting and writing each chunk. This way, you'll only hold a fraction of the file at each time:
If (even when the buffer size is more optimal at 1024) you still have memory issues, it might be because of
Replace and ToCharArray.To avoid holding all in memory, you should stream the data, converting and writing each chunk. This way, you'll only hold a fraction of the file at each time:
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
using (FileStream fs = File.OpenRead(filename))
{
byte[] b = new byte[50];
while (fs.Read(b,0,b.Length) > 0)
{
writer.WriteLine(BitConverter.ToString(bytes).Replace("-", " "));
}
}
}If (even when the buffer size is more optimal at 1024) you still have memory issues, it might be because of
BitConverter.ToString(bytes).Replace("-", " ")... you could use some other methods to make to conversion, as suggested here, for example:int itemsInLineCounter = 0;
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
using (FileStream fs = File.OpenRead(filename)) {
byte[] b = new byte[1024];
int read;
while ((read = fs.Read(b,0,b.Length)) > 0) {
for (int i = 0; i NUM_OF_ITEMS_IN_LINE) {
writer.WriteLine();
itemsInLineCounter = 0;
} else {
writer.Write(' ');
itemsInLineCounter++;
}
}
}
}
}Code Snippets
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
using (FileStream fs = File.OpenRead(filename))
{
byte[] b = new byte[50];
while (fs.Read(b,0,b.Length) > 0)
{
writer.WriteLine(BitConverter.ToString(bytes).Replace("-", " "));
}
}
}int itemsInLineCounter = 0;
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(filename,false)) {
using (FileStream fs = File.OpenRead(filename)) {
byte[] b = new byte[1024];
int read;
while ((read = fs.Read(b,0,b.Length)) > 0) {
for (int i = 0; i < read; i++) {
writer.Write(bytes[i].ToString("X2"));
if (itemsInLineCounter > NUM_OF_ITEMS_IN_LINE) {
writer.WriteLine();
itemsInLineCounter = 0;
} else {
writer.Write(' ');
itemsInLineCounter++;
}
}
}
}
}Context
StackExchange Code Review Q#44457, answer score: 9
Revisions (0)
No revisions yet.