Implementing faster StringBuilder that only works with ASCII strings

I wrote a StringBuilder that works only with ASCII strings. the current append performance is almost identical to BCL StringBuilder append performance. I want some tips on how to make it faster, and general tips about the code. I did this for educational purposes.

Implementation:

public sealed class FastStringBuilder
{
    private byte[] chars;

    public int Length { get; private set; }

    public FastStringBuilder(int capacity)
    {
        chars = new byte[capacity];
    }

    public FastStringBuilder()
    {
        chars = new byte[16];   
    }

    private void GrowBy(int length)
    {
        var temp = chars;
        chars = new byte[length];
        Array.Copy(temp, 0, chars, 0, Length);
    }

    public unsafe void Append(string str)
    {
        if (str.Length + Length >= chars.Length)
            GrowBy((str.Length + Length) * 4);
        else if(Length == chars.Length)
            GrowBy(chars.Length*2);

        fixed (char* p = str)
        {
            for (int i = 0; i < str.Length; i++)
                chars[Length++] = (byte)(p + i);
        }
    }

    public override string ToString()
    {
        unsafe
        {
            fixed (byte* bp = chars)
            {
                return new string((sbyte*)bp,0,Length);
            }
        }
    }
}

Performance tests spec:

public static void StringBuilderTest()
{
    Stopwatch sw = Stopwatch.StartNew();

    StringBuilder builder = new StringBuilder();

    for (int i = 0; i < 1000000; i++)
    {
        builder.Append("F0000000000000000000000000000000000000000000000000000" );
    }

    var result = builder.ToString();

    Console.WriteLine(sw.Elapsed);
}

Results:

Compiled in release mode.

• FastStringBuilder: ~0.36 seconds

• StringBuilder: ~0.27 seconds

• FasterStringBuilder (ChrisWue's implementation): ~0.17 seconds

EDIT

mjolka spotted that FastStringBuilder had a bug.

this code was causing the bug:

```
fixed (char* p = str)
{
for (int i =

This is very, very buggy. This is the most reliable way I could find to reproduce the bug.

static void Main(string[] args)
{
    var sb = new FastStringBuilder();
    var chars = new[] { 'a', 'b', 'e', 'd' };
    sb.Append(new string(chars));
    Console.WriteLine(sb.ToString());
}

Note how args is not used. Now run the program with different arguments:

$ ./Test
dfhj
$ ./Test abc
OZ?'
$ ./Test abcdef
"-~s

StackExchange Code Review Q#64913, answer score: 5