HiveBrain v1.2.0
Get Started
← Back to all entries
patterngoMinor

Lexer for an experimental tiny language in Go

Submitted by: @import:stackexchange-codereview··
0
Viewed 0 times
experimentaltinylanguagelexerfor

Problem

I am implementing a lexer for an experimental language by the name "Phoenix", the language supports four primary arithmetic expression for integers only (add, subtract, multiply, divide), variable assignment and print statement.

Typical input:

# this is a comment
value = 1 + 7 / 4 * (30)
print('Value = ', value)


lexer.go:

```
package lexer

import (
"bytes"
"fmt"
"io"
"os"
"phoenix/log"
"phoenix/token"
"unicode"
)

// Lexer type for our lexer
type Lexer struct {
SourceFileName string
fh *os.File
UnScannedChar byte
CurrentLine, CurrentCol uint64
}

// New sets up the lexer
func (l *Lexer) New(fileName string) {
tmpFh, err := os.Open(fileName)
l.fh = tmpFh

if err != nil {
log.Error(fmt.Sprintf("couldn't open file '%v'", fileName))
os.Exit(0)
}

l.SourceFileName = fileName
l.CurrentLine = 1
}

// NextChar updates Lexer.unScannedChar
func (l *Lexer) NextChar() (err error) {
tempChar := make([]byte, 1)
_, er := l.fh.Read(tempChar)

if er != nil {
if er == io.EOF {
err = io.EOF
return
}
log.Error(fmt.Sprintf("error while reading from file '%v'",
l.SourceFileName))
os.Exit(0)
}
l.UnScannedChar = tempChar[0]

if l.UnScannedChar == '\n' {
l.CurrentLine++
l.CurrentCol = 0
} else {
l.CurrentCol++
}
return
}

func (l *Lexer) isIdentifierStart() bool {
return unicode.IsLetter(rune(l.UnScannedChar)) ||
l.UnScannedChar == '_'
}

func (l *Lexer) isIdentifierPart() bool {
return l.isIdentifierStart() || unicode.IsNumber(rune(l.UnScannedChar))
}

func (l *Lexer) scanInteger() (newToken token.Token) {
var buffer bytes.Buffer

buffer.WriteString(string(l.UnScannedChar))
l.NextChar()

for unicode.IsDigit(rune(l.UnScannedChar)) {
buffer.WriteString(string(l.UnScannedChar))
l.NextC

Solution

This will not be a full review, but rather a couple of improvements you can make. Firstly, why do all those string []byte conversions when writing into a buffer? You can just do e.g.

buffer.Write([]byte{l.UnScannedChar})


instead of

buffer.WriteString(string(l.UnScannedChar))


Also, since you're only using file's Read and Seek methods, why not change os.File to io.ReadSeeker?

Code Snippets

buffer.Write([]byte{l.UnScannedChar})
buffer.WriteString(string(l.UnScannedChar))

Context

StackExchange Code Review Q#133305, answer score: 2

Revisions (0)

No revisions yet.