Scanner in Scala

I wanted to implement Java's Scanner in Scala. The goal for this class is to:

• Implement a Scala collection interface (probably Iterator[String]?) so I can access all the Scala collection goodies like map, .toList etc.

• Have better performance than java.util.Scanner

Please review my code for bugs/design advice/performance advice:

import java.io._
import java.util.StringTokenizer
/**
 * Scala implementation of a faster java.util.Scanner
 * See: http://codeforces.com/blog/entry/7018
 */
class Scanner(reader: BufferedReader) extends Iterator[String] {
  def this(inputStreamReader: InputStreamReader) = this(new BufferedReader(inputStreamReader))
  def this(file: File) = this(new FileReader(file))
  def this(inputStream: InputStream) = this(new InputStreamReader(inputStream))
  def this(str: String) = this(new ByteArrayInputStream(str.getBytes))

  private[this] var tokenizer: StringTokenizer = _

  private[this] def nextTokenizer() = {
    while(tokenizer == null || !tokenizer.hasMoreTokens) tokenizer = new StringTokenizer(reader.readLine())
    tokenizer
  }

  def nextLine() = {
    tokenizer = null
    reader.readLine()
  }

  override def hasNext = nextTokenizer().hasMoreTokens

  override def next() = nextTokenizer().nextToken()

  def nextInt() = next().toInt
  def nextLong() = next().toLong
  def nextDouble() = next().toDouble
}

Benchmarks here

While I would not say the code is awful, there are certainly a few areas that could stand to be cleaned up. I am not going to post modified code because I think there are areas where the "right thing to do" is going to depend on the intended use.

First of all, your description implies that this will be used as a library class in your code. Library code should be held to a higher standard than other code, and the higher standard certainly means proper documentation including full ScalaDoc and declared return types for all public methods. A proper test suite would also be good. You will thank yourself later.

Second, there is a problem in nextTokenizer - it naively keeps reading lines from the reader without checking for null (i.e. the end of file condition). This test code will show you what I mean:

object ScannerTest {
  def main(args: Array[String]) {
    val test = new Scanner("ab cd")
    test.foreach((s) => println("[ScannerTest$.main] " + s))
  }
}

Fixing this is going to require some thought as to what the code should do, but the current implementation of hasNext() is going to have to be changed either way. Some would say that nextTokenizer() should return Option[StringTokenizer] and then hasNext() would look something like the code below. Note that if we get a tokenizer, we know it has tokens because nextTokenizer() checks for that and throws away empty lines.

def hasNext: Boolean = nextTokenizer() match {
  case None => false
  case Some(t) => true
}

But at this point, it should use the proper Scala idiom and just do this.

def hasNext: Boolean = nextTokenizer().nonEmpty

Also, the "next" value methods need to be rethought. Currently, nextLine() will return null at end of file, and the rest will throw an exception. None of that seems like a good idea.

Finally, I wonder about the constructor taking a String. It would seem simpler to use a StringReader. The way the code is now: string converted to bytes, wrapped in stream, chains to a second constructor where it gets wrapped in an input stream reader, chains to a third constructor where it gets wrapped in a buffered reader, where it finally chains to the default constructor.

It may be that the JRE's JIT compiler will reduce either approach to something equivalent, but I think something like the code below would be easier to follow.

def this(str: String) = this(new BufferedReader(new StringReader(str)))

StackExchange Code Review Q#99141, answer score: 9