snippetrustModerate
How can I find out why this Rust program to echo lines from stdin is slow?
Viewed 0 times
thiscanwhyhowstdinprogramechoslowfindrust
Problem
Given the following Rust program:
How do I go about finding out what's keeping the Rust version from being faster? While I would appreciate if you just told me what to change to make it faster, I'm far more interested in how to find out what needs to be changed.
Edit: I tried changing
fn main() {
let mut reader = io::stdin();
for line in reader.lock().lines() {
match line {
Ok(l) => print!("{}", l),
Err(_) => continue,
}
}
}yes | program achieves 1.04MiB/s throughput, according to pv. The trivial C program below, which I fully realize does less, gives me a throughput of 141MiB/s.int main() {
int c;
while((c=getchar()) != EOF)
putchar(c);
}How do I go about finding out what's keeping the Rust version from being faster? While I would appreciate if you just told me what to change to make it faster, I'm far more interested in how to find out what needs to be changed.
Edit: I tried changing
lines() to chars() in the Rust version, to approximate the C version better, but it didn't seem to make any difference.Solution
I'm going to treat this as a code review question where the goal is to figure out how to write high-performance I/O code in Rust.
There are two critical steps to speed up Rust I/O:
Here's some code which drills down to
Of course, this means that you need to find the line breaks yourself, and be prepared for lines to be split over multiple buffers. Also, notice how I pass
Trying this with
…gives us a throughput of 527 MB/s.
There are two critical steps to speed up Rust I/O:
- Make sure the optimizer is turned on. Rust loops have terrible performance in debug mode. Turning the optimizer on should probably get you near 25 MB/sec or so, at least in my experience.
- Avoid line-based I/O, or anything else which uses
Stringvalues. For maximum performance, we want to work with raw byte buffers and avoid ever letting Rust callmallocorfree.
Here's some code which drills down to
fill_buf, which is about as low as we can go before losing generality:use std::io;
use std::io::{IoError, IoErrorKind};
fn main() {
let mut reader = io::stdin();
let mut buffer = reader.lock();
let mut writer = io::stdout();
loop {
let consumed = match buffer.fill_buf() {
Ok(bytes) => { writer.write(bytes).unwrap(); bytes.len() },
Err(IoError{kind: IoErrorKind::EndOfFile, ..}) => break,
Err(ref err) => panic!("Failed with: {}", err)
};
buffer.consume(consumed);
}
}Of course, this means that you need to find the line breaks yourself, and be prepared for lines to be split over multiple buffers. Also, notice how I pass
bytes.len() outside of the match block before trying to call buffer.consume(). This is necessary to placate the lifetime checker.Trying this with
pv:cat /dev/zero | target/release/throughput | pv > /dev/null…gives us a throughput of 527 MB/s.
Code Snippets
use std::io;
use std::io::{IoError, IoErrorKind};
fn main() {
let mut reader = io::stdin();
let mut buffer = reader.lock();
let mut writer = io::stdout();
loop {
let consumed = match buffer.fill_buf() {
Ok(bytes) => { writer.write(bytes).unwrap(); bytes.len() },
Err(IoError{kind: IoErrorKind::EndOfFile, ..}) => break,
Err(ref err) => panic!("Failed with: {}", err)
};
buffer.consume(consumed);
}
}cat /dev/zero | target/release/throughput | pv > /dev/nullContext
StackExchange Code Review Q#73753, answer score: 15
Revisions (0)
No revisions yet.