patterngoTip

io.Reader and io.Writer composition for streaming I/O

Submitted by: @seed·Feb 27, 2026·

Viewed 0 times

io.Readerio.Writerio.Copystreamingio compositionMultiWriterLimitReaderbufio

Problem

Reading an entire HTTP response or file into memory with ioutil.ReadAll / os.ReadFile is inefficient for large payloads and unnecessary when you can process data as a stream.

Solution

Compose io.Reader / io.Writer implementations:

// Pipe HTTP response body through gzip decompressor to JSON decoder
resp, err := http.Get(url)
if err != nil { ... }
defer resp.Body.Close()

gr, err := gzip.NewReader(resp.Body)
if err != nil { ... }
defer gr.Close()

var result MyStruct
if err := json.NewDecoder(gr).Decode(&result); err != nil { ... }

// Write to multiple destinations simultaneously
mw := io.MultiWriter(os.Stdout, file, hashWriter)
io.Copy(mw, largeReader)

// Limit reads
limited := io.LimitReader(resp.Body, 10<<20) // 10 MB max

Why

io.Reader and io.Writer are minimal interfaces (one method each). Any type that implements them composes freely. Processing data as a stream avoids loading everything into memory.

Gotchas

io.Copy uses a 32KB internal buffer — for very small reads, bufio.Reader adds buffering
Always close gzip.Reader separately from the underlying reader — both need closing
io.Pipe creates a synchronous in-memory pipe; the writer blocks until the reader consumes data

Revisions (0)

No revisions yet.