StreamReader#read returns the wrong number of bytes and #close always raises

[sribalakumar]

Hi! Thanks for maintaining zstd-ruby.

I wanted to stream a large .tar.zst through Gem::Package::TarReader and tried Zstd::StreamReader (added in #59, still marked experimental). While doing that I ran into two bugs that block this use case outright, plus one spot where the behaviour doesn't line up with the usual Ruby IO convention. I've written all three up below, and there's a patch in #151 — though the direction is genuinely open, so do say if you'd prefer I approach it differently.

Environment: zstd-ruby 2.0.6, also reproduced on main @ d0adf3f; Ruby 3.3.11.

1. read(length) returns the wrong number of bytes (bug)

data = @io.read(length)    # length COMPRESSED bytes
@stream.decompress(data)   # returns an unrelated number of DECOMPRESSED bytes

The size of the return value isn't related to length at all — it tracks the compression ratio instead of what the caller asked for:

tiny = Zstd.compress("hello world\n" * 20_000)   # 240_000 bytes -> 44 compressed
Zstd::StreamReader.new(StringIO.new(tiny)).read(512).bytesize
# => 240000

big = Zstd.compress(ndjson)                      # ~840 KB -> ~20 KB compressed
r = Zstd::StreamReader.new(StringIO.new(big))
4.times.map { r.read(512).bytesize }
# => [0, 0, 0, 0]      (zstd is still filling an internal block)

The existing spec captures this too — read(10) currently returns 'a', then 'bcdef'.

Worth flagging that it's data-dependent: with incompressible input the compressed and decompressed sizes stay close, so read looks like it's working. It only misbehaves on compressible data, which would make it easy to miss.

2. EOF raises instead of returning nil (convention mismatch)

raise StandardError, "EOF" if @io.eof?

IO#read returns nil at EOF, so generic consumers won't expect an exception here. StandardError is also broad enough that a caller rescuing it to detect EOF would end up swallowing a genuine decompression failure too (the C ext raises RuntimeError).

3. close raises NoMethodError on every call (bug)

def close
  @io.write(@stream.finish)
  @io.close
end

This isn't occasional — it fails every single time close is invoked. StreamingDecompress only defines [:decompress, :decompress_with_pos], so there's no #finish to call, and @io is open for reading anyway, so writing to it would be wrong even if finish existed. Looks like it may have been copied over from StreamWriter#close.

Status

#151 has a patch that buffers decompressed output so length means decompressed bytes, returns nil at EOF, adds eof?, and fixes close. Full spec suite passes (88 examples).

It does change read's behaviour, so it's breaking for anyone relying on the current semantics — the existing spec's expectations change as part of the patch. Given the class is marked experimental and close has never worked, actual usage is probably small, but that's your call to make. If you'd rather not break read, I'm just as happy to add a separate IO-conformant class instead and leave StreamReader as it is. Happy to rework the PR either way, whichever you'd prefer.

[sribalakumar]

@SpringMT @Watson1978 — you've done most of the recent memory-safety work here, so I'd value your read on this if you have a moment.

For context on the use case: we use zstd-ruby at GitLab to read .tar.zst archives of security advisory data, and we want to stream them through Gem::Package::TarReader instead of decompressing each archive into memory. Archives are tens of MB compressed today and projected to reach ~200 MB, which is what pushed us to look at StreamReader.

• How we use the library today — full decompress, no streaming: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240026/diffs
• The limitations that blocked us and what we'd need instead: https://gitlab.com/gitlab-org/gitlab/-/work_items/602885

The concrete question for you and : is the breaking change in #151 acceptable, or would you prefer a separate IO-conformant class that leaves StreamReader as-is? Happy to go either way.