Skip to content

GH-3716: Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers - #3717

Open
yimingli-vmware wants to merge 1 commit into
apache:masterfrom
yimingli-vmware:fix-fixed-len-byte-array-position-clobber
Open

GH-3716: Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers#3717
yimingli-vmware wants to merge 1 commit into
apache:masterfrom
yimingli-vmware:fix-fixed-len-byte-array-position-clobber

Conversation

@yimingli-vmware

Copy link
Copy Markdown

Rationale for this change

Fixes #3716.

FixedLenByteArrayPlainValuesReader hands out Binary values that all
share one page-wide ByteBuffer, advancing its live position on every
readBytes() call. Binary.ByteBufferBackedBinary.getBytes() and
toStringUsingUTF8() (non-array-backed branch) called
value.limit(offset + length) directly on that same shared buffer before
capturing value.position(). Since ByteBuffer.limit() clamps position
down whenever position > newLimit, calling getBytes() on an earlier
value after later values have already advanced the buffer permanently
rewinds the buffer's live position -- corrupting every readBytes() call
that follows.

This surfaces as data corruption when reading a repeated (LIST)
FIXED_LEN_BYTE_ARRAY column with 2+ elements per row across 2+ rows:
record assembly stores each Binary and only materializes it once a full
row/group has been built, which is exactly the lazy-after-later-value
pattern that triggers the clamp. Each subsequent row reads back the
previous row's last-written element instead of its own (see #3716 for a
minimal standalone repro).

What changes are included in this PR?

  • Binary.ByteBufferBackedBinary.getBytes() and .toStringUsingUTF8() now
    duplicate() the buffer before adjusting position/limit, so the shared
    buffer's own position is never mutated.
  • Added regression tests to
    TestFixedLenByteArrayPlainValuesWriterReader that read values out of
    the order they were materialized, matching the lazy-consumption pattern
    from record assembly, and fail against unpatched 1.18.0.

Are these changes tested?

Yes -- two new tests
(testLazyGetBytesDoesNotCorruptSubsequentReadsDirectBuffer,
testLazyToStringUsingUTF8DoesNotCorruptSubsequentReadsDirectBuffer) in
parquet-column/src/test/java/org/apache/parquet/column/values/plain/TestFixedLenByteArrayPlainValuesWriterReader.java
fail on the unpatched code and pass with this fix. Also verified against
the full-file-roundtrip repro from #3716.

Are there any user-facing changes?

No API changes. This fixes a silent data-corruption bug introduced in
1.18.0; no user-facing behavior changes other than correct results.

…y-backed buffers

FixedLenByteArrayPlainValuesReader hands out Binary values that all
share one page-wide ByteBuffer, advancing its live position on every
readBytes() call. getBytes() and toStringUsingUTF8() on the
non-array-backed path called value.limit(offset + length) directly on
that shared buffer before capturing position. ByteBuffer.limit()
clamps position down whenever position > newLimit, so calling
getBytes() on an earlier value after later values have already
advanced the buffer permanently rewinds its live position -- corrupting
every readBytes() call that follows.

This surfaces as data corruption when reading a repeated
(LIST) FIXED_LEN_BYTE_ARRAY column with 2+ elements per row across 2+
rows: record assembly stores each Binary and only materializes it once
a full row is built, which is exactly the lazy-after-later-value
pattern that triggers the clamp. Each subsequent row reads back the
previous row's last-written element instead of its own.

Fix both methods to duplicate() the buffer before adjusting its
position/limit, so the shared buffer's own position is never mutated.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@yimingli-vmware yimingli-vmware changed the title Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers GH3716 - Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers Aug 13, 2026
@yimingli-vmware yimingli-vmware changed the title GH3716 - Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers GH3716: Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers Aug 13, 2026
@yimingli-vmware yimingli-vmware changed the title GH3716: Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers GH-3716: Fix data corruption in ByteBufferBackedBinary.getBytes() for non-array-backed buffers Aug 13, 2026
@dossett

dossett commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Nice catch. +1 (non-binding). I confirmed the new test code fails under 1.18.0

@yimingli-vmware

Copy link
Copy Markdown
Author

Nice catch. +1 (non-binding). I confirmed the new test code fails under 1.18.0

Hi @dossett, thanks for the confirmation. I wish we could have a patch release for addressing this issue soon, since we are blocked by the Jackson CVEs which are resolved in 1.18.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[parquet-java 1.18.0] data-corruption for reading NUMERIC arrays

2 participants