Skip to content

[feature](lance) Add index inspection TVFs and REST metadata - #66671

Draft
u70b3 wants to merge 14 commits into
apache:branch-4.1from
u70b3:feature/lance-index-inspection
Draft

[feature](lance) Add index inspection TVFs and REST metadata#66671
u70b3 wants to merge 14 commits into
apache:branch-4.1from
u70b3:feature/lance-index-inspection

Conversation

@u70b3

@u70b3 u70b3 commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: #66497

Related PR: #66637

Problem Summary:

This is PR2 of the Lance index lifecycle work and depends on #66637. Do not merge this PR before #66637. After PR1 is merged, this branch will be rebased onto branch-4.1 with only the four PR2 commits retained.

This PR adds bounded, read-only inspection of Lance logical and physical index metadata:

  • lance_indexes("table" = "catalog.db.table") exposes the strict logical view.
  • lance_index_entries("table" = "catalog.db.table") exposes physical diagnostic entries.
  • SHOW INDEX now uses the same provider-neutral logical snapshot for Directory and REST catalogs.
  • Directory reads schema, logical descriptions, and physical entries from one Dataset snapshot and never calls getIndexStatistics().
  • REST reads a pinned table version, bounded paginated index lists, and at most four rolling stats requests with one shared absolute deadline.
  • Both analysis and master execution enforce SHOW privilege before catalog initialization or provider access.
  • A bounded FE executor provides global/per-catalog admission, stable overload/timeout/failure tokens, metrics, and retains task ownership after caller timeout without claiming native cancellation.

The feature is independently releasable and strictly read-only. Lance index mutation remains disabled and no mutation route is added.

User-visible schemas:

  • lance_indexes: CatalogName, DatabaseName, TableName, IndexName, Columns, IndexType, nullable RowsIndexed, PhysicalEntryCount, Properties.
  • lance_index_entries: CatalogName, DatabaseName, TableName, IndexName, IndexUuid, Columns, nullable IndexType, nullable DatasetVersion, nullable ProviderStatus, MetadataConsistency.
  • Columns is a deterministic JSON string array so commas, quotes, and Unicode column names remain unambiguous.

Safety bounds include 256 logical names, 10,000 physical entries, 64 columns per index, 1,000 REST entries per page, 100 pages, four stats requests in flight, bounded external strings/properties, exact-case grouping, repeated-token detection, and all-or-error logical reads.

Release note

Add read-only Lance logical and physical index inspection through lance_indexes, lance_index_entries, and REST-backed SHOW INDEX.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test
      • Added Directory and REST regression coverage, including pagination and privilege denial.
      • Focused FE suites (all Lance tests plus SHOW INDEX) passed 125/125 with checkstyle enabled.
      • test_lance_show_index and test_lance_index_inspection passed in verify mode against a fresh local FE+BE cluster backed by the docker MinIO and REST mock environment.
      • BE ASAN meta_scanner.cpp.o compilation passed.
      • REST mock passed Python compilation and local describe/list pagination/stats/401 probes.
      • FE build.sh --fe package and BE ASAN build.sh --be both passed. The full Docker regression suite (all external suites) has not been run.

  • Behavior changed:

    • No.
    • Yes. Adds the two read-only TVFs and enables REST SHOW INDEX through the shared logical snapshot.

  • Does this need documentation?

Check List (For Reviewer who merge this PR)

u70b3 added 8 commits August 11, 2026 19:54
@u70b3
[feature](lance) Support SHOW INDEX for Lance Directory tables
b2e5594 
### What problem does this PR solve?

Issue Number: apache#66497

Related PR: apache#65730

Problem Summary: Lance external tables can contain logical vector and scalar indexes, but SHOW INDEX currently only handles internal OlapTable metadata and returns an empty result for Lance tables. Read authoritative logical index descriptions from one latest Directory Dataset snapshot, resolve field IDs from the same snapshot schema, and expose deterministic bounded metadata through the existing 13-column SHOW INDEX result without changing internal-table behavior.

### Release note

Support SHOW INDEX for Lance filesystem/Directory Catalog tables. Lance REST index inspection remains unsupported in this phase.

### Check List (For Author)

- Test: Unit Test and Regression test
- Behavior changed: Yes. SHOW INDEX now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs.
- Does this need documentation: Yes. Document the supported Directory-only scope and REST limitation.
@u70b3
[fix](lance) Guard SHOW INDEX Lance table cast and narrow exception
13a5fdb 
- Reject tables that are not LanceExternalTable with AnalysisException
  instead of failing on a raw ClassCastException.
- Narrow getLanceIndexRows to throw AnalysisException only.
- Add unit coverage for the non-Lance table guard.

### Release note

Support SHOW INDEX for Lance filesystem/Directory Catalog tables.
@u70b3
[fix](lance) Build vector index fixture deterministically
d327255
@u70b3
[fix](lance) Align logical index metadata bound
f48707d
@u70b3
[refactor](lance) Unify logical and physical index metadata snapshot
3e71f86
@u70b3
[feature](lance) Add index inspection table-valued functions
0dffcd7
@u70b3
[feature](lance) Support bounded REST index inspection
ec79454
@u70b3
[test](lance) Cover read-only index inspection
7167bfa
@hello-stephen

hello-stephen commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

u70b3 added 6 commits August 12, 2026 02:16
@u70b3
[style](lance) Fix index inspection test checkstyle violations
edf1ee5 
A full mvn test run (checkstyle validate phase) flagged two violations in
the PR2 test files that focused surefire runs had skipped:

- Restore the single lexicographic third-party import group in
  LanceRestIndexMetadataProviderTest.
- Replace unicode escapes with the literal character in
  LanceIndexTableValuedFunctionTest; the runtime value is unchanged.

### Release note

None.
@u70b3
[fix](lance) Accept index type vocabulary divergence in consistency c…
bca9858 
…hecks

End-to-end verification against a dataset built by the regression fixture
exposed false MISMATCH classifications that failed the whole logical view
with LANCE_INDEX_METADATA_INCONSISTENT. On the pinned SDK (9.1.0-beta.3),
a fresh IVF_PQ index reads back as umbrella VECTOR through getIndexes()
while describeIndices() derives IVF_PQ from the details, and a BTREE index
reads back as BTREE versus BTree. Strict string equality therefore
misclassified healthy indexes.

Compare normalized types (case and underscore insensitive) and accept the
observed umbrella families: physical VECTOR may pair with any vector
algorithm, physical SCALAR with any scalar algorithm. Internal types
(FRAGMENT_REUSE, MEM_WAL) are never family members and still classify as
MISMATCH, so the check stays fail-closed for genuinely corrupt metadata.
The physical TVF keeps reporting the raw physical type, so the regression
golden now shows VECTOR for the fixture dataset.

### Release note

None.
@u70b3
[fix](lance) Align Lance index regression suites with REST support
74e0306 
- test_lance_show_index still asserted the PR1-era rejection
  "SHOW INDEX is not supported for Lance REST catalogs", which this PR
  removes. Replace it with a golden assertion that locks the new REST
  SHOW INDEX output against the deterministic mock.
- test_lance_index_inspection's unprivileged user could not open a
  session on a fresh cluster because it held no grant on the default
  database, so the privilege-denial section never ran. Grant SELECT on
  regression_test like the sibling suite does; the Lance table
  privileges stay denied and remain what the assertions exercise.

Both suites passed in verify mode against a fresh local cluster backed
by the docker MinIO and REST mock environment.

### Release note

None.
@u70b3
[fix](lance) Log sanitized wrapper for index metadata failures
facd091 
The read executor intentionally converts provider failures to stable
tokens and drops the raw cause, and the generator logged only the table
identity, leaving production failures undiagnosable. Log the wrapper as
well: provider failures arrive already tokenized or wrapped with
sanitized messages by the catalog, and initialization failures were
already logged raw by makeSureInitialized, so this adds no new exposure.
The client-visible message is unchanged.

### Release note

None.
@u70b3
[docs](lance) Document serialized REST stats fan-out intent
4c05828 
Note on the provider class that the rolling stats window pipelines
submissions through the bounded executor while every provider call still
holds the namespace lock for its full duration. The window bounds
outstanding per-catalog admissions under one shared deadline; it does
not create parallel wire I/O, and holding up to MAX_STATS_IN_FLIGHT
slots for the read's duration is the intended trade-off.

### Release note

None.
@u70b3
[fix](lance) Exclude reserved system index entries from inspection
9121087 
The pinned 9.1.0-beta.3 list path returns reserved system index entries
unfiltered (load_indices filters only by index version), and a
fragment-reuse entry carries no fields. On datasets that went through
compaction, the __lance_frag_reuse entry would either fail structural
validation (empty field IDs) and brick every inspection surface, or
surface a system index that the logical model must exclude.

Drop the reserved names __lance_frag_reuse and __lance_mem_wal at the
normalization boundary of both the logical and the physical views.
Also lock the remaining internal-type path: a non-system name carrying
an internal index type still classifies as MISMATCH and fails the
strict logical view.

### Release note

None.
@u70b3 u70b3 mentioned this pull request Aug 13, 2026
Open
12 tasks
@u70b3

u70b3 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Author note — review-fix round pushed (7167bfa..9121087):

  • Aligned test_lance_show_index with REST support (the PR1 assertion expected the removed rejection); granted the inspection suite's unprivileged user SELECT on the default database so its denial section runs on a fresh cluster.
  • Fixed checkstyle violations in two PR2 test files (caught by the full mvn test gate).
  • Fixed a false-MISMATCH defect found by end-to-end verification: the pinned SDK reports umbrella VECTOR vs describe-derived IVF_PQ (and BTREE vs BTree), so strict type equality bricked the logical view for healthy datasets. The consistency comparison now normalizes case/underscores and accepts umbrella families; internal types never match.
  • Excluded reserved system index entries (__lance_frag_reuse, __lance_mem_wal) so compacted datasets stay inspectable.
  • The generator now logs the sanitized wrapper for tokenized failures; client-visible messages unchanged.

Verification: focused FE suites green with checkstyle; FE package and BE ASAN builds pass; both Lance regression suites pass in verify mode on a fresh local cluster (docker MinIO + REST mock). Full external regression suite not run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei will be requested when the pull request is marked ready for review yiguolei is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@u70b3 @hello-stephen