feat: add plan_splits function for distributed compute

[hamersaw]

Adding a plan_splits function to the Scanner to facilitate a single solution for partitioning Lance dataset for efficient distributed compute. This function (1) filters the dataset (using index looking / delete vectors) producing a mapping of fragment IDs to valid row ranges and (2) bin packs these fragment rows ranges into "splits" that target a configurable partition size (in rows count or bytes).

Perhaps the two most important aspects to align on are:

(1) Using a Splits enum that returns different "split types". In the case of a filterable query this returns a Vec<FilteredReadPlan> where each can be fed to the new execute_filtered_read_plan function to read (without incurring traditional read overhead through index lookup , deletion vector application, etc). In every other case (currently nearest / vector search) we return a list of Fragments (existing Spark partition method). This is meant to be a sane default that will be improved upon in the future. For example, we may want to partition these search types based on index files rather than fragment-level boundaries, etc.

(2) Removing the current FilteredReadPlan and making the existing FilteredReadInternalPlan the default. The differentiating factor between these is the latter stores row ranges (ie. Range<u64>) and the former a bitmap of row indexes. IIUC the intuition is that for network transfer a bitmap will be more efficient so we should use that user-facing. IMO we use row ranges in our internal APIs and so the bitmap is ONLY useful if we are network transfering "splits" AND and bitmap representation is smaller. You can do a simple calculation to understand if the serialization will be smaller (ex. # ranges * 2 * bytes per range index <> # of rows / 8). So rather than forcing the bitmap conversion on this API, we can quickly identify in the serialization logic if it makes sense to use bitmap or row range and perform that logic inline. Additionally, this is something we can punt on for now and just work with row ranges until it becomes a problem.

[codecov]

Codecov Report

❌ Patch coverage is 84.92707% with 93 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/scanner.rs	84.62%	65 Missing and 28 partials ⚠️

📢 Thoughts on this report? Let us know!