[Draft] Allow setting ClientOptions for all datafusion.object_store contexts

[nathschmidt]

Which issue does this PR close?

Closes #1082

Rationale for this change

This change adds a PyClientOptions wrapper around object_store.ClientOptions so that it can be instantiated and then used from Python.

What changes are included in this PR?

Introduces a new PyClientOptions struct and necessary wiring to use it.

Are there any user-facing changes?

New optional user-facing arguments for setting ClientOptions.

---

Given this is my first change - and I don't usually work in Rust - looking to get some confirmation on approach and acceptability of this PR.

Outstanding TODO's:

• Get some confirmation on overall approach
• Add necessary wiring for all contexts
• Add some simple tests

[nathschmidt]

Main question I have is does this need to be Arc<ClientOptions> rather than just ClientOptions?

[kylebarron]

This is somewhat related to #899.

Up for discussion with maintainers but I would personally argue for adopting my project https://docs.rs/pyo3-object_store, which exposes all possible client options automatically, and would give less of a maintenance burden here. You can see the exported client API, which matches these python docs .

[nathschmidt]

@kylebarron - this seems ideal to me. I'm happy to have a go at it - potentially as a separate API to ease any compatibility woes (i.e. deprecate datafusion.store, replace with datafusion.obstore or something - though tbh given datafusion-python is mostly major releases it should be an acceptable API change so long as there are updated examples).

While we're still on version 0.0.11 of object_store I can try and put together a demo branch with some cargo hackery. Unsure if there's a timeline on getting an official datafusion release depending on 0.0.12 out yet that'd obviously be a blocker to actually merging.

[kylebarron]

I think the current exports are datafusion.object_store? So we could deprecate that and use datafusion.store instead?

I've been thinking perhaps it makes sense to cut an 0.3 release of pyo3-object_store that uses object_store 0.11, so that projects (including my own) can start integrating pyo3-object_store.

It's mostly the python-facing obstore API that actually needed 0.12

[kylebarron]

I published new versions of pyo3-object_store for pyo3 0.23 and object_store 0.11: https://github.com/developmentseed/obstore/tree/main/pyo3-object_store#version-compatibility

[timsaucer]

#1143 updates us to DF48 which uses object store 0.12 so I think we may be soon unblocked on moving to pyo3-object_store!

[kylebarron]

Oh, I published versions of pyo3-object_store for compatibility with object_store 0.11, so that wasn't actually a blocker 😅