feat: integration s3 with arrow filesystem

[MisterRaindrop]

I have implemented Arrow FileSystem to access S3, but I'm still not sure if it meets the requirements.

There are still task or question to complete for the current PR, and it is not ready for merging yet.

Question:
Currently, the object storage options include Azure, AWS, and GCS. I have chosen AWS as the implementation for now is ok?

Task:
I need to deploy MinIO to facilitate testing access to S3, but I'm not sure where it would be best to set it up?

[wgtmac]

Thanks for adding this!

I have chosen AWS as the implementation for now is ok?

Yes, I believe this is worth doing. I supposed to reuse ArrowFileSystemFileIO by passing an arrow::FileSystem of S3 which is supported by Arrow if built with ARROW_S3=ON. But I haven't explored it yet.

I need to deploy MinIO to facilitate testing access to S3

There is a related discussion with regard to minio's recent license change: https://lists.apache.org/thread/vnw9jonmfcsz6bwojhfch1nmywyl50h3. I'm not sure if there is any good alternative to test it.

[MisterRaindrop]

I recommend using MinIO. It is relatively stable and suitable for the current project development phase. Once the community reaches a consensus, the cost of replacing MinIO will not be high.

[wgtmac]

I think it is fine to use minio at this moment to unblock us. Let me know what you think on my proposed approach above. We might also need to add a FileIO registry to provide default implementation on us and enable users to override their own implementations of s3 and others. The key in the FileIO registry can be associated with table property io-impl.

[wgtmac]

We may also need to add top-level CMake options like ICEBERG_S3, ICEBERG_AZURE, ICEBERG_GCS to mirror Arrow equivalents and vendor Arrow with ARROW_S3=ON. This might be tricky in the CI because we need to deal with Arrow's dependency on AWS SDK and others.

[zhjwpku]

I recommend using MinIO. It is relatively stable and suitable for the current project development phase. Once the community reaches a consensus, the cost of replacing MinIO will not be high.

FYI, there is a PR to replace MinIO with RustFS, apache/iceberg#14928

[MisterRaindrop]

Yes, I believe this is worth doing. I supposed to reuse ArrowFileSystemFileIO by passing an arrow::FileSystem of S3 which is supported by Arrow if built with ARROW_S3=ON. But I haven't explored it yet.

ArrowFileSystemFileIO is ok, I referenced MakeLocalFileIO and implemented a simple MakeS3FileIO interface using arrowfilesystem.

I think it is fine to use minio at this moment to unblock us. Let me know what you think on my proposed approach above. We might also need to add a FileIO registry to provide default implementation on us and enable users to override their own implementations of s3 and others. The key in the FileIO registry can be associated with table property io-impl.

you mean this io-impl=org.apache.iceberg.aws.s3.S3FileIO ?

It's equivalent to setting the io-impl string in the catalog's properties. Then, RestCatalog the FileIORegistry looks up the implementation in the io-impl map. Is that roughly how it works? If so, I can try implementing some simple code to see if it's correct.

[wgtmac]

Yes, I think it looks reasonable.

[MisterRaindrop]

The current code is only simple implemented. Could you help me check it is ok?
@wgtmac