expose write options

[matko]

Which issue does this PR close?

Closes #1005 .

This is still a draft. Todo list:

• DataFrameWriteOptions
• CsvOptions
• JsonOptions
• TableParquetOptions

What changes are included in this PR?

Generic write options

dataframe.write_csv(...), dataframe.write_json(...) and dataframe.write_parquet now take an additional optional argument write_options, corresponding with DataFrameWriteOptions in datafusion. This is a dictionary with the following optional keys:

• "insert_operation": one of "append", "overwrite" or "replace", corresponding to InsertOp in datafusion.
• "single_file_option": a boolean
• "partition_by": list of strings, names of columns to hive partition on
• "sort_by": list of sort expressions

Csv

Todo..

JSON

Todo..

Parquet

Todo..

Are there any user-facing changes?

The api changes described above. These should be backwards compatible.

[kylebarron]

If you wanted, you could use an Enum here instead of a String, and then defer more of the validation to pyo3 instead of manually checking the string values below.

[matko]

Thanks for your comment. I did initially try to do this with an enum. Unfortunately, to use an enum in an object that derives FromPyObject, the enum itself has to also derive FromPyObject, but tagging a simple enum (no variants, just tags) with #[derive(FromPyObject)] results in error: cannot derive FromPyObject for empty structs and variants.

So this errors:

#[derive(FromPyObject)]
pub enum PyInsertOperation {
    Insert,
    Overwrite,
    Replace,
}

This might be where my pyo3 knowledge is lacking. Can you point me in the right direction on how to do this as an enum properly?

[kylebarron]

I think I usually separate out the enum but then implement the FromPyObject directly on that enum. It's simpler with better separation of concerns, I think. And if you ever need to use PyInsertOperation from multiple functions, then you can reuse the same implementation