How are temporary files created in S3 when calling the driver?

[peidaqi]

Can someone here explain how the temporary files are created in S3 when calling the driver? It seems:

1. When calling spark.read, a temporary file folder will be created with a file of length 1
   When calling any functions to evaluate the returned Dataframe, another temporary file folder will be created with the actual data
   What is the file created in first step?
2. The temporary files created seems to be in plain text without schema. Is there anyway to make that parquet or export parquet files with schema?

Thanks!

[peidaqi]

I figured out. It's actually a sliced CSV by default and read using RedshiftInputFormat class.

But the question still holds - since Redshift now supports UNLOAD directly into parquet (https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html), can we just use it instead to avoid the many escaping problems with CSV?