Best Practices for Complex Types (Arrays, Unions, etc.)

[wrbriggs]

First off, thank you for creating these examples, as they are a great starting point for working with Avro and Parquet in Spark!

One suggestion I have is that it would be really nice if this example could be updated to show best practices when using Avro schemas that contain complex nested types, unions and arrays. Spark does not seem to play well with these by default (e.g., https://issues.apache.org/jira/browse/SPARK-3601), and while it's possible to cobble something together by digging through mailing lists and JIRA tickets, it would be really helpful to have it officially documented somewhere.

[sryza]

I agree, though unfortunately don't have bandwidth in the near future to work on this. If you have any interest in taking this on, I'd be happy to review and merge it.

[prateek]

Ran into this while doing something today, 1 & 2 discuss the issue and solution well.