Should CSVY support be included

[mlell]

I am working as a bioinformatician in a mixed team of biologists and mathematicians. I found it not possible to make everyone conform to a similar set of tools for data analysis, e.g. whether to use base R, data.table, readr + tidyverse, python, etc. Problems with badly formatted data, undescribed column names and encoding are common.

I feel that using the frictionless standard to describe data might be a way out because it is a small addition to CSV that can be easily grasped by non-technical people at least at a basic level (not properties like "profile" or "encoding", but things like "title" and "fields". Furthermore, CSV is readable regardless of which data analysis style you have.

However, using a data package will be overkill in most situations where only one table is involved or it is not clear yet whether data will be given to someone else. A remedy for this would be the CSVY format, where the tabular-resource data structure is prepended to the csv file as a YAML header, possibly prefixed by a hash as a comment character.

• The header is part of the file and therefore cannot be lost by careless file copying.
• It spares users the burden of thinking about data packages, and enables them to start with just creating a data resource, which is an easier starting point for novices
• It increases interoperabilty with other data parsing packages, e.g. data.table.

CSVY is understood by data.table::fread(). There is a package csvy but it is abandoned since 2018.

Would you be open to include the possibility of reading and writing CSVY into this package? I could write it up but wanted to know if you consider this within the scope of this package

[peterdesmet]

Hi @mlell, I have explored the CSVY format a while ago myself for another use case (including how it is handled by csv readers). On face value it looks promising and keeps data and metadata together. But I eventually abandoned it because of the lack of support (both the format and applications). Although frictionless requires an extra file, there is at least a lot of support and community engagement, and can be expanded upon better. So I won't support CSVY for this package.

Your use case might be supported by the git2rdata package, which allows you to better control incoming data. It does also rely on an external metadata file.