Currently, a large number of different data formats for storing machine learning data sets are in use, some of which will be reviewed below. I think it would be highly desirable if there is a certain convergence to a small number of standard file formats (maybe one or two), as has happened with images. Currently, there basically are two or three formats, jpeg for lossy compression, and gif/png for lossless compression. If such standard formats existed it would be much easier to exchange, use, or publish data sets. Instead of having to write a special file format reader (as is often the case at the moment), you could simply use one of the existing readers and concentrate on the actual machine learning task.

Of course, the main problem is to find a data format which is simple enough such that it is actually usable, and powerful enough to be usable. Machine learning data sets can usually be thought of as a number of data points consisting of some features, such that a two-dimensional array would be enough. However, there usually also exists some meta-information which you need to actually use the data set, for example, which features are input and output features. One could also think about including predefined splits for experiments in the data sets.

Now, a format like CSV (comma separated values) is very simple, but lacks meta-information like identifiers for features. On the other hand, something like XML would be powerful enough, but actually parsing an XML file from, say matlab, is much too complex. So the question is what we actually want.

Basically all existing formats focus on a two-dimensional array where columns are features and rows are data points. They mostly differ on the flexibility to include different data types, and on the amount of meta-information you can put in there.

There exist very simple formats (R, SVMlight, pyML, UCI, R, csv), formats which allow to add some meta-information (ARFF, orange), and very complex formats which may also span several files (delve), or require interaction with a database server (SQL).

So, the main question is what we want, after all. The simplest possible thing would be a space, comma, or tab-separated format. IMHO, the main problem is that you cannot "just use" such a data set as you first have to supply additional information like what the input and output features are. Also, the SVMlight family of formats, for example, can only deal with numeric features (no strings).

Very complex formats like delve pose a different problem. On the one hand the format is as complete as you could wish, even allowing you to define new subtasks for a given data set, but the format is almost too powerful, and I think most people won't be willing to invest in getting all of the information which is out there - in fact, that is how I have been dealing with these data formats. I would also definitely prefer a format where you have just one file, this makes distribution much easier.

So, currently the ARFF and orange formats seem the most promising, with ARFF maybe even being the better of the two. A typical file looks like this

% 1. Title: Iris Plants Database
%
% 2. Sources:
%      (a) Creator: R.A. Fisher
%      (b) Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
%      (c) Date: July, 1988
%
@RELATION iris

@ATTRIBUTE sepallength  NUMERIC
@ATTRIBUTE sepalwidth   NUMERIC
@ATTRIBUTE petallength  NUMERIC
@ATTRIBUTE petalwidth   NUMERIC
@ATTRIBUTE class        {Iris-setosa,Iris-versicolor,Iris-virginica}

@DATA
5.1,3.5,1.4,0.2,Iris-setosa
4.9,3.0,1.4,0.2,Iris-setosa
4.7,3.2,1.3,0.2,Iris-setosa
4.6,3.1,1.5,0.2,Iris-setosa
5.0,3.6,1.4,0.2,Iris-setosa
5.4,3.9,1.7,0.4,Iris-setosa
4.6,3.4,1.4,0.3,Iris-setosa
5.0,3.4,1.5,0.2,Iris-setosa
4.4,2.9,1.4,0.2,Iris-setosa
4.9,3.1,1.5,0.1,Iris-setosa

The question is whether we want to be able to store more information in there, like for example, predefined splits. I think from a comparability of experiments point of view, this would be a very nice feature. On the other hand, you could also supply the indices of the splits in an independent file.

A good think about the ARFF format is also that you could potentially extend it quite easily by introducing new @-commands.

I personally detest comma-seperated lists, in particular if they also allow spaces around the comma, since you cannot just split the tokens on a line, but first have to split by commas and then look for leading and ending commas as a second step, but I guess we don't live in a perfect world... .

Anyway, I think the ARFF format looks like a good starting point.

So, what do you think?

Short review of existing formats

ARFF

• Part of Weka (http://www.cs.waikato.ac.nz/~ml)
• Basically has a number of data points with defined data types.
• format: @header attribute definition @data the actual data...
• Supported data types: numeric, nominal, string, date
• Support for missing values (indicated by '?').
• Data presented as comma-seperated list (with white-spaces).
• Support for sprase data ('{index data, index data, index data...}')
• code exists for ARFF in Java (weka), R, matlab (spider)

orange style tab delimited

old version

• three-line header: names, types, options
• Supported data types: discrete, continuous, string, basket (collects the rest, python attributes
• data comes as tab-separated list

new version

• types are inferred from the data, or made explicit via "i#name" notation.
• there is only one class variable (the last one)
• support for missing values ('don't care' indicated by '?', 'NA', 'don't know ' by '~' and '*')
• sparse data possible by 'baskets'
• possible to store meta data, which is ignored by learning algorithm
• implementations: python (orange), C++ (via orange)

SVMlight

• support for sparse data
• only numerical features
• non-numerical features are passed as an information string
• implementations: SVMligth

libsvm

• the same as svmlight

pyML

sparse file format

• basically like SVMlight, with the ability to give an optional pattern ID

SQL Databases

• supports virtually any data set conceivable
• large overhead to access data set
• how do you store sql data bases portably?
• often stored as csv, but meta-data is missing there.

UCI

• basically a CSV format
• meta-information stored in additional file (only human readable)

statlib http://lib.stat.cmu.edu

• does this have a uniform file format?

delve

• data sets are actually tar archives which expand into a whole directory
• contains detailed information in an extra file (including ranges, for example)
• data file itself is space separated list (with line continuations using '')
• actually, its a whole set of (command-line) level tools to work with data sets and methods and to automatically evaluate data sets etc.

R

• actually comes with a large number of different formats, so let me consider the write.table format
• first line has column names, each next line starts with the row name and then the data in a space separated format
• no support for mixed string (?)
• no support for sparse data

Perhaps there are two separate issues:
1) which container format to store the data.
2) which standard fields or structures to use for meta-data and defining tasks.

1) this seems less important as convertors can always be written. Also in sensible code the important thing is 2), as 1) is easily hidden in replaceable functions/objects.

However, on the storage side, I'll mention that for huge data sets some will prefer binary formats (compressing a textual representation of millions of numbers seems like a bizarre thing to do). Useful container formats are HDF (a "does-everything" format supported by R, Octave, Matlab, ... see http://hdf.ncsa.uiuc.edu/), or XDR (an old, simple portable binary format http://www.faqs.org/rfcs/rfc1014.html).

2) seems important and difficult. Making data as simple to use as possible seems vital. Users who just want to do something simple, like try out a generic classification or regression problem should be able to ignore most of the metadata and pull out the "x" and "y" inputs they want with minimum fuss.

Hi iam23,

thanks for your post. I think you are right that it is slightly more important to find a sensible set of metadata to include. And you're right, this is important and difficult, but it is also exactly what we're trying to discuss here. Ideally, in the end we have developed some format and got some of the relevant projects commited to supporting this format.

You also raised two other issues, namely, how important it is to have one concrete format (compared to a number of "equivalent" ones, plus converters betweens the formats), and the need of a binary format.

I think it is important to really have one format. When there are more than one format, you'll typically end up having the data always in the "wrong" format for the program you're working on. Also, the conversion sounds practical in theory, but often it isn't. As an example for a domain which has many formats, take video encodings. There exist a really large number of different formats, and conversion is anything but trivial.

I'm also somewhat opposed to generic container classes. These tend to have too much power. Even if the data you are going to store is structured easily, having a format which is completely generic (like XML) is considerably more work, for example, just to ensure that your data is valid. The same holds for binary formats, which come with their own problems (for example, big-endian vs. small-endian). Until you really deal with large data sets, I don't think it is worth it.

Also, for complex formats, you really depend on having libraries available for everything as writing a reader or writer yourself is usually too complex.

So, you might already have guessed, I'm really in favor of something like tab-separated (or space or comma-separated) lists with a bit of meta-information. Such files can be read and written in almost everything, even in Excel.

I think ARFF is quite good in this respect.

So what would be a sensible starting point. I think you at least need names for the features, and their data types. Support for sparse data would also be nice.

Let's try and start from there... . What do you think?

I've read over this again.

I now agree there should be one common format at download sites. Otherwise one would have to know about all possible container formats. One way I might choose to implement reading them is to locally convert to .hdf with a single script and then read into Matlab/Octave/R/Python+SciPy with existing built-in libraries. Of course people are free to write native readers for each language too.

ARFF does look reasonable for all but huge datasets. I agree with your first post's suggestion that predefined splits should be added to the meta-data. There needs to be strong encouragement for people not to look at the test set until the very very end of their experiments.

I like the idea of using hdf as it is a library already interfacing to quite a number of languages (including C,C++,java) too.. The only downside is that it is not human readable and quite complex. The question is whether we need a human readable format? If we do, we could as well have converters to .hdf ... It would be interesting to know what the dataset-collecting people (like UCI) think of this and what their experiences with all these diverse data formats are.

Thinking of it again, I am very much in favor of using an ascii based format. Really most of the people doing ML are rather interested in and dealing with small/medium scale datasets. And there are always tradeoffs. One could think of binary extensions to ARFF (not sure if this is allowed), e.g:

@BINDATA
$*(%($*)%)*(Y%)*($Y%)H$^KL^K%JLJLKJ

But even if not I could live with this limitation, as one could allow the arff file to be .gz/.bz2 compressed.

Soeren

Hi Soeren,

yes, I also think that ascii based (UTF-8 is also okay with me ;) ) is fine. It also has the added advantage that it is possible to "just write out" a data file from whatever tool (say Visual Basic in Excel) without relying on a supporting library.

-M

Another data file format worth considering is RapidMiner's XML-based data file format.

RapidMiner supports its format as XML, zipped XML, and binary. RapidMiner can also handle Weka's ARFF and XRFF formats, the SPSS file format, configurable CSV file formats, all standard databases, Microsoft Excel sheets, text documents (e-mail, news, PDF, HTML, XHTML), time series formats, audio files, dbase, and many more. Hence RapidMiner can also be used as file converter for conversion from one file format to another.

Cheers, Frank

I was thinking about the amount of meta-data you would want to have... . While a format which "just contains the data" has a lot going for it in terms of simplicity, one big advantage of meta-data is that you might be able to automatically validate your method on the data set in a standardized fashion.

Ideally, you'll have a tool and could just tell it to run your method (which has to provide some standard interface) against the data in a standardized manner. The tool then takes care to extract the relevant splits into training and test, applies the method, computes the performance measure, and collects the result. The meta-data might also include achieved performance of other methods, such that you can very easily compare your methods against other approaches.

In contrast, if the meta-data were missing, you would have to make sure that you are really following the same protocol, apply the same amount of preprocessing, use the same splits, and so on. And you would have to write this stuff by hand every time.

So in summary, you should have meta-data about the precise splits, preprocessing (if any), and maybe also performance measures of existing methods.

@Frank: I really really don't want to start any holy wars here, but I think we should avoid XML in favor of line-based text formats. XML is a huge overhead in terms of complexity. You almost always need an extra XML library, you cannot parse XML just by yourself. Also, editing XML by hand is very tedious. Of course, you can use an XML-aware editor, but that means that you actually want to think in a simpler "meta-language". After all, the information we're about to store is structurally quite simple.

At our recent NIPS workshop, we proposed using ARFF as the file format for machine learning data. As expected, there was immediately an explosion of objections, which can be roughly categorized into 3 groups:

1. It is a container format, but does not define the semantics of the data, e.g. whether binary classification is {+1,-1} or {0,1}.
2. How does one deal with structured data, e.g trees and graphs?
3. It is an ascii format, which makes in less scalable.

Other options proposed include HDF and netCDF. I recently learned that netCDF-4 also supports HDF5 as a container format, so netCDF-4 seems to be the way to go. The biggest plus for netCDF is that it has interfaces in many languages, although not officially.