sonne ( on December 20, 2008, 12:45:55 ) I am actually all still in favour of ARFF. Why? Well it is clear from the beginning that we won't be able to satisfy all and every needs and that we need to start somewhere. ARFF is easily parse-able and we even intended its use only for small to medium scale datasets which (for now) make up 90% of the machine learning problems. Also note that the underlying datatype float32 or byte or int32 does not really matter as we are dealing with small scale data sets. Finally the data size does not have to be known upfront as we provide functions for reading the data (and it is small scale anyway so reading the file twice in the worst case won't be a problem). It is human readable (easy to read and parse for human beings) and the simplest tools (i.e., a text editor or viewer) will allow you spot problems or have a glance at the data. But I agree, for anything else there is hdf5 and if you look at its feature list it looks like it covers everything we need. That and given that it has strong support for C/C++ and java, matlab and octave, R and also recently python. As it features a portable library too it won't be too hard to convert to languages for which a C compiler exists. However this format is complex and still only a container but definitely features everything I could ever think of . No wonder as this format was developed for 20 years. So I would propose to use ARFF for now and later on define the same things for hdf5 for the most frequently used cases (to have a more efficient drop in replacement for ARFF). Yes we could easily provide converters too. Anyway as was suggested in the workshop we should open a forum (which already exists for a long time btw) to publicly discuss this. So we are happy about further comments. Oh well and if none appear we have to decide on something on our own which we thing is best...

mgashler ( on January 14, 2009, 19:55:09 ) What if we settled on two formats instead of just one? For example "Use ARFF if it meets your needs. If not, use hdf5." People would naturally tend to use the simpler format anyway, but in cases where they couldn't, it would also be clear where to turn next.

ong ( on January 22, 2009, 12:15:00 ) There is something in between, which is netCDF-4. Quoting the FAQ from Unidata: The National Center for Supercomputing Applications (NCSA) developed HDF and makes it freely available. HDF is a data model and extensible data format for self-describing files that was developed independently of netCDF. HDF supports both C and Fortran interfaces, and it has been successfully ported to a wide variety of machine architectures and operating systems. HDF emphasizes a single common format for data, on which many interfaces can be built. NCSA implemented software that provides a netCDF interface to HDF version 4. With this software, it was possible to use the netCDF calling interface to place data into an HDF4 file. HDF5 provides a richer data model, with emphasis on efficiency of access, parallel I/O, and support for high-performance computing. The netCDF-4 project has implemented an enhanced netCDF interface on the HDF5 storage layer to preserve the desirable common characteristics of netCDF and HDF5 while taking advantage of their separate strengths: the widespread use and simplicity of netCDF and the generality and performance of HDF5.

fracpete ( on January 22, 2009, 20:47:20 ) There is also a Java library available: http://www.unidata.ucar.edu/software/netcdf-java/ Though, netCDF-4 support is currently only alpha. I might have some time over the weekend and have a play with it (maybe I can integrate it in Weka). Cheers, Peter

sonne ( on January 22, 2009, 23:18:55 ) If I get things right netCDF-4 supports an ascii representation of the data too. And there is also a python and octave module... http://code.google.com/p/netcdf4-python/ http://ocgmod1.marine.usf.edu/mediawiki/index.php/NetCDF_toolbox_for_Octave So we have java, c/c++, fortran, python and octave. I guess thats enough no? I am not so sure yet if we want to require netCDF-4 or if netcdf is just enough (with -4 based on hdf5 for efficiency) at your option. Soeren

Post a Reply