agijsberts ( on September 11, 2008, 14:09:16 ) At the moment I am writing a framework for online and batch learning experiments. The emphasis of the framework is first of all to facilitate implementing and evaluating algorithms in a easy (i.e. Matlab-like) way. Therefore I have opted for Python + SciPy. However, I am currently stuck on a small problem in my design. Although I am not sure whether this is the proper place to ask for comments (and the forum seems quite silent...), I just give it a shot. My problem can be split in two distinect (but related) issues: Algorithms and evaluation methods tend to produce several byproducts. Two examples: the LSSVM algorithm (when using explicit matrix inversion) can calculate the LOO error for free; when testing an online method on a training/testing split, one obtains an error on both the training data and the testing data. It would be a waste to simply disregard this data, but I find it hard to capture this extra 'meta'-information in a nice way in my design. Algorithms may use certain extra information, if available, to improve their working. For instance, a Perceptron may opt to train using the same Loss function as will be used for evaluation. Further, this Loss function can depend on the type of problem (i.e. classification or regression). How to provide algorithms with this information, without overloading the signature of the interface with optional and possibly unnecessary parameters? I have been looking at other frameworks, but did not find any hints to what a proper solution might be. At the moment I am thinking of solving this by passing around an Context object, which would basically be a container of experiment related meta-data. This could solve both problems at the same time: the evaluation class fills in meta-data regarding the experiment (problem type, loss function for error measure, etc.) which can be used by the algorithm. Further, the algorithm can add its meta-information in the context, making it accessible to the evaluation class. Has any of you encountered similar difficulties in writing generalized ML toolboxes? Any advices?

Post a Reply