Implementing SVM, scikits learn to reduce runtime, faster svm

Question

I work with scikit-learn to create some predictive models with SVM. I have a data set with about 5,000 examples and about 700 functions. I cross-check 5 times using the 18x17 setting on my training set, and then using the optimal parameters for my test set. runs go much longer than I expected, and I noticed the following:

1) Some individual SVM training iterations seem to take only one minute, while others may take up to 15 minutes. Is this expected using different data and parameters (C and gamma, I use the rbf core)?

2) I'm trying to use 64-bit python for Windows to take advantage of extra memory, but all my python processes seem to end in 1 gig in my task manager, I don't know if this has anything with runtime.

3) I used 32bit before and worked on about one dataset, and I remember (although I didn’t save the results), but it was pretty fast. I used a third-party scikit-learn assembly for 64-bit windows, so I don’t know if I should try this on 32-bit python? (source http://www.lfd.uci.edu/~gohlke/pythonlibs/ )

Any suggestions on how I can reduce lead time are welcome. I suppose that reducing the search space of my grid search will help, but since I'm not even sure about the range of optimal parameters, I would like to keep it as large as I can. If there are faster implementations of SVM, let me know and I can try them.

Application: I came back and tried again to launch the 32-bit version. For some reason this is much faster. It took about 3 hours to get to the 64-bit version before 16 hours. Why such difference?

Answer 1

1) It is expected that the small gamma and small regularization will choose more support vectors, so the model will be more complex and longer suitable.

2) There is a cache_size argument that will be passed to the libsvm base library. However, depending on your data, libsvm may or may not use all available cache.

3) I do not know. I am running longer experiments on both platforms, please report your findings to the project mailing lists. This may merit further study.

First, check that you have normalized your functions (for example, remove the middle and scale functions by deviations if your data is a dense numpy array). For sparse data, simply scale the functions (or use the TF-IDF transform for text data, for example). See Document Preprocessing Section.

Then you should start with a coarse grid (with large logarithmic steps), say a 3x3 grid, and then focus on interesting areas, restarting the 3x3 grid in that area. In general, the params C x gamma SVM mesh is pretty smooth .

Answer 2

If you can afford it, consider using LinearSVC: libsvm-based SVCs have learning difficulty between O(n_features * n_samples^2) and O(n_features * n_samples^3) , while LinearSVC (liblinear-based) has O(n_features*n_samples) learning complexity O(n_features*n_samples) and the complexity of testing O(n_features) .

Answer 3

SGD is very fast, but 1) only linear, not rbf, 2) parameters alpha eta0 ... which I have no idea how to vary: O. to expert O. Grisel.

On a 32-bit, 64-bit python (what equipment, what version of py?), I have no idea, but it may be worth the general question about SO - there should be control kits. Can you see CPU usage> 90%, count garbage collections?