Vowpal Wabbit ignores linear terms, supports only interaction conditions

Question

Hi, a Vowpal Wabbit file with two namespaces, for example:

1.0 |A snow |B ski:10 0.0 |A snow |B walk:10 1.0 |A clear |B walk:10 0.0 |A clear |B walk:5 1.0 |A clear |B walk:100 1.0 |A clear |B walk:15 

Using -q AB, I can get the interaction conditions. Is there a way to preserve only interaction conditions and ignore linear terms?

In other words, the result of vw sample.vw -q AB --invert_hash sample.model right now is:

 .... A^clear:24861:0.153737 A^clear^B^walk:140680:0.015292 A^snow:117127:0.126087 A^snow^B^ski:21312:0.015803 A^snow^B^walk:28234:-0.010592 B^ski:107733:0.015803 B^walk:114655:0.007655 Constant:116060:0.234153 

I would like it to be something like this:

 .... A^clear^B^walk:140680:0.015292 A^snow^B^ski:21312:0.015803 A^snow^B^walk:28234:-0.010592 Constant:116060:0.234153 

The -keep and --ignore options do not give the desired effect, because they seem to be considered before creating quadratic expressions. Is it possible to do this with vw or do I need a custom preprocessing step that creates all the combinations?

Answer 1

John Langford (VW lead author) wrote: There is currently no good way to do this. The simplest approach would be to make --ignore applicable to the foreach_feature<> pattern in the source code.

You can use the trick to convert each original example into four new examples:

 1 |first:1 foo bar gah |second:1 loo too rah -1 |first:1 foo bar gah |second:-1 loo too rah 1 |first:-1 foo bar gah |second:-1 loo too rah -1 |first:-1 foo bar gah |second:1 loo too rah 

This makes quadratic functions completely interdependent with the label, but linear functions have zero correlation with the label. Therefore, soft l1-regularization should destroy linear functions.

I am skeptical that this will improve the performance sufficient for maintenance (hence design), but if you find it useful, let us know.

View original posts:

https://groups.yahoo.com/neo/groups/vowpal_wabbit/conversations/topics/2964 https://groups.yahoo.com/neo/groups/vowpal_wabbit/conversations/topics/4346