If you need a border around the tab (not a label), you should use this:
.tab-pane > .tab-header-area > .headers-region > .tab { -fx-background-color: red; -fx-padding: 20px; -fx-border-color: black; -fx-border-width: 1px; }

If you want to manipulate the tab container (where the label is), you need the following:
.tab-pane > .tab-header-area > .headers-region > .tab > .tab-container{ -fx-border-color: black; -fx-border-width: 1px; } .tab-pane > .tab-header-area > .headers-region > .tab { -fx-padding: 20px; -fx-background-color: red; }

UPDATE
The default value for the selected tab is as follows:
.tab-pane:focused > .tab-header-area > .headers-region > .tab:selected .focus-indicator { -fx-border-width: 1, 1; -fx-border-color: -fx-focus-color, -fx-faint-focus-color; -fx-border-insets: -4 -4 -6 -5, -2 -2 -5 -3; -fx-border-radius: 2, 1; }
And this is what it looks like:
.tab-pane > .tab-header-area > .headers-region > .tab { -fx-padding: 20px; -fx-background-color: red; } .tab-pane > .tab-header-area > .headers-region > .tab:selected { -fx-padding: 20px; -fx-background-color: red; -fx-border-width: 1px; -fx-border-color: black; } .tab-pane > .tab-header-area > .headers-region >.tab:selected .focus-indicator{ -fx-border-width: 0px; }

See the modena.css file (the default JavaFX stylesheet) for information on the things you need to change.
The font size will not change dynamically, you should take care of the font size with the listener by the size / width / height of the tab property (relative to the font size).
And there are many pseudo tags like .tab: selected.tab: top etc. Be aware of this if you want the default behavior with just a new design.
And finally, look at the css selectors, you missed the downstream selectors ('>'): http://www.w3schools.com/cssref/sel_element_gt.asp