I am afraid that you cannot make every line contained in the rectangle using border
in the display:inline
element.
But the workaround that works is to use box-shadow
.
span { line-height: 32px; box-shadow: 0 0 0 1px black; }
jsfiddle
Below is a screenshot from FireFox:

As Jan said, it's better to use box-decoration-break
According to canIuse , this should work for all latest browsers except IE:
span { border: 1px solid black; -webkit-box-decoration-break: clone; box-decoration-break: clone; }
updated jsFiddle
source share