The following is a bit wasteful, but it works. The idea is to simply watch for cases where the Listable Plus attribute puts the same pt in all the elements of the list (i.e. the starting point), and then pull it back. First, define a function to add pt objects:
SetAttributes[ptPlus, {Orderless}] ptPlus[pt[pa : {_, _}], pt[pb : {_, _}], r___] := ptPlus[pt[pa + pb], r]; ptPlus[p_pt] := p;
Then we make sure that any Plus that includes pt maps to ptPlus (associate the rule with pt).
Plus[h___, a_pt, t___] ^:= ptPlus[h, a, t];
The above rules mean that: {x0,y0}+pt[{x1,y1}] will be expanded from {x0+pt[{x1,y1}],y0+pt[{x1,y1}]} to {ptPlus[x0,pt[{x1,y1}]],ptPlus[y0,pt[{x1,y1}]]} . Now we just make a rule to convert this to pt[{x0,y0}]+pt[{x1,y1}] (note the deferred condition, which checks that pt are equal):
{ptPlus[x__], ptPlus[y__]} ^:= Module[{ ptCases = Cases[{{x}, {y}}, _pt, {2}]}, ptCases[[1]] + pt[Plus @@@ DeleteCases[{{x}, {y}}, _pt, {2}]] /; Equal @@ ptCases]
A more opaque, but slightly more thorough version, which is easier to generalize to higher sizes:
ptPlus /: p : {_ptPlus, _ptPlus} := Module[{ptCases, rest, lp = ReleaseHold@Apply [List, Hold[p], {2}]}, ptCases = Cases[lp, _pt, {2}]; rest = Plus @@@ DeleteCases[lp, _pt, {2}]; ptCases[[1]] + pt[rest] /; And[Equal @@ ptCases, VectorQ@rest ]]
This whole approach, of course, will lead to terribly subtle errors when {a+pt[{0,0}],a+pt[{0,b}]} /. {a -> pt[{0,0}]} {a+pt[{0,0}],a+pt[{0,b}]} /. {a -> pt[{0,0}]} will evaluate pt[{0,0}] when c==0 and {pt[{0,0}],pt[{0,c}]} otherwise case ...
HTH, the guy said to himself ...