SetYou must use the Orddefault instance for the item type.
If you want to use another instance Ord, the standard way to do this is to use a special wrapper newtypeand then write an instance Ordfor this:
newtype Y = Y { unY :: Vector } deriving Eq
instance Ord Y where compare = comparing ((y . unY) &&& (x . unY))
But since this comparison method is equivalent to how binary tuples are compared, KennyTM is the simplest solution here.