disableSelection is your friend here.
.disableSelection () is undocumented (at the moment). This is useful for creating text elements or elements that contain text, not text of your choice. For example, if you have a draggable item, you may not want the text selection to occur when the user drags the item.
You might want to create an assistant so that it is obvious that something is being dragged.
Updated demo
source share