In your case, yours $randomElemwill have the same scope as its parent (directive container). Therefore, it will not be destroyed.
Now you had to create a new area for this element:
$element.append($compile($randomElem)($scope.$new()));
$element.append($compile($randomElem)($scope.$new(true)));
Then you need to manually destroy the area when you delete the item. For this you can use$scope.$destroy()
For example:
var newScope = $scope.$new();
$element.append($compile($randomElem)(newScope));
newScope.$destroy(); // or $randomElem.scope().$destroy();
$randomElem.remove();
source
share