If they are not used, be sure to remove them.
However, leaving them around should not hurt everyday performance, as they do not reference any application.
They will affect server performance in areas such as backups, etc., since they have more data that needs to be verified and backed up, etc.