TFS adds these labels to the global list so that work items can be associated with specific assemblies. Theoretically, you should not “clear” the assembly list from the global list, because you will leave any work items that use these lists as a value that is in an invalid state.
There are other lists that inflate over time, for example, the list of users "assign". Anyone who has ever been a valid user for this team project will appear on this list for the same reason as above. If you delete a user, the user leaves the work items in an invalid state. You will receive an error message when you try to associate change groups with other actions or perform other operations with invalid work items.
There should be no performance issues with long lists of items. It becomes an annoying selection from a huge list in the drop-down list ... but it's a kind of TFS beast character. It saves all kinds of information so that you can stretch metrics over time.
See the Tfs_Analysis cube or transactional tables for assembly. You will find data rows for every assembly you have ever run. (TFS 2008 deleted rows from transactional tables when deleting an assembly ... in TFS 2010, the data is saved and the IsDeleted flag is set in the database, so the data remains there forever)
source share