Recently, we decided a very similar setting with the participants of my trainings.
If you want to add custom sorting to Magento Admin, you need to override the method Mage_Catalog_Model_Resource_Eav_Mysql4_Product_Collection::addAttributeToSortto add custom sorting logic. In your case, this is attaching the attribute of the category name to the selection of the product collection.
Mage_Catalog_Model_Config::getAttributeUsedForSortByArray, .
, . used_for_sort_by 1.