Can someone shed some light on why this is so?
Because Google does not have a time machine. Or, if they are, they do not use it to "retcon" previous versions of Android.
it just uses Java 8 features, such as the default methods for collections, and adds some new code to the java util library?
Right. However, Google does not have the means to modify the java.util classes in previous versions of Android. The problem with adding new java.util classes through the library will be the problem.
source share