Reason (s) are political, not technical, and date back to the early 2000s.
OpenGL 3 was the first version to break compatibility. Designers wanted to create an API for experienced users, programmers, and high-performance visualization coders who knew everything about shaders and wrote their own matrix code. The goal was that the OpenGL 3 API should exactly match the actual hardware. (Even in OpenGL 1/2, the matrix stack was usually implemented on the processor side, not on the GPU.)
From the point of view of the game engine programmer, this was better. And hey, if you need to develop a new game engine every couple of years anyway, what's the big deal to throw away the old code?
The result of this design process is the OpenGL 3/4 kernel profile.
As soon as the “new generation” of OpenGL was announced, all the not very experienced coders at universities and companies realized that they would be screwed. These are people (like me) who teach 3D graphics or write useful programs for research or design. We do not need more advanced lighting than the usual external diffuse-mirror. We often have to mix code from different sources together, and it’s easy if everyone uses the same matrices, lighting and texturing rules as those supplied by OpenGL 2.
In addition, I have heard, but cannot confirm, that the large CAD / CAM companies realized that they too would be screwed. Throwing out two million lines of code from ten years of development is not an option when you pay (and pay well: compare prices for Quadro and GeForce users or FireGL versus Radeon).
Thus, both NVIDIA and ATI said they would support the old API as much as they could.
The result of this pressure is compatibility profiles. And now, OpenGL ARB realized that although they want everyone to switch to the main profile, this just won't happen: read the extension specification for tessellation shaders in OpenGL 4, and he mentions that GL_PATCHES will work with glBegin.