Is there a way so that I can "upgrade" my Java SE to EE? Since EE is just basically a few additional libraries ...
In principle, no. Your characteristic of Java EE is incorrect.
Java EE (typically) is implemented as a "web container" or "EJB application server", and its large parts only make sense as part of a web container. (And indeed, a significant portion of the Java EE implementation is actually specific to the web / application container.)
Having said that, some (abstract) APIs and some of the libraries are available as separate JARs and can be installed piecewise. And if what you want to do is to develop for EE, then there are Eclipse plugins / functions / something for that ...
Therefore, it is generally recommended (or safer) to simply uninstall my Java SE, install Java EE. Then uninstall Eclipse and install "Eclipse for EE Development"?
No need to remove Java SE. Indeed, some Java EE implementations require the installation of Java SE. Read the instructions ...
For Eclipse, this may depend on your OS, but on Linux, the “install” process simply extends the ZIP file. I found that two Eclipse installations can coexist side by side in separate directories.
But...
Be careful not to accidentally install one version / version on top of another. IIRC, the root directory of a ZIP file is always an "eclipse" ... which makes it shoot in the foot! (Sigh).
It's wise to back up everything, including the old Eclipse installation and your workspaces.
In fact, some people recommend throwing away all your workspaces and starting over by checking / materializing projects from version control.