You want to create your own DE (desktop environment). Common examples are GNOME and KDE, although many have been popular. In fact, it is very popular for creating your own DE, and specialized scripting tools are often created.
Usually you need to combine several tools: window manager, toolbar, desktop manager (icons), session manager, possibly more. Of this, only the window manager is a mandatory part (it usually does not make sense not to run any WM inside the X server), while others are optional. Usually you do not write these tools from scratch (this is a lot of work), but use existing components.
Note that both GNOME and KDE actually consist of these elements (well integrated). For example, GNOME consists of a window manager ( metacity ), toolbars ( gnome-panel ), a desktop manager ( nautilus ), etc. You can change one element to another if you want: it was used to use sawfish as a window manager instead of metacity , keeping all other elements intact. Now it is popular to use compiz instead of metacity .
I used FVWM , which is a window manager with additional (good) scripting capabilities that can be used to create toolbars, menus and other things. One nice example is FVWM-Crystal , which is a complete DE built-in to FVWM, and several other tools written in a combination of FVWM and Python scripts. In fact, you can change any behavior at runtime by simply opening the FVWM script console and input commands. FVWM-Crystal can be a good starting point if you want to start with something already usable; Naked FVWM is good if you want to build your DE from scratch.
[UPDATE: it seems that gnome-shell is very similar in this regard to FVWM . Most of his behavior is written using JavaScript (instead of the custom language in FVWM), and it is very easy to change. It is also very active, as it is an important part of GNOME3, with very good composition support. This is a hacker sky; -)]
[UPDATE2: it seems like I was right on gnome-shell. There's Cinnamon , which uses the same mutter window manager as gnome-shell , but creates a desktop that works differently from gnome-shell .]
There was a website that was a good source of ideas for implementation: lynucs.org . He has not been working for two years, but he received in the archive .
Warning: many of these components are no longer supported, especially since GNOME / KDE is actually being used. Development in this area is usually driven by the special needs of embedded devices: telephones or netbooks .
Now, to answer your current questions:
- Linux / OS flaver: everything is easy to configure: Debian, Arch, Gentoo, LFS ...
- Tools / framework: it depends on what you want to achieve. If you want to write a custom toolbar, you can simply use some scripting languages (like FVWM), custom tools like adesklets , or write your own in Qt (if most of your other components are in Qt) or GTK (if most of your other components located in GTK).
- Personal interfaces: there were many, and sites like lynucs.org were a good compilation of them (lynucs lists the components used on each desktop).
- Complicated transparency effects: old tools usually don’t process it initially or do simple things, for example, just reuse part of the desktop as their own background (the so-called fake transparency). Most likely, you will need to write your own code, i.e. as a plugin before
compiz .