The usual way is to expose some CLI commands and allow users to use them in any language that writes a custom shell / script language; see fe imagemagick , which provides several commands for converting images between formats and applying transforms. This works well on any OS.
It can also work with interactive programs, although this is rare. You can use the D-BUS interface (which is becoming increasingly popular in both GNOME and KDE), although it is more suitable for handling events or sending simple commands. You might want to create an interactive or daemon program that provides a D-BUS interface (or even a simple socket / pipe), as well as some simple CLI calls that wrap send commands to make the interface much easier. See moc / mocp ("Music on the console player") or xmms2 . This works well in any OS, but it usually takes some time to develop implementation details on different OSs.
Don't be afraid to embed a full language. Languages such as Lua or Guile have been designed in such a way that they are very easy to use and quite powerful. Standardization in one such language is not always bad, because it means repeatedly reusing code between users ... and the language actually matters only if you plan on users to write large pieces of code in the form of plugins.
There are several ways to open the API for several scripting languages using special libraries. You can read about them fe here: Kross @Wikipedia . I have no experience with them.
I assume that your program will be closed-source ... Then the last option that I see is to open some kind of API / ABI interface that can be used by C user programs (fe compiled into a dynamic library). This way, users can create wrappers for any language they want, plus they can make code in simple C for speed. This solution can be difficult to make portable, but it gives you (and your users) the flexibility.
Please note that scripting is easy to rewrite: it is better to leave programming constructs in external languages and provide only simple means of interaction with your program. I saw programs that added their own looping tools to scripting languages, although they did not add any value to the user: fe the ability to transfer multiple images at once for conversion, although this did not speed up the processing.
liori source share