I started generosity for this matter
... because I really want a community. I can (and have) looked at several languages ​​/ and thought “well, it will probably work well”, but I would really appreciate advice based solely on the problem I am facing, and especially from anyone who has experience integration / using what you recommend.
I am working on scientific analysis software. It provides many tools for mathematical transformation of data. One tool allows the user to enter their own equation, which is performed over a data set (large two-dimensional or three-dimensional matrix of values) and evaluated.
This tool has a graphical equation editor that internally creates an object-oriented expression tree with a different object for each operation (for example, there will be an instance of the Logarithm class, which is a node in the tree to add the calculation of the logarithm of the value to the base, it has two children who are by its inputs.) Screenshot of its part:
You can see the tree that he is building on the left, and several of the many (fifty?) Potential operations in the menu on the right.
This has several disadvantages:
- The graphical editor becomes awkward for complex equations
- There are some operations that are difficult to represent graphically, for example, creating large matrices (for example, the kernel for convolution nxn)
- It only solves equations: no branching or other logic
It was neat when it was much easier, but nothing more, for those things that our users want to have to do with it. If I wrote it now, I would have done it in a completely different way - and this is my chance :)
I would like to give the user something more powerful and let them write code - script or compiled - that can perform much more complex operations. I am looking for SO advice on what technology he should use or the best approach to it.
The rest of this question is quite long - sorry. I tried to describe the problem in detail. Thanks in advance for reading :)
Important limitations:
Our math works with large matrices. In the above equation, V1 represents an input (one of potentially many) and represents 2D or 3D, and each dimension can be large: on the order of thousands or hundreds of thousands. (We rarely calculate all of this at once, only fragments / segments. But if the answer includes something that requires sorting the data, pay attention to the size and speed of this consideration.)
The operations we provide allow us to write, say, 2 x V , which multiplies each element in V by 2. The result is another matrix of the same size. In other words, a scripting or programming language that includes standard math primitives is not enough: we need to be able to control which primitives are available or how they are implemented.
These operations can be complicated: the input can be as simple as a number (2, 5.3, pi) or complicated as a 1, 2 or 3-dimensional matrix that contains a boolean or complex number (pair values). My modern thinking is a language powerful enough so that we can expand our data types as classes and implement standard operators. A simple appraiser will be inadequate.
- Instead of just writing operations that are evaluated iteratively on one or more inputs to provide output, as is currently the case (which is easily implemented using an expression evaluator), I would like the user to be able to: provide outputs of different sizes to the inputs; to call other functions; etc. It would be useful for the host program to ask the user code which part or fragment of the inputs is needed to evaluate the fragment or part of the output. I think exposing some of our classes and using the OO language is probably the best way to achieve these goals.
Our audience is primarily academic researchers who are either not used for coding, or are probably used for a language such as Matlab or R.
We use Embarcadero C ++ Builder 2010 for development, with a small amount of Delphi. This may limit what we can use - simply because something C ++, say, does not mean that it will work if it is encoded only against VC ++ or GCC. It should also be suitable for use with commercial software.
Our software currently has a COM interface, and part of the application can be automated with our application, which is a COM server outside the process. We could add COM interfaces to some internal objects or, if necessary, make a second COM infrastructure.
Tools, including this one, are migrated to a multi-threaded environment. The final solution must be performed in any thread , and several instances of it in many threads at the same time . This can affect the runtime of a hosted language - Python 2.x, for example, has a global lock.
It would be great to use the language that comes with the libraries for mathematical or scientific use.
Backward compatibility with the old expression tool is not important. This is version 2: clean slate!
Current ideas:
- RemObjects Pascal Script and DWScript are languages ​​that are easily associated with
TObject tailored classes. I do not know if operator overload can be ensured. - Hosting .Net runtime and loading C # (say) based DLLs as plugins. I like this idea: I saw how this is done where the host program provides a syntax shortcut, debugging, etc. I understand that this was a huge amount of coding. This would allow the use of IronPython and F #.
- RemObjects Hydra looks like an interesting way to achieve this. Unfortunately, he advertises himself for Delphi, not C ++ Builder; I am studying compatibility.
- The hosting is something like Python, which can be done from RAD Studio
- Providing a BPL interface and providing users with code directly against our program if they buy a copy of RAD Studio (i.e. provide a plug-in interface and expose classes via interfaces; it may be necessary that the plug-ins be compiled with a binary compatible version of our IDE)
- ...
Thanks for your input! I appreciate all the answers, even if they are not quite perfect - I can investigate, I’m just after instructions on where to go and opinions (please, opinions for the reasons included in the answer: p) about how to approach it or what may be suitable. Each answer, no matter how short, will be appreciated. But if you recommend something in detail, and not just "use the X language," I will be very interested to read it :)
Greetings
David
Update:
So far, the following has been recommended:
Python: 2.6 has a global lock that sounds like a game killer. 3 (apparently) does not yet have broad support from useful libraries. It sounds to me (and I know that I am a proponent of the Python community), as if it is fragmenting a bit - is it really safe?
Lua: It seems to be not a direct OO, but it provides "meta-mechanisms for implementing functions, and does not provide many functions directly in the language . " It sounds very cool from a programmer’s point of view, but it’s not aimed at programmers who need to play around with cool things. I'm not sure how well this will work with the target audience - I think a language that provides a more basic foundation will be better.
MS script / ActiveScript. We already provide an external COM interface that our users use to automate our software, usually in VBScript. However, I need a more powerful (and, frankly, better developed) language than VBS, and I don’t think JScript is suitable. I’m also not sure what problems may be associated with COM data - we have a lot of data that is often printed very accurately, so the speed and persistence of these types are important.
Lisp: I didn’t even think about this language, but I know that he has a lot of fans.
Hosting Pure plugins: no one is mentioned. Isn't that a good idea? You get C #, F #, Python ... Does it have the same problems with COM sorting? (Does CLR work through COM?)
A few clarifications: on the "matrix", I mean the matrix in the sense of the Matlab variable, i.e. a huge table of values ​​- not, say, a 4x4 transformation matrix, as you could use for 3D software. This data was collected over time, thousands and thousands of values ​​often many times per second. We are also not after a computer algebra system, but something where users can write complete plugins and write their own mathematics - although a system with the ability to process complex mathematics, like a computer algebra system, can be useful. I would take a “full language” over “algebra”, though if they don't mix to allow complex branches / paths in user code, as well as an OO interface.