What C ++ libraries should be used for a large parallel calculating number-crystal project using third-party applications

Introduction

I want to ask for a lot of advice on a new programming project that I am about to start on my own. I am going to be very precise in what I would like to fulfill and what my basic requirements are. Therefore, this will be a long question. Please carry me.

I am going to divide the question into five sections:

  • Real problem
  • Simulation problem
  • Requirements and Preferences
  • Additional Information
  • Consultation Request List

1. The real problem

Skyscrapers and large bridges suffer from dynamic wind loads. This means that if improperly designed, they can collapse due to wind fluctuations (this really happened in 1940: http://www.youtube.com/watch?v=3mclp9QmCGs ). To properly design such structures, analysis and modeling require efficient cracking software.

2. The problem of modeling

There are many programs that can either simulate fluid flows or structural mechanics. Many of them have already been developed for over 30 years and are proven and mature technologies. Therefore, writing a multiphysics program that can simultaneously simulate fluid flow and structural mechanics from scratch is unreasonable. First of all, it will take you years of development before you reach maturity, and it’s very difficult to enter a world that is more than 30 years old dependent on specific software. But more importantly ... why recreate when you can reuse it? Instead of pursuing a monolithic approach, I prefer a partitioned approach where I can reuse existing simulation software.

In a partitioned approach, I will use the X software to model the flows, and I will use the Y software to model the structures. Then I will write my own communication algorithm, which establishes the connection between X and Y and uses them to simulate a multiphase problem (for example, wind vibrations of skyscrapers or bridges). The reason I use X and Y, rather than the actual names of the programs, is because X and Y should be black boxes. In no case will my communication algorithm depend on the implementation of X and Y. The algorithm will depend only on the output of X and Y. Thus, the end user can choose which one of X or Y is available to them, or when X or Y are capable of doing what the end user wants to achieve.

Since I want to use a partitioned approach with a black box, the X software knows nothing about Y and vice versa. But how can I simulate the deformation of a bridge without knowing anything about the surrounding air flow and how do I know how the surrounding air flow is disturbed by the structure without knowing anything about its deformation? The answer is simple: start with a hunch and use an iterative approach to converge to the right solution. However, this approach is very expensive. To reduce computational cost, the communication algorithm can be written in a smart way using very efficient technologies, so as not to be discussed here. All I would like to say is that it requires heavy linear algebra algebra.

3. Requirements and Preferences

What do I need to do:

  • Establish a connection between third-party open source software or proprietary software.
  • perform heavy crunch number (linear algebra)
  • visualize results (2D / 3D graphics and animation)
  • provides an interactive environment for analysis and development.
  • create an intuitive graphical user interface

I want my software to be:

  • open source
  • cross platform
  • extensible through scripts and / or shared libraries

What I will use:

  • C ++ for heavy crunching numbers
  • CPython for programming logic
  • NumPy / SciPy for some crunch in CPython
  • Matplotlib for visualizing results in CPython

4. Additional information

Data:

  • one-person project at startup if the company succeeds
  • The main OS is a KDE-based Linux distribution.

Business model:

  • Free software and basic documentation.
  • Paid services and detailed documentation.

5. List of recommendations

I want to do all the number-crunching in C ++ by writing many functions that individually perform only a tiny task. The logic of the program should be contained in the CPython package, which performs the entire simulation, relying on C ++ functions to execute the crunch. The C ++ / CPython algorithm should be extended with scripts written in CPython (using NumPy, SciPy, SymPy, and Matplotlib) to generate and visualize the results from the original numerical data. I want to be able to do parallel computing, and I need to communicate with several third-party open source software and proprietary software.

To accomplish all this, I will need a whole bunch of existing libraries / packages / technologies, etc. And for all the relevant issues, I know what I can use, but I don’t know what I should use. The best solution is, as always, to try everything and see what works best. However, if an experienced user can get rid of some of the most unlikely candidates, I will gladly receive his or her advice, suggestion, pro / con list on:

  • Bonding C ++ and CPython (e.g. CTypes, SIP, SWIG, etc.).
  • C ++ library of linear algebra with crunching (e.g. Armadillo, Eigen, PETSc, etc.).
  • GUI development library (e.g. Qt, GTK, wxWidgets, etc.).
  • Software communications and parallel computing (e.g. MPICH, OpenMPI, OpenMP, etc.).
  • CPython 2.7.x or CPython 3.x

NOTE. I have summarized some of the options above, but they are only exemplary, not limitations. I am open to everyone as long as it is written in C, C ++, Fortran or Python. Also, I do not expect an answer in all five categories listed above from one person. Let community knowledge take care of this.

I thank all participants and wish you all the best in your own endeavors.

+6
source share
4 answers

GUI development library (e.g. Qt, GTK, wxWidgets, etc.)
If your "main OS is a KDE-based Linux distribution", then QT wins this combination.
The logic behind this is: -

  • KDE is written to QT. KT's QT app is an eagle in the air! It is in its element. Since KDE, being the target number 1, your QT GUI will most likely work out of the box without having to download additional gui files. Your GUI will also look super-native.
  • QT is the most portable of the three. (you mentioned the "main OS", hinting that other platforms will follow). Therefore, with qt you can port to Windows, OSX, GNOME, Embeded Linux, Android, Symbian, HAIKU, Solaris..etc
  • QTs are arguably the best RAD tools of the three main GUIs (IMHO). Think QTCreator vs wxSmith vs Anjuta / Glade.
  • wxWidgets on linux is basically wapper for GTK + (v1 - v3) + additional helpers. Tho I prefer his GTK. It also wraps around X11 and the motive, but trust me chef, you will not like these ports.
  • WxWidgets portability is not as seamless as you might think. Each port represents a completely different implementation, each of which wraps completely different backends! I once ported a small application that uses wxDataViewCtrl with a custom tree model. SIGSEGVs has become the app of the day. So I finally decided to go with a generic wxDataViewCtrl (which looks ridiculous in GTK + 3). I still like wxWidgets tho.

NB: also consider using the latest web technologies for the C&V MVC part (model view controller).
HTML5 + CSS3 + JS can be controlled by a web widget on the desktop. All 3 GUI Libs listed above support this control (for wx, wx2.9.3 and higher).
Web Technologies: -

  • Have (possibly) the fastest time to market for any graphical user interface.
  • Have (possibly) the most affordable and affordable developers of any technology of the graphical user interface.
  • Create the most awesome user interfaces of any graphical user interface.
  • Create the least "RIGID" user interfaces of any GUI lib ... you can rotate the gui element, for example. html-table around any axis with fantastic animation just with the mouse, without any overhead! .. no js..no C ++ .. nada!

CPython 2.7.x or CPython 3.x
- CPython may not be very well suited to your project requirements (I think), mainly because of the mutex monster, which is the Global Lock Interpreter Lock (GIL) bottleneck.
Perhaps PyPy will be the best python implementation for your project?

By the way, you also considered: - Javascript on V8 against Python (PyPy, CPython, etc.)? Javascript executed by V8 can interact with Native Code (C ++), sort of like Ctypes with python

I also came across this interesting blog (JS on V8 vs Py).

+1
source

You mentioned parallelism, but not how you are going to make this project parallel. This is a much more complicated problem than just choosing a pair of libraries. There are several important considerations to be made before driving.

You briefly mention the proposed platform, but you also need to think about whether the simulation will work on one computer / node or several. Given that you are doing iterative modeling of a building, you probably need a lot more computing power than any computer can provide. This means that if you do not want to use a hybrid multi-processor multi-threaded approach, you are limited to the model of several parallelism processes. OpenCL and MPI are all options for your implementation. (note: MPICH and OpenMPI are just an implementation of MPI, and your code should be agnostic). Messaging with MPI is a good general parallelism model, however it can be quite difficult for those who do not work with parallel code. My personal experience is with MPI and some hybrid programming, so I can not say more about your choice of parallel model.

The problem arising from the parallel model question is that it directly affects simulation software. I'm not quite sure how you plan to keep your algorithm from modeling. If you plan to use a separate fork development process to run the simulation, you will have problems with cross-platform support, as you may not have the luxury of being able to perform arbitrary simulations this way. If you intend to run simulations in your software, the parallel model should be consistent throughout. Although this puts limits on the black box strategy, it can do anything that is much more doable.

We have already talked about applicable libraries. I have nothing more to say about specific libraries that have not yet been talked about. Just keep in mind that many of the same problems must be dealt with with them, as when running simulations.

TL; DR: parallelism should not be viewed. You need to know which parallel model you will use before deciding on libraries.

+1
source

Maybe gmp. You can find more information here: http://gmplib.org/

0
source

sounds like premature optimization. You need to write code, a lot of code, put a lot of prints around the world to get statistics, and then try at least two different approaches to get a guide, and then make a smart decision. There is no replacement for the job; everything else is just manual work.

0
source

Source: https://habr.com/ru/post/945334/


All Articles