Better to build or buy a grid platform?

Question

I want to do some brute force processing processing with an intensive processor to match strings. I ran my prototype in a multi-threaded environment and compared performance with an implementation using Gridgain with multiple nodes (also multi-threaded).

The performance that I observed was that my Gridgain implementation was slower for my multi-threaded implementation. This may have been a mistake in my gridgain implementation, but it was just a prototype, and I thought the results were indicative. So my question is this:

What are the advantages of training and subsequent implementation for a particular grid (hadoop, gridgain or EC2, if you are going to a hosting - other suggestions are welcome), when it was easy to assemble a lightweight computational grid platform with a much deeper learning curve? ... i.e. . What do we get for free with these cloud / network platforms that are / are difficult to implement?

(Note that I do not need a data grid)

Greetings

-James

(ps We’ll be happy to create this community wiki if necessary)

Answer 1

What network are you dealing with? Ten hosts running the same OS would be pretty easy to run the grid - all you really have to do is send work to each host, maybe a little load balancing, maybe consider what to do if the host goes down, maybe To deal with the distribution of the new service code to the hosts when updating the service, but if you do not deal with any of them, this is not very important, since the grid is size controlled. If you are dealing with thousands of hosts or with a service that should never drop or have errors due to dropping single hosts, you suddenly have to worry about:

• does not overload any host
• distribution of new utility code
• detection when the host does not respond and does not send its new job, and also retransmits everything that it worked on
• possibly working in different OS and architectures (a bit against the big endian)
• energy saving - shutting down hosts at low load and backing them up for high load.
• scaling - if you add 100 hosts to your grid tomorrow, how long does it take to connect and work?
• reliability - some services can actually perform calculations on 2-3 different hosts and return only a response to which all hosts agree

This is a short list of things that most grid software should do for you if you need it. If you are working on something small or uncritical, then by all means, roll your own. If you are working on something that should work, or it is large enough that any manual steps in the deployment process will be a maintenance nightmare, then you will probably want to go with what already exists.