Hardware for MongoDB closed + redundant environment

Question

I'm trying to plan a new database environment from scratch, and I wonder how many servers are needed and how much performance they need to provide.

Since I want it to be fast, I am considering using SSDs and RAM downloads. However, flash memory is expensive and makes up the bulk of the server cost. Thus, the whole system must be configured to scale horizontally from the very beginning, so I can add more nodes when I need more memory / performance.

To get started, I am considering using 2 shards, each of which consists of a master and a replica slave for redundancy. The MongoDB documentation suggests using 1 master and 2 slaves, but I'm afraid this will not be in an affordable budget, since each of these servers will be equipped with about 200 GB of RAM and 6x400 GB SSDs like Raid 10.

When using fragments, it is also recommended to use 3 configuration servers for fault tolerance / high availability. As above, I think 1 master and 1 slave as a start.

• What equipment do you recommend for hosting configuration servers on? If they are somewhat equally effective, how are the fragments nodes in the terms cpu / memory / harddisk? Or can I put them on virtualization or on cheaper hardware?
• Is the setting that I described even makes sense? How about the ratio of RAM and hard drive on fragmentation nodes? At the moment, it would probably be easier and cheaper to just put twice as many disks in 1 shard (1 master, 1 slave) and skip the shards until I really need it. However (as mentioned above) - the system must be ready for edging from the very beginning, because storage requirements can change overnight. Or you can install everything, but at the moment it works only on 1 shard?
• Since I plan to use 2 instead of 3 servers for high availability / fault tolerance. Perhaps I need some arbitrators. Do they also need special equipment? Or can I use a single arbiter in a virtual machine serving configuration servers and fragmentation nodes? Or uses 3 separate servers for backup, in your opinion, an absolute obligation?

Answer 1

Scala on. Sounds like an awesome setting. Given your configuration options, I could not imagine a budget that would limit your choices too much.

• You do not need a dedicated physical server for configuration servers. They work quite easily. You will need low latency between your mongos and your configuration servers. You will always be sure that the hosts are reliable and ready for disaster. Make sure you double check backup procedures for your plastered environment. Backups require coordination between the moving parts of a broken cluster. If possible, run configuration servers on virtual servers in the same data center.

• Yes, the hardware you described makes sense if you intend to run multiple shards on the same machine. One MongoDB on this powerful machine will leave the machine mostly inactive. A single mongod process cannot use a lot of RAM, I / O or CPU. You will need a "main splinter" of the host. In MongoHQ, we do this by running each mongod in a container that we can isolate from other instances on the same machine. With your specifications, you can run up to 10 steps on one host or more if you want to stretch hosts.

• You can launch this with a single shard and later move on to the cluster cluster. This is our recommended outline approach: don't splinters until you should. By delaying shards, you increase your flexibility to make changes to your system. When the fine is in place, you are committed to a certain path, without flexibility (which is good when you know the future). By holding up the fragments, you have no compromises.

• Arbitrators do not need specialized equipment. You can run them in virtual machines. They do not require the same level of backup requirements, but should have good time periods.

• Using 3 servers is not a requirement for long runtimes. However, when one data node drops for several hours, you are relegated to one data node. While dedicated to a single data host, a single host will work fine. Since you have only one second of work, you risk losing time. At the same time, 2 nodes plus an arbiter are suitable for most use cases and remain valid if one of the data nodes fails.

Hope this helps! We run similar configurations in MongoHQ and are very pleased with the level of performance we get from the hosts.

Answer 2

I am going to bring some thoughts here.

This answer is practically useless, not knowing your working set, maybe it can create some pointers.

What equipment would you recommend to include in configuration servers?

Configuration servers, despite the fact that it is better on their own equipment (a server, rather than on a virtual machine), can easily be launched on the basis of commercial equipment itself, for this you do not need anything. All they do is save the settings of your sets, and even then they are not used all the time, mongos will cache the cluster configuration for intervals.

Or can I put them on virtualization or on cheaper hardware?

I would not put them on virtualization, because this usually means that they are either physically located on the same server, or are closed. However, you need to put them on real redundant servers, yes, you can get cheap equipment for them.

After making sure you have a decent network between your shards and your configuration servers, you should be natural.

Is the setting that I described even makes sense?

There is no idea without understanding your working set, however, by the sounds of this you think that MongoDB SHOULD EVERYTHING fit into memory. This is incorrect, only a working set ( http://docs.mongodb.org/manual/faq/storage/#what-is-the-working-set ), which can be an absolute share of your data over a certain period of time (usually 10 minutes ) if you play cards correctly.

What about the relationship of RAM to the hard drive on fragmentation nodes?

A bit, MongoDB can use this server, but I’m sure that it will sit idle most of the time, I’m sure you didn’t actually calculate your working set correctly.

At the moment, it would be simpler and cheaper to just double-add the number of disks to 1 shard (1 master, 1 subordinate) and skip the shard until I really need it.

Yes. This is a safe bet if you need these discs. I am going to go so far as to say that you really have to study whether you are doing it or not.

the system should be ready for edging from the start

As said in the answer above, you can create a set of 1 member and just increase it from there.

Since I plan to use 2 instead of 3 servers to provide high availability / fault tolerance, I probably need arbiters as well.

Yes, using 3 servers is usually stable enough for automatic failover, which is important here. If most server crashes (50% or more), you will have to manually fix your replica set.

This means that on two server environments, for each splinter, you will not have automatic recovery after a failure, and therefore MongoDB will not fulfill one of its main advantages: automatic switching to another resource (this is also the main goal of replica sets ...).

Not only that, but your users must wait until you do this fix, I’m sure this in itself has problems.

Do they also need special equipment?

No, as stated in my comment (and in the documentation), arbitrators should not be run on specialized equipment. However, this suggests that it can be useful, especially if you want to create a backup backup system by refusing to arbitrate in another data center.

The arbiter can work on the bare bones of the server, so if you decide to separate it, do not worry about 200-bit RAM and 6x400GB SSD. Just get ... well, a mobile phone can technically run an arbiter (if MongoDB supports Android and iOS).

Or uses 3 separate servers for backup, in your opinion, an absolute obligation?

This is not so, but something less, as mentioned above, will cause problems in the event of a failure.

A good option is to actually put an arbiter on your application server, possibly even on one application server.