A swarm, cubernetes or meso for batch processing

My application needs to run many containers as work nodes (to perform various batch processing jobs), and I'm not really interested in supporting web servers or databases - just short jobs that can take anywhere from 1 second to 1 hour. My idea is to work against a cloud of nodes, without worrying about which machine from these nodes has the resources available to handle my work (meso is very good at that - as advertised).

I'm playing right now with DC / OS, and I was wondering if any of the other clustering technologies offers this feature: given I need 1CPU, 2GB RAM and 2GB of disk - run X docker container against my nodes .

I like the idea of โ€‹โ€‹swarm because I am very familiar with the docker, and I think that it is easiest to configure and automate (scale up or down). I like kubernets (but no experience) because itโ€™s free and Iโ€™m sure it will stay so long. I like DC / OS because it combines a lot, but I'm not sure about my future plans, and I'm used to disabling functions to include them in a plan that charges your soul for x number of nodes.

What are your thoughts?

+6
source share
1 answer

Kubernetes, Swarm, and Mesos can technically schedule tasks for you and handle restraining resources for you.

Unlike the other two, Mesos was designed primarily to manage distribution, tasks and resources at a lower level. Focusing on these bits has led to more power and flexibility, but also more complex work at a lower level. That's why DC / OS exists to provide you with a set of microservice tools that work well as a higher-level platform.

Mesos has also been designed to allow your own scheduler to complete task life cycle tasks, which are typically required for stateful tasks. Kubernetes and Swarm were designed primarily to handle stateless use of services, and later adapted to handle services and jobs with the scheduler enabled.

DC / OS is built on Mesos and comes with built-in schedulers for tasks and services, but it allows you to create your own scheduler, if necessary.

Kubernetes recently added support for its own planners, but it is much less mature than the Mesos implementation and ecosystem, and it still revolves around using primitive core sets and replicas that can be empowered or limited depending on your needs.

Mesosphere recently built a new dcos-commons infrastructure to make it trivial to create JVM-based Mesos schedulers. This can improve performance on DC / OS. https://github.com/mesosphere/dcos-commons

Mesos and DC / OS also provide more containerization capabilities. If you want, you can use Docker images and Docker containers. Or you can use the Mesos container runtime with or without Docker images, giving you more flexibility in terms of workloads and packaging.

DC / OS and Kubernetes have package managers as well, which can be useful for installing dependencies such as Spark, Kafka, or Cassandra. But DC / OS tends to have more reliable data services as they are built with their own custom schedulers, while the Kubernetes ecosystem tends to create a complex lifecycle that manages Docker containers around their systems due to the late arrival of custom schedulers. Docker also includes package management if you consider docker "package packages". The difference is that DC / OS and Kubernetes package higher-level abstractions (applications and containers) that can contain multiple containers. More recently, Docker added โ€œstacks,โ€ which are higher-level abstractions, but I donโ€™t think there is an external repository mechanism or a lot of package management around them.

Roy is by far the easiest, but its original API was designed in the same way as the node API, which is great for familiarization and navigation, but rather limits it as a higher level abstraction. Since then, they have effectively rewritten the swarm API and integrated it into the docker engine as "swarm-mode". This combination of an orchestration mechanism and container runtime makes the installation and management easier for the user, but also combines two different levels of abstraction earlier. Therefore, instead of being just a dependency of orchestral engines, Docker now competes with them, contrary to the Unix philosophy, which does something well, and creates political turmoil in the respective open source communities. Twitter, hacker news and chat conversations have escalated into forking docker conversations, which lead to K8s experimenting on alternatives , DC / OS, supporting Docker images without using the Docker engine , and Docker extracting containerd .

Everything is working fine. The choice of one type depends on your needs. I generally recommend DC / OS because it solves a wider range of problems and consists of many different tools and layers of microservice, allowing you to support several use cases, programming against the layer, than this makes sense. Disclosure of this, I work in the Mesosphere!;)

+7
source

Source: https://habr.com/ru/post/1013357/


All Articles