All geek questions in one place

How to run Hadoop on a Mesos cluster?

I am trying to configure an Apache Mesos cluster and run Hadoop-Job on it. The documentation here is not at my level, so I cannot understand, and maybe someone here can explain to me:

Should I create a working Hadoop cluster first? Or did you configure the Mesos cluster first? where should i point the slaves? in a Hadoop-slaves file or registered Mesos slaves should be used only?

+6
source share
2 answers

Mesos goal is to run an abstraction for your cluster, where Hadoop will be just one service among others. For this to work, you first need to configure the Mesos cluster as the main component, and then you can start adding services such as Hadoop to this cluster using the Mesos abstraction.

Mesosphere has a great tutorial that you should take a look at and explains in detail how to run Hadoop on top of Mesos, so this will be a good place to start.

Alternatively, this company has recently started serving Elastic Mesos , which is similar in nature to Amazon Elastic MapReduce. Therefore, if you want to quickly get started with Hadoop on Mesos without the pain of setting everything up, this is a good place to start.

+9
source

Once you have installed and started the cluster installation of mesos so that slaves are displayed in the Mesos WebUI, you can add hasoop to it by following these steps:

  • First you need to configure HDFS. Distributing Cloudera hadoop is an easy way to do this. Just follow the instructions on this link to configure it. This will automatically configure user accounts to run MapReduce jobs.
  • You just need to run namenode on your master and datanode running on your subordinates. Navigating to localhost: 50070 will show you that namenode is up and running, and it will also give a list of running datanodes. At the moment, you do not need supervisors or labor relations.
  • Next, to integrate hasoop into mesos, go to this github repository . Copy it to your computer and follow the instructions on the Readme
  • By this time, you will have a jobtracker working on your host. Go to localhost: 50030 to see jobtracker working. You do not need controllers for your slaves. Mesos will automatically launch them for you using the tar file you uploaded to HDFS.

You can also consult the Mesosphere guide for any doubts. Just do not follow all the steps, as they are not for the latest versions of Mesos.

+2
source

Source: https://habr.com/ru/post/957951/

More articles:


All Articles