Hadoop job status tracking via web interface? (Providing Hadoop to internal customers in the company)

Question

I want to create a website that will allow analysts within the company to run Hadoop tasks (choose from a set of assigned tasks) and see their work status \ progress.

Is there an easy way to do this (get job status, etc.) through Ruby \ Python? How can you open your Hadoop cluster to internal customers in your company?

Answer 1

I found one way to get job information on JobTracker. This is the code:

    Configuration conf = new Configuration();
    conf.set("mapred.job.tracker", "URL");

    JobClient client = new JobClient(new JobConf(conf));

    JobStatus[] jobStatuses = client.getAllJobs();
    for (JobStatus jobStatus : jobStatuses) {

        long lastTaskEndTime = 0L;

        TaskReport[] mapReports = client.getMapTaskReports(jobStatus.getJobID());
        for (TaskReport r : mapReports) {
            if (lastTaskEndTime < r.getFinishTime()) {
                lastTaskEndTime = r.getFinishTime();
            }
        }

        TaskReport[] reduceReports = client.getReduceTaskReports(jobStatus.getJobID());
        for (TaskReport r : reduceReports) {
            if (lastTaskEndTime < r.getFinishTime()) {
                lastTaskEndTime = r.getFinishTime();
            }
        }
        client.getSetupTaskReports(jobStatus.getJobID());
        client.getCleanupTaskReports(jobStatus.getJobID());

        System.out.println("JobID: " + jobStatus.getJobID().toString() + 
                            ", username: " + jobStatus.getUsername() + 
                            ", startTime: " + jobStatus.getStartTime() + 
                            ", endTime: " + lastTaskEndTime + 
                            ", Durration: " + (lastTaskEndTime - jobStatus.getStartTime()));

    }

Answer 2

'beta 2' Cloudera Hadoop Distribution Hadoop User Experience (HUE), Cloudera Desktop.

. , . , .

Answer 3

, Cloudera Destktop. - . .

Answer 4

There is nothing like that that comes with haop. To create this function, it must be trivial. Some of them are available on the JobTracker page, and some of them you will have to create yourself.