Hadoop Profile Exit - Where and What?

Question

I am trying to profile my application to see if I can play this blog post . I added -D mapred.task.profile = true on the command line and checked the job configuration that was completed.

Hadoop: The Definitive Guide says that the profile information will appear in the Unix directory from which I completed the task. The initial file that I started has a try_201305011806_0042_m_000002_0.profile file, which is the correct job identifier, but there was no resolver # 2 (only 1 cartographer, and it did not work). The output file has only header information in the profile file; there is no actual profiling information.

Hadoop docs say that the output will be in the user's log directory, but I can't find anything. If I go into the task logs for the cartographer, there will profile information under "profile.out logs" with legitimate information. My HDFS output file does not have profiling information at all. Should the profiling file be in HDFS somewhere?

In addition, it provides only text output in the log, but all the tools I found render a profile assuming the binary format is hprof. Any ideas on how I can get a binary profile or load a text profile into an hprof tool?

Answer 1

I noticed there is a space in

-D mapred.task.profile=true 

This is a missprint? If so, just delete it and see what happens. In addition, you should be able to see the profiler files in the user's log directory, usually from which you start the task. In addition, hprof is the default for hadoop, so check if you are rewriting it with

 -Dmapred.task.profile.params