All geek questions in one place

WebHDFS does not work on secure hadoop cluster

I am trying to protect my HDP2 Hadoop cluster using Kerberos.

While working Hdfs, Hive, Hbase, Hue Beeswax and Hue Job / task browsers; however Hue File Browser does not work, it responds:

WebHdfsException at /filebrowser/ AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] (error 500) Request Method: GET Request URL: http://bt1svlmy:8000/filebrowser/ Django Version: 1.2.3 Exception Type: WebHdfsException Exception Value: AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] (error 500) Exception Location: /usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py in _stats, line 208 Python Executable: /usr/bin/python2.6 Python Version: 2.6.6 (...)

My hue.ini file hue.ini configured with all security_enabled=true parameters and other related parameters.

I believe the problem is with WebHDFS.

I tried the curl commands specified at http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#Authentication

 curl -i --negotiate -L -u : "http://172.19.115.50:14000/webhdfs/v1/filetoread?op=OPEN"

the answers are:

 HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly Content-Type: text/html;charset=utf-8 Content-Length: 1027 Date: Wed, 08 Oct 2014 06:55:51 GMT <html><head><title>Apache Tomcat/6.0.37 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 403 - Anonymous requests are disallowed</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Anonymous requests are disallowed</u></p><p><b>description</b> <u>Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.37</h3></body></html>

And I could reproduce the Hue error message by adding a user with the following curl request:

 curl --negotiate -i -L -u: "http://172.19.115.50:14000/webhdfs/v1/filetoread?op=OPEN&user.name=theuser"

he answers:

 HTTP/1.1 500 Internal Server Error Server: Apache-Coyote/1.1 Set-Cookie: hadoop.auth=u=theuser&p=theuser&t=simple&e=1412735529027&s=rQAfgMdExsQjx6N8cQ10JKWb2kM=; Path=/; Expires=Wed, 08-Oct-2014 02:32:09 GMT; HttpOnly Content-Type: application/json Transfer-Encoding: chunked Date: Tue, 07 Oct 2014 16:32:09 GMT Connection: close {"RemoteException":{"message":"SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}

There seems to be no Kerberos negotiation between WebHDFS and curl.

I was expecting something like:

 HTTP/1.1 401 UnauthorizedContent-Type: text/html; charset=utf-8 WWW-Authenticate: Negotiate Content-Length: 0 Server: Jetty(6.1.26) HTTP/1.1 307 TEMPORARY_REDIRECT Content-Type: application/octet-stream Expires: Thu, 01-Jan-1970 00:00:00 GMT Set-Cookie: hadoop.auth="u=exampleuser& p=exampleuser@MYCOMPANY.COM &t=kerberos&e=1375144834763&s=iY52iRvjuuoZ5iYG8G5g12O2Vwo=";Path=/ Location: http://hadoopnamenode.mycompany.com:1006/webhdfs/v1/user/release/docexample/test.txt?op=OPEN&delegation=JAAHcmVsZWFzZQdyZWxlYXNlAIoBQCrfpdGKAUBO7CnRju3TbBSlID_osB658jfGfRpEt8-u9WHymRJXRUJIREZTIGRlbGVnYXRpb24SMTAuMjAuMTAwLjkxOjUwMDcw&offset=0 Content-Length: 0 Server: Jetty(6.1.26) HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 16 Server: Jetty(6.1.26) A|1|2|3 B|4|5|6

Any idea what could go wrong?

I have in my hdfs-site.xml on each node:

 <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.web.authentication.kerberos.principal</name> <value>HTTP/ _HOST@MY-REALM.COM </value> </property> <property> <name>dfs.web.authentication.kerberos.keytab</name> <value>/etc/hadoop/conf/HTTP.keytab</value> <!-- path to the HTTP keytab --> </property>
+5
source share
2 answers

It looks like you do not have access to WebHDFS (default port = 50070), but HttpFS (default port = 14000), which is a "simple" webapp that is not protected in the same way.

The WebHDFS URL is often similar to http://namenode:50070/webhdfs/v1 ; try changing hue.ini with this parameter (it is recommended to use WebHDFS over HttpFS)

+3
source

I also ran into the same problem. The hdds port of hpds 50070 is not included in my secure kerberoised hadoop with tls encrypthion in cdh

HDFS-site.xml

  <property> <name>dfs.http.policy</name> <value>HTTPS_ONLY</value> </property>

But when I change this property to

  <property> <name>dfs.http.policy</name> <value>HTTP_AND_HTTPS</value> </property>

port 50070 becomes active, and the webhdfs command successfully executed curl -i - anonymously -u root: root " http: // HOSTNAME: 50070 / webhdfs / v1 / tmp? user.nfs & op = GETFILESTATUS "

but then my datanode is not working

Is there a way to run webhdfs in secure hadoop?

0
source

Source: https://habr.com/ru/post/1204195/

More articles:


All Articles