I export information from HDFS to MS-SQL using SQOOP. I am running SQOOP through OOZIE. Right now I have hardcoded uid, pwd for jdbc connection in OOZIE workflow. As soon as I switch to prod, I cannot do this. What is the best way to transmit authentication information in such a situation?
<sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <arg>export</arg> <arg>--connect</arg> <arg>jdbc:sqlserver://$sqlServerIP:1433</arg> <arg>--table</arg> <arg>tableName</arg> <arg>--export-dir</arg> <arg>/user/sqoop/file</arg> <arg>--username</arg> <arg>me</arg> <arg>--password</arg> <arg>password</arg> </sqoop>
I could pass them as parameters like $ userName, $ password. But the actual uid / pwd will still appear in the oozie web console.
UPDATE
I tried two ways (as suggested below) to do this ... In VIM, I created pwd to just have a password (without spaces or anything else). This pwd is called.
1) I tried to use the file system. However, I have an IOException saying that the file does not exist. Looking at the code, it looks like sqoop is using the passed conf to access fs. Therefore, I assume that when oozie starts, it will only have access to HDFS.
2) I uploaded the password file to a random location on hdfs./users/my-name/pwd (pwd is the file). Now it can access the file (since I am not getting an IOException). However, it cannot connect to SQLServer. I'm not sure what I need to do to make it work?
UPDATE 2 I created the password file as follows: echo "pwd" > my.password This adds an EOL to the file my.password I changed to echo -n "pwd" > my.password and now it works.
source share