Sqoop Import using Hue - using inner query

Question

I am trying to sqoop data in to hive table from mysql db in apache cloudera .

Below is the sqoop query that i am executing: sqoop import --connect jdbc:postgresql://localhost/test --username admin --password admin --incremental append --check-column sale_id --last-value 0 --query 'SELECT s.sale_id,c.cust_id,c.state,s.qty FROM customer c JOIN sales s on c.cust_id = s.cust_id WHERE $CONDITIONS' --split-by s.sale_id --hive-import --hive-table sales --target-dir /user/cloudera/sales_products --hive-delims-replacement '/t'

Query works fine when I execute it from command prompt/console. However when I am trying to execute same by HUE UI job designer I am getting below error: Error parsing arguments for import

I think it is because of $CONDITIONS argument.

Can you please let me know if it is possible to execute the same query using Hue and how it is feasible.

Thanks in advance,

Answer 1

Hue will submit this script through Oozie Sqoop Action. It has a particular way to specify the arguments.

Hue also comes with a built in Sqoop example that you could try to modify with your import.

Answer 2

The Where $CONDITIONS is required when running sqoop statements through Hue, at least I have not been successful without it.

I use the method I describe in the answer to this question to load data from SQL Server into External Hive tables, tables that sit in an HDFS folder. Here is the DDL I use to create the table from Impala but it can also be used from the Hive editor.

CREATE TABLE HiveDB_NameSpace.ExternalHiveTableName (
    ColumnName1 STRING
    ColumnName2 STRING,
    ColumnName3 STRING,
    ColumnName4 STRING,
    ColumnName5 TIMESTAMP,
    ColumnName6 INT
    ColumnName7 STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' LOCATION '/user/foldername/dbname/tablename';

This works very well for landing 100+ tables. I can then clean, refine, and conform the data as I insert it into internal Hive, Kudu or other tables.