Apache Sqoop 系列(二) 从MySQL把数据导入到Hive
Hadoop
2020-01-15
426
0
一、基本导入
sqoop import \ --connect jdbc:mysql://airawat-mySqlServer/employees \ --username myUID \ --password myPWD \ --table departments \ --direct \ -m 1 \ --hive-import \ --create-hive-table \ --hive-table departments_mysql \ --target-dir /user/hive/warehouse/ \ --enclosed-by '\"' \ --fields-terminated-by , \ --escaped-by \\ \
二、使用分区表
初始导入
$ sqoop import
--connect jdbc:mysql://airawat-mySqlServer-node/employees
--username myUID
--password myPwd
--query 'select EMP_NO,birth_date,first_name,last_name,hire_date from employees where gender="M" AND $CONDITIONS'
--direct
-m 6
--split-by EMP_NO
--hive-import
--create-hive-table
--hive-table employees_import_parts
--target-dir /user/hive/warehouse/employee-parts
--hive-partition-key gender
--hive-partition-value 'M'
--enclosed-by '"'
--fields-terminated-by ,
--escaped-by \ </pre> 查看报表:显示表头hive> set hive.cli.print.header=true;显示分区hive> show partitions employees_import_parts;**增量分区**$ sqoop import \ --connect jdbc:mysql://airawat-mySqlServer-node/employees \ --username myUID \ --password myPWD \ --query 'select emp_no,birth_date,first_name,last_name,hire_date from employees where gender="F" AND $CONDITIONS' \ --direct \ -m 6 \ --split-by emp_no \ --hive-import \ --hive-overwrite \ --hive-table employees_import_parts \ --target-dir /user/hive/warehouse/employee-parts_F \ --hive-partition-key gender \ --hive-partition-value 'F' \ --enclosed-by '\"' \ --fields-terminated-by , \ --escaped-by \\ \三、动态分区
当前版本不支持 (2013年6月的文章,现在是否支持不清楚回头看看官方文档)
设置Hive 支持动态分区
hive> hive.exec.dynamic.partition = true;四、参考
http://hadooped.blogspot.com/2013/06/apache-sqoop-part-2-for-data.html
0条评论