大数据基础设施搭建 – Sqoop

本文介绍: 部署节点：在Had oop 集群的任意节点部署都可以，因为Had oop 配置文件中指出了Na m eNo de、ResourceMana g er在哪台节点上，并且配置文件分发到了所有节点上！–null–non–string：如果指定列为非字符串类型，使用指定字符串替换值为 null的该类列的值。–null–string：如果指定列为字符串类型，使用指定字符串替换值为 null的该类列的值。–field s–terminate d–b y：Hdfs 文本文件中，字段分隔符？–compress ion–code c：指定压缩格式。

[hadoop@hadoop102 software]$ tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /opt/module/

sudo vim /etc/profile.d/my_env.sh

新增内容：

#SQOOP_HOME
export SQOOP_HOME=/opt/module/sqoop-1.4.7
export PATH=$PATH:$SQOOP_HOME/bin

[hadoop@hadoop102 sqoop-1.4.6.bin__hadoop-2.0.4-alpha]$ cd conf/
[hadoop@hadoop102 conf]$ mv sqoop-env-template.sh sqoop-env.sh
[hadoop@hadoop102 conf]$ vim sqoop-env.sh

新增内容：

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/module/hadoop-3.1.3

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3

[hadoop@hadoop102 software]$ cp mysql-connector-java-5.1.37.jar /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib/

[hadoop@hadoop102 lib]$ cd /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/
[hadoop@hadoop102 sqoop-1.4.6.bin__hadoop-2.0.4-alpha]$ bin/sqoop list-databases --connect jdbc:mysql://mall:3306/ --username root --password 123456

/opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/sqoop import 
--connect jdbc:mysql://mall:3306/gmall 
--username root 
--password 123456 
--target-dir /warehouse/db/gmall/sku_info/2023-11-17 
--delete-target-dir 
--query "select id,spu_id,price,sku_name,sku_desc,weight,tm_id,category3_id,is_sale,create_time from sku_info where $CONDITIONS" 
--num-mappers 1 
--fields-terminated-by 't' 
--compress 
--compression-codec gzip 
--null-string '\N' 
--null-non-string '\N'