一，環境

CentOS6.5

JDK1.8.0

Hadoop2.7.1

Zookeeper3.7.1

Scala-2.13.0

二，下載及解壓

機器：hadoop01

下載地址??http://spark.apache.org/releases/spark-release-2-2-2.html??

解壓到/home/software下

三，配置

機器：hadoop01

進入conf目錄

1，復制spark-env.sh

cp spark-env.sh.template spark-env.sh

復制緩存的文件spark-env.sh.template改為spark識別的文件spark-env.sh，在文件尾部編輯如下內容：

export JAVA_HOME=/home/software/jdk1.8.0

export SCALA_HOME=/home/software/scala-2.13.0

export HADOOP_HOME=/home/hadoop-2.7.1

export HADOOP_CONF_DIR=/home/hadoop-2.7.1/etc/hadoop

export SPARK_MASTER_IP=hadoop01

export SPARK_WORKER_MEMORY=2g

export SPARK_WORKER_CORES=2

export SPARK_WORKER_INSTANCES=1

2，復制slaves

cp slaves.template slaves

編輯如下內容：

hadoop02

hadoop03

3，復制到hadoop02和hadoop03上

[root@hadoop01 software]# scp -r spark-2.2.2 hadoop02:/home/software/

[root@hadoop01 software]# scp -r spark-2.2.2 hadoop03:/home/software/

四，啟動

1，進入spark/sbin下

sh start-all.sh

2，在web上訪問

??http://hadoop01:8080??

3，進入bin/spark-shell

a，啟動并加載jar包準備進入hive庫

[root@hadoop01 bin]# sh spark-shell --master local[3] --jars /home/software/hive-1.2.0/lib/mysql-connector-java-5.1.38-bin.jar

b，查看庫

scala> spark.sql("show databases").show;

c，創建庫

scala> spark.sql("create database test").show;

d，使用庫

scala> spark.sql("use test").show;

e，創建表

scala> spark.sql("create table student(id int, name string,age int) row format delimited fields terminated by ","").show;

f，增加數據

spark.sql("insert into student values(1,"Zenghaoxuan",10)").show;

g，查看數據

scala> spark.sql("select * from student").show;

+---+---------+---+

| id| name|age|

+---+---------+---+

| 1|Zenghaoxuan| 10|

+---+---------+---+