https://www.xiongan.host/index.php/tag/Spark/ https://www.xiongan.host/index.php/archives/211/ 2023-05-27T19:52:58+08:00 Spark安装上传压缩包并解压修改用户权限[root@master-tz src]# chown -R hadoop:hadoop spark切换hadoop用户，进入目录[hadoop@master-tz conf]$ pwd /usr/local/src/spark/conf修改spark-env.sh文件[hadoop@master-tz conf]$ cp spark-env.sh.template spark-env.sh [hadoop@master-tz conf]$ vim spark-env.sh //添加如下内容 export JAVA_HOME=/usr/local/src/java export HADOOP_HOME=/usr/local/src/hadoop export SPARK_MASTER_IP=master-tz export SPARK_MASTER_PORT=7077 export SPARK_DIST_CLASSPATH=$(/usr/local/src/hadoop/bin/hadoop classpath) export HADOOP_CONF_DIR=/usr/local/src/hadoop/etc/hadoop export SPARK_YARN_USER_ENV="CLASSPATH=/usr/local/src/hadoop/etc/hadoop" export YARN_CONF_DIR=/usr/local/src/hadoop/etc/hadoop修改slaves[hadoop@master-tz conf]$ cp slaves.template slaves [hadoop@master-tz conf]$ vim slaves //添加如下 master-tz slave01-tz slave02-tz将spark分发到slave1,slave2[root@master-tz conf]# scp -r /usr/local/src/spark/ slave01-tz:/usr/local/src/ [root@master-tz conf]# scp -r /usr/local/src/spark/ slave02-tz:/usr/local/src/发送完毕后，启动spark集群[hadoop@master-tz conf]$ /usr/local/src/spark/sbin/start-all.sh浏览器查看（ip:8080）使用本地运行模式测试。使用本地模式运行Spark Pi程序/usr/local/src/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[*] /usr/local/src/spark/examples/jars/spark-examples_2.11-2.0.0.jar 10Standalone模式运行Spark Pi程序[hadoop@master-tz conf]$ /usr/local/src/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://master-tz:7077 /usr/local/src/spark/examples/jars/spark-examples_2.11-2.0.0.jar 10运行成功后，在浏览器内也可以看到Spark的集中运行模式首先修改配置文件<!--spark-env.sh文件，增加以下内容--> export SPARK_DIST_CLASSPATH=$(/usr/local/src/hadoop/bin/hadoop classpath) export HADOOP_CONF_DIR=/usr/local/src/hadoop/etc/hadoop export SPARK_YARN_USER_ENV="CLASSPATH=/usr/local/src/hadoop/etc/hadoop" export YARN_CONF_DIR=/usr/local/src/hadoop/etc/hadoop<!--yarn-site.xml文件，增加以下内容--> <property> <name>yarn.nodemanager.pmem-checkenabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-checkenabled</name> <value>false</value> </property>再次分发文件到slave01、slave02yarn-cluster模式使用指令[hadoop@master-tz hadoop]$ /usr/local/src/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/local/src/spark/examples/jars/spark-examples_2.11-2.0.0.jar 2yarn-client使用指令/usr/local/src/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /usr/local/src/spark/examples/jars/spark-examples_2.11-2.0.0.jar 2