Apache Spark 1.5.1
2、安装所需的一切 # sudo apt-get install git -y # sudo apt-add-repository ppa:webupd8team/Java -y # sudo apt-get update -y # sudo apt-get install Oracle-java8-installer -y # sudo apt-get install oracle-java8-set-default # sudo apt-get install maven gradle -y # sudo apt-get install sbt -y # sudo wget Hadoop2.6.tgz # sudo tar -xvf spark*.tgz # sudo chmod 755 spark* # sudo apt-get update # sudo apt-get install -y openjdk-7-jdk # sudo apt-get install -y autoconf libtool # sudo apt-get -y install build-essential Python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev # sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release -cs)添加到软件仓库:
# echo "deb ${DISTRO} ${CODENAME} main" | \ sudo tee /etc/apt/sources.list.d/mesosphere.list # sudo apt-get -y update # sudo apt-get -y install mesos为了便于Spark集群未来从独立集群模式升级,还安装了Apache Mesos。
针对Spark独立集群,使用了spark-1.5.1-bin-hadoop2.6
conf/spark-env.sh #!/usr/bin/env bash export SPARK_LOCAL_IP=MYIP 3、启动一个节点 # sbin/start-slave.sh masterIP:7077可参考文档:
4、安装其它的工具和服务器1)安装MongoDB 3.0.4版
# sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10 # echo "deb "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list # sudo apt-get update # sudo apt-get install -y mongodb-org # sudo apt-get install -y mongodb-org=3.0.4 mongodb-org-server=3.0.4 mongodb-org-shell=3.0.4 mongodb-org-mongos=3.0.4 mongodb-org-tools=3.0.4 # sudo service mongod start # sudo tail -5000 /var/log/mongodb/mongod.log2)安装PostgreSQL
可参考文档:
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-14-04
3)安装Redis
可参考文档:
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-redis
4)安装Scala 2.11.7版
可参考文档:
执行命令:
# sudo wget # sudo dpkg -i scala-2.11.7.deb # echo "deb /" | sudo tee -a /etc/apt/sources.list.d/sbt.list # sudo apt-get update # sudo apt-get install sbt # sudo apt-get install unzip # curl -s get.gvmtool.net | bash # source "/root/.gvm/bin/gvm-init.sh" # gvm install gradle更多Spark相关教程见以下内容:
CentOS 7.0下安装并配置Spark
CentOS 6.2(64位)下安装Spark0.8.0详细记录
安装Spark集群(在CentOS上)