Ubuntu 18.04下搭建单机Hadoop和Spark集群环境(2)

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

如下图:

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

修改解压后的目录下的子目录文件 etc/hadoop/hdfs-site.xml

linuxidc@linuxidc:/opt/hadoop/hadoop-2.7.7/etc/hadoop$ sudo nano hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

如下图:

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

设置免密登陆

linuxidc@linuxidc:~/www.linuxidc.com$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/linuxidc/.ssh/id_rsa.
Your public key has been saved in /home/linuxidc/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:zY+ELQc3sPXwTBRfKlTwntek6TWVsuQziHtu3N/6L5w linuxidc@linuxidc
The key's randomart image is:
+---[RSA 2048]----+
|        . o.*+. .|
|        + B o o.|
|        o o =o+.o|
|        B..+oo=o|
|        S.*. ==.+|
|        +.o .oo.|
|        .o.o... |
|          oo .E .|
|          ..  o==|
+----[SHA256]-----+
linuxidc@linuxidc:~/www.linuxidc.com$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
linuxidc@linuxidc:~/www.linuxidc.com$ chmod 0600 ~/.ssh/authorized_keys

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

使用命令:ssh localhost 验证是否成功,如果不需要输入密码即可登陆说明成功了。


linuxidc@linuxidc:~/www.linuxidc.com$ ssh localhost
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 5.4.0-999-generic x86_64)

* Documentation:  https://help.ubuntu.com
 * Management:    https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage


 * Canonical Livepatch is available for installation.
  - Reduce system reboots and improve kernel security. Activate at:
    https://ubuntu.com/livepatch

188 个可升级软件包。
0 个安全更新。

Your Hardware Enablement Stack (HWE) is supported until April 2023.
Last login: Sat Nov 30 23:25:35 2019 from 127.0.0.1

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

接下来需要验证Hadoop的安装

a、格式化文件系统

linuxidc@linuxidc:/opt/hadoop/hadoop-2.7.7$ bin/hdfs namenode -format
19/11/30 23:29:06 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:  host = linuxidc/127.0.1.1
STARTUP_MSG:  args = [-format]
STARTUP_MSG:  version = 2.7.7
......

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

b、启动Namenode和Datanode

linuxidc@linuxidc:/opt/hadoop/hadoop-2.7.7$ sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/hadoop/hadoop-2.7.7/logs/hadoop-linuxidc-namenode-linuxidc.out
localhost: starting datanode, logging to /opt/hadoop/hadoop-2.7.7/logs/hadoop-linuxidc-datanode-linuxidc.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:OSXsQK3E9ReBQ8c5to2wvpcS6UGrP8tQki0IInUXcG0.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop-2.7.7/logs/hadoop-linuxidc-secondarynamenode-linuxidc.out

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

c、浏览器访问:50070

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

3、Scala安装:

下载地址:https://www.scala-lang.org/download/2.11.8.html

Ubuntu 18.04下搭建单机Hadoop和Spark集群环境

下载好后解压到:/opt/scala

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/1e0cf01dd9e69390f716b5886196a4d5.html