一、测试写入速度
向HDFS文件系统中写入数据,10个文件,每个文件10MB,文件存放到/benchmarks/TestDFSIO中
1.启动YARN集群
start-yarn.sh |
|---|
2.启动写入基准测试
hadoop jar /export/server/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 10MB |
|---|
我们可以看到Hadoop启动了一个MapReduce作业来运行benchmark测试。
等待约2-5分钟，MapReduce程序运行成功后，就可以查看测试结果了。
3.查看写入速度结果
2020-09-25 09:56:21,431 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write2020-09-25 09:56:21,431 INFO fs.TestDFSIO: Date & time: Fri Sep 25 09:56:21 CST 20202020-09-25 09:56:21,431 INFO fs.TestDFSIO: Number of files: 102020-09-25 09:56:21,431 INFO fs.TestDFSIO: Total MBytes processed: 1002020-09-25 09:56:21,431 INFO fs.TestDFSIO: Throughput mb/sec: 0.482020-09-25 09:56:21,431 INFO fs.TestDFSIO: Average IO rate mb/sec: 2.822020-09-25 09:56:21,431 INFO fs.TestDFSIO: IO rate std deviation: 3.242020-09-25 09:56:21,431 INFO fs.TestDFSIO: Test exec time sec: 102.392020-09-25 09:56:21,431 INFO fs.TestDFSIO: |
|---|
我们看到目前在虚拟机上的IO吞吐量约为：0.48MB/s
二、测试读取速度
测试hdfs的读取文件性能，在HDFS文件系统中读入10个文件,每个文件10M
hadoop jar /export/server/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 10MB |
|---|
同样，Hadoop也会启动一个MapReduce程序来进行测试。
查看读取果
2020-09-25 10:06:14,023 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read2020-09-25 10:06:14,024 INFO fs.TestDFSIO: Date & time: Fri Sep 25 10:06:14 CST 20202020-09-25 10:06:14,024 INFO fs.TestDFSIO: Number of files: 102020-09-25 10:06:14,024 INFO fs.TestDFSIO: Total MBytes processed: 1002020-09-25 10:06:14,024 INFO fs.TestDFSIO: Throughput mb/sec: 118.622020-09-25 10:06:14,024 INFO fs.TestDFSIO: Average IO rate mb/sec: 162.192020-09-25 10:06:14,024 INFO fs.TestDFSIO: IO rate std deviation: 76.442020-09-25 10:06:14,024 INFO fs.TestDFSIO: Test exec time sec: 30.142020-09-25 10:06:14,024 INFO fs.TestDFSIO: |
|---|
可以看到读取的吞吐量为：118Mb/s
三、清除测试数据
测试期间，会在HDFS集群上创建 /benchmarks目录，测试完毕后，我们可以清理该目录。
[root@node1 mapreduce]# hdfs dfs -ls -R /benchmarksdrwxr-xr-x - root supergroup 0 2020-09-25 10:05 /benchmarks/TestDFSIOdrwxr-xr-x - root supergroup 0 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_0-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_1-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_2-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_3-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_4-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_5-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_6-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_7-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_8-rw-r--r-- 3 root supergroup 112 2020-09-25 10:05 /benchmarks/TestDFSIO/io_control/in_file_test_io_9drwxr-xr-x - root supergroup 0 2020-09-25 09:56 /benchmarks/TestDFSIO/io_data-rw-r--r-- 3 root supergroup 10485760 2020-09-25 09:56 /benchmarks/TestDFSIO/io_data/test_io_0…… |
|---|
执行清理：
hadoop jar /export/server/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -clean |
|---|
删除命令会将 /benchmarks目录中内容删除
[root@node1 mapreduce]# hadoop jar /export/server/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -clean2020-09-25 10:11:03,278 INFO fs.TestDFSIO: TestDFSIO.1.82020-09-25 10:11:03,280 INFO fs.TestDFSIO: nrFiles = 12020-09-25 10:11:03,280 INFO fs.TestDFSIO: nrBytes (MB) = 1.02020-09-25 10:11:03,280 INFO fs.TestDFSIO: bufferSize = 10000002020-09-25 10:11:03,280 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO2020-09-25 10:11:03,892 INFO fs.TestDFSIO: Cleaning up test file |
|---|
