shell 多线程及线程数控制实现

2020-07-28 15:37:27 浏览数 (1)

生成测试文件，模拟多个任务

代码语言：javascript复制
[root@centos76 parallel]# for i in {1..10};do echo $i >> example.txt;done

单进程

代码语言：javascript复制
[root@centos76 parallel]# cat parallel_run.sh 
#!/bin/bash

file_name=${1:?we need a parameter!}
if [ ! -f "$file_name" ];then
    echo "Can't find the file $file_name !"
    exit 1
fi
while read ;do
    {
        echo $REPLY
        sleep 1
    }
done < $file_name
echo done

运行：

代码语言：javascript复制
[root@centos76 parallel]# time ./parallel_run.sh example.txt 
1
2
......
9
10
done

real 0m10.114s
user 0m0.030s
sys 0m0.071s

多进程

&实现多进程，wait实现父进程等待子进程完成后退出

代码语言：javascript复制
[root@centos76 parallel]# cat parallel_run.sh 
#!/bin/bash

file_name=${1:?we need a parameter!}
if [ ! -f "$file_name" ];then
    echo "Can't find the file $file_name !"
    exit 1
fi
while read ;do
    {
        echo $REPLY
        sleep 1
    } &
done < $file_name
wait
echo done

运行，数字全部一起输出，耗时1s：

代码语言：javascript复制
[root@centos76 parallel]# time ./parallel_run.sh example.txt 
6
7
2
3
4
8
9
10
5
1
done

real 0m1.024s
user 0m0.011s
sys 0m0.012s

通过管道控制进程数

代码语言：javascript复制
[root@centos76 parallel]# cat parallel_run.sh 
#!/bin/bash

file_name=${1:?we need a parameter!}
if [ ! -f "$file_name" ];then
    echo "Can't find the file $file_name !"
    exit 1
fi

thread_num=4
# create pipe
[ ! -p tmp ] && mkfifo tmp
exec 9<>tmp

# fill in the pipe
for ((i=0;i<thread_num;i  )); do
    echo 1>&9
done

while read ;do
    # remove one line per thread
    (read -u 9) # In order to avoid the "read" command in while loop, we use the parentheses here
    {
        echo $REPLY
        sleep 1
        # thread run finish, put line back
        echo 1>&9
    } &
done < $file_name
wait
echo done

运行如下，数字四个一组输出，共三组，耗时3s：

代码语言：javascript复制
[root@centos76 parallel]# time ./parallel_run.sh example.txt 
1
2
3
4
5
6
7
8
9
10
done

real 0m3.019s
user 0m0.014s
sys 0m0.019s

另辟蹊径，通过xargs控制进程数

代码语言：javascript复制
[root@centos76 parallel]# cat parallel_run.sh 
#!/bin/bash

whereami=`cd $(dirname $0); pwd`
file_name=${1:?we need a parameter!}
if [ ! -f "$file_name" ];then
    echo "Can't find the file $file_name !"
    exit 1
fi

thread_num=4

cat $file_name |xargs -n 1 -I {} -P ${thread_num} sh -c "$whereami/print.sh {}"
echo done

[root@centos76 parallel]# cat print.sh 
#!/bin/bash

echo $* 
sleep 1

运行如下，数字四个一组输出，共三组，耗时3s：

代码语言：javascript复制
[root@centos76 parallel]# time ./parallel_run.sh example.txt 
1
2
3
4
5
6
8
7
9
10
done

real 0m3.033s
user 0m0.033s
sys 0m0.024s
wait xargs 测试 多进程 进程

0 人点赞