Parallel Computing with Shell Commands

There are some great parallel utilities that you can use on Linux and UNIX that will make work a lot easier for you


* Parallel ssh (pssh)
* Parallel scp (pscp)
* Parallel rsync (prsync)
* Parallel nuke (pnuke)
* Parallel slurp (pslurp) 
You can download them at http://code.google.com/p/parallel-ssh/downloads/list/


I'll use pssh as an example of how all of these utilities work

1) First I'll create a file called myhosts. Each line of that file contains a hostname (one per line)

2) Now I can run this command:
pssh -h myhosts -o /tmp/biffstuff -t 5 "uptime"

-h myhosts is the filename with my list of hosts
-o is the directory to put the output (one file per host)
-t is timeout after n seconds

The command goes at the end between quotes

When I run the command the screen output will look like this:

[1] 12:05:17 [SUCCESS] ctnfs1
[2] 12:05:17 [SUCCESS] ctnfs2
[3] 12:05:17 [SUCCESS] ctnfs3
[4] 12:05:17 [SUCCESS] ctnfs4
[5] 12:05:17 [SUCCESS] ctnfs5
[6] 12:05:17 [SUCCESS] ctnfs6
[7] 12:05:17 [SUCCESS] elmnis1
[8] 12:05:17 [SUCCESS] flash
[9] 12:05:17 [SUCCESS] mlpnfs1
[10] 12:05:17 [SUCCESS] mlpnfs2
[11] 12:05:17 [SUCCESS] mlpweb1
[12] 12:05:17 [SUCCESS] monitor
[13] 12:05:17 [FAILURE] trdlogger Exited with error code 255
[14] 12:05:17 [FAILURE] imagrep1 Exited with error code 255
[15] 12:05:17 [FAILURE] rfprep2 Exited with error code 255
[16] 12:05:17 [FAILURE] tlmsybdev Exited with error code 255
[17] 12:05:17 [SUCCESS] bosdbatch1
[18] 12:05:17 [FAILURE] ssbatch1 Exited with error code 255
[19] 12:05:17 [FAILURE] imag9 Exited with error code 255
[20] 12:05:17 [SUCCESS] rfpnfs14
[21] 12:05:17 [SUCCESS] ctomsprod1
[22] 12:05:17 [SUCCESS] rfpnfs10
[23] 12:05:17 [SUCCESS] wpnfs1
[24] 12:05:17 [SUCCESS] oms666-2
[25] 12:05:17 [SUCCESS] bosdbatch2
[26] 12:05:17 [SUCCESS] bosdpm1
[27] 12:05:17 [SUCCESS] pm1
[28] 12:05:17 [SUCCESS] pm2
[29] 12:05:17 [SUCCESS] homemade-new
[30] 12:05:17 [SUCCESS] homemaker-new
[31] 12:05:18 [SUCCESS] xray1
[32] 12:05:18 [SUCCESS] omsprod1
[33] 12:05:18 [SUCCESS] mddl-666
[34] 12:05:18 [SUCCESS] rfpjumpstart
[35] 12:05:18 [SUCCESS] bosdbatch5
[36] 12:05:18 [SUCCESS] rs1
[37] 12:05:18 [SUCCESS] imaghist2010
[38] 12:05:18 [SUCCESS] mxrep1
[39] 12:05:18 [SUCCESS] bosdbatch3
[40] 12:05:18 [SUCCESS] bosdbatch4
[41] 12:05:18 [SUCCESS] mlpsyb
[42] 12:05:18 [SUCCESS] recap1
[43] 12:05:18 [SUCCESS] md1b1700
[44] 12:05:18 [FAILURE] netdata1-elm Exited with error code 255
[45] 12:05:18 [SUCCESS] nycprodsup01
[46] 12:05:18 [SUCCESS] md2b1700
[47] 12:05:18 [SUCCESS] 666-nb-master
[48] 12:05:18 [FAILURE] netdata2-elm Exited with error code 255
[49] 12:05:18 [SUCCESS] imag4
[50] 12:05:18 [FAILURE] netdata3-elm Exited with error code 255
[51] 12:05:18 [SUCCESS] 666-nb-media1
[52] 12:05:18 [SUCCESS] gr-nb-master
[53] 12:05:18 [SUCCESS] md5b1700
[54] 12:05:18 [SUCCESS] md2a1700
[55] 12:05:18 [SUCCESS] p2ps666d
[56] 12:05:18 [SUCCESS] trace1
[57] 12:05:18 [SUCCESS] 666-nb-media2
[58] 12:05:18 [SUCCESS] p2ps666a
[59] 12:05:18 [SUCCESS] md19b1700
[60] 12:05:18 [SUCCESS] md4a1700
[61] 12:05:18 [SUCCESS] md1a1700
[62] 12:05:19 [SUCCESS] batch6
[63] 12:05:19 [SUCCESS] gr-nb-media1
[64] 12:05:19 [SUCCESS] md5a1700
[65] 12:05:19 [SUCCESS] batch7
[66] 12:05:19 [SUCCESS] rfp-nb-master
[67] 12:05:19 [SUCCESS] batch8
[68] 12:05:19 [SUCCESS] rfp-nb-media1
[69] 12:05:19 [SUCCESS] wp-nb-master
[70] 12:05:19 [SUCCESS] p2ps666i
[71] 12:05:19 [SUCCESS] p2ps666j
[72] 12:05:19 [SUCCESS] p2ps666k
[73] 12:05:19 [SUCCESS] p2ps666f
[74] 12:05:19 [SUCCESS] rfprep1
[75] 12:05:19 [SUCCESS] md18b1700
[76] 12:05:19 [SUCCESS] mxdevsrv
[77] 12:05:19 [SUCCESS] netdata3-666
[78] 12:05:19 [SUCCESS] p2ps666c
[79] 12:05:19 [SUCCESS] p2ps666b
[80] 12:05:19 [SUCCESS] mxqasrv
[81] 12:05:19 [SUCCESS] mxprodsrv3
[82] 12:05:19 [SUCCESS] mxdrsrv
[83] 12:05:19 [SUCCESS] mxprodsrv
[84] 12:05:19 [SUCCESS] mlp-dacs1
[85] 12:05:19 [SUCCESS] mxprodsrv2
[86] 12:05:19 [SUCCESS] p2ps666e
[87] 12:05:19 [SUCCESS] netdata1-666
[88] 12:05:19 [SUCCESS] mxprodsrv1
[89] 12:05:19 [SUCCESS] sim23
[90] 12:05:19 [SUCCESS] netgraph1
[91] 12:05:19 [SUCCESS] cttrdcap7
[92] 12:05:19 [SUCCESS] cttrdcap4
[93] 12:05:19 [SUCCESS] netgraph2
[94] 12:05:19 [SUCCESS] cttrdcap1
[95] 12:05:19 [SUCCESS] netdata4-666
[96] 12:05:19 [SUCCESS] netdata2-666
[97] 12:05:19 [SUCCESS] lnnis2
[98] 12:05:19 [SUCCESS] netdata2-1700
[99] 12:05:19 [SUCCESS] sim20
[100] 12:05:19 [SUCCESS] netdata1-wp
[101] 12:05:19 [SUCCESS] lnnis1
[102] 12:05:21 [SUCCESS] siadmin001
[103] 12:05:22 [FAILURE] mlpnfs2-new Timed out, Killed by signal 9
[104] 12:05:22 [FAILURE] nis5 Timed out, Killed by signal 9
[105] 12:05:22 [FAILURE] nis1 Timed out, Killed by signal 9
[106] 12:05:22 [FAILURE] nis3 Timed out, Killed by signal 9
[107] 12:05:22 [FAILURE] nis6 Timed out, Killed by signal 9
[108] 12:05:22 [FAILURE] nis7 Timed out, Killed by signal 9
[109] 12:05:23 [FAILURE] rfpnis1 Timed out, Killed by signal 9
[110] 12:05:23 [FAILURE] bosdweb1 Timed out, Killed by signal 9
[111] 12:05:23 [FAILURE] bosdweb2 Timed out, Killed by signal 9
[112] 12:05:23 [FAILURE] ssbatch3 Timed out, Killed by signal 9
[113] 12:05:23 [FAILURE] imagrep2 Timed out, Killed by signal 9
[114] 12:05:23 [FAILURE] hanuman Timed out, Killed by signal 9
[115] 12:05:23 [FAILURE] md19a1700 Timed out, Killed by signal 9
[116] 12:05:23 [FAILURE] md19a666 Timed out, Killed by signal 9
[117] 12:05:23 [FAILURE] md3b1700 Timed out, Killed by signal 9
[118] 12:05:23 [FAILURE] md4b1700 Timed out, Killed by signal 9
[119] 12:05:23 [FAILURE] md6a666-new Timed out, Killed by signal 9
[120] 12:05:23 [FAILURE] sink2a666 Timed out, Killed by signal 9
[121] 12:05:23 [FAILURE] imag6 Timed out, Killed by signal 9
[122] 12:05:23 [FAILURE] md2e1700 Timed out, Killed by signal 9
[123] 12:05:23 [FAILURE] md3a1700 Timed out, Killed by signal 9
[124] 12:05:23 [FAILURE] rs2 Timed out, Killed by signal 9
[125] 12:05:24 [FAILURE] ladybug Timed out, Killed by signal 9

notice that it will show me the success or failure of each host.

3) I can then cd /tmp/biffstuff for the output

[root@samojo ~]# cd /tmp/biffstuff
[root@samojo biffstuff]# ls
666-nb-master  ctnfs3         imag6         md2e1700     mxdevsrv       netdata4-666  p2ps666f       rs2
666-nb-media1  ctnfs4         imag9         md3a1700     mxdrsrv        netgraph1     p2ps666i       siadmin001
666-nb-media2  ctnfs5         imaghist2010  md3b1700     mxprodsrv      netgraph2     p2ps666j       sim20
batch6         ctnfs6         imagrep1      md4a1700     mxprodsrv1     nis1          p2ps666k       sim23
batch7         ctomsprod1     imagrep2      md4b1700     mxprodsrv2     nis3          pm1            sink2a666
batch8         cttrdcap1      ladybug       md5a1700     mxprodsrv3     nis5          pm2            ssbatch1
bosdbatch1     cttrdcap4      lnnis1        md5b1700     mxqasrv        nis6          recap1         ssbatch3
bosdbatch2     cttrdcap7      lnnis2        md6a666-new  mxrep1         nis7          rfpjumpstart   tlmsybdev
bosdbatch3     elmnis1        md18b1700     mddl-666     netdata1-666   nycprodsup01  rfp-nb-master  trace1
bosdbatch4     flash          md19a1700     mlp-dacs1    netdata1-elm   oms666-2      rfp-nb-media1  trdlogger
bosdbatch5     gr-nb-master   md19a666      mlpnfs1      netdata1-wp    omsprod1      rfpnfs10       wp-nb-master
bosdpm1        gr-nb-media1   md19b1700     mlpnfs2      netdata2-1700  p2ps666a      rfpnfs14       wpnfs1
bosdweb1       hanuman        md1a1700      mlpnfs2-new  netdata2-666   p2ps666b      rfpnis1        xray1
bosdweb2       homemade-new   md1b1700      mlpsyb       netdata2-elm   p2ps666c      rfprep1
ctnfs1         homemaker-new  md2a1700      mlpweb1      netdata3-666   p2ps666d      rfprep2
ctnfs2         imag4          md2b1700      monitor      netdata3-elm   p2ps666e      rs1
[root@samojo biffstuff]#
[root@samojo biffstuff]# 
[root@samojo biffstuff]# 
[root@samojo biffstuff]# cat ** 
12:05:24 up 198 days, 13:02,  0 users,  load average: 0.57, 0.15, 0.04
12:05:24 up 198 days, 13:08,  0 users,  load average: 0.00, 0.01, 0.00
12:05:24 up 62 days, 17:37,  0 users,  load average: 0.11, 0.13, 0.09
12:05pm  up 4 day(s), 10:49,  4 users,  load average: 0.04, 0.03, 0.06
12:05pm  up 4 day(s), 10:34,  0 users,  load average: 0.18, 0.43, 0.34
12:05pm  up 4 day(s), 10:49,  0 users,  load average: 0.03, 0.02, 0.04
12:05pm  up 332 day(s),  8:57,  1 user,  load average: 0.66, 0.55, 0.75
12:05pm  up 332 day(s), 20:14,  3 users,  load average: 6.55, 2.94, 1.67
12:05pm  up 689 day(s), 22:25,  0 users,  load average: 0.02, 0.07, 0.05
12:05pm  up 26 day(s), 20:31,  0 users,  load average: 0.00, 0.01, 0.00
12:05pm  up 786 day(s), 19:12,  1 user,  load average: 0.03, 0.04, 0.04
12:05pm  up 332 day(s), 20:29,  0 users,  load average: 0.12, 0.18, 0.18
12:05pm  up 786 day(s), 19:19,  0 users,  load average: 0.01, 0.01, 0.01
12:05pm  up 77 day(s), 38 min(s),  0 users,  load average: 0.31, 0.42, 1.13
12:05pm  up 54 day(s), 16:41,  1 user,  load average: 0.01, 0.01, 0.01
12:05pm  up 786 day(s), 19:19,  0 users,  load average: 0.01, 0.01, 0.01
12:05pm  up 2 day(s), 12:52,  1 user,  load average: 0.01, 0.01, 0.01
12:05pm  up 146 day(s), 22:46,  0 users,  load average: 0.01, 0.01, 0.01
12:05:23 up 92 days, 27 min,  0 users,  load average: 0.04, 0.03, 0.00
12:05:25 up 4 days, 11:01,  7 users,  load average: 1.48, 1.22, 1.28
12:05:25 up 4 days, 10:52,  4 users,  load average: 0.22, 0.39, 0.59
12:05:25 up 4 days, 10:52,  6 users,  load average: 0.22, 0.47, 0.51
12:05:23 up 658 days, 55 min,  0 users,  load average: 0.35, 0.94, 0.47
12:05pm  up 463 day(s), 13:11,  0 users,  load average: 0.00, 0.00, 0.01
12:05:24 up 12 days, 21:51,  1 user,  load average: 3.73, 3.79, 3.96
12:05:25 up 12 days, 21:51,  1 user,  load average: 0.50, 0.38, 0.36
12:05pm  up 1521 day(s),  3:22,  0 users,  load average: 0.00, 0.00, 0.01
12:05pm  up 1520 day(s), 23:28,  0 users,  load average: 0.00, 0.01, 0.02
12:05pm  up 37 day(s),  6:26,  0 users,  load average: 0.02, 0.01, 0.01
12:05pm  up 33 day(s), 18:01,  0 users,  load average: 0.04, 0.04, 0.04
17:05:25 up 207 days, 20:29,  1 user,  load average: 0.99, 0.85, 0.88
17:05:25 up 285 days, 21:04,  1 user,  load average: 0.50, 0.46, 0.40
12:05:25 up 4 days,  6:31,  0 users,  load average: 0.00, 0.00, 0.00
12:05:24 up 4 days,  7:02,  0 users,  load average: 0.06, 0.01, 0.00
12:05:25 up 4 days,  7:01,  0 users,  load average: 0.27, 0.55, 0.58
12:05:24 up 4 days,  7:01,  0 users,  load average: 0.65, 0.74, 0.86
12:05:24 up 4 days,  7:00,  0 users,  load average: 2.91, 2.99, 2.85
12:05:24 up 4 days,  7:01,  0 users,  load average: 1.79, 2.08, 2.36
12:05:24 up 4 days,  6:31,  0 users,  load average: 0.10, 0.09, 0.31
12:05:25 up 291 days,  3:16,  0 users,  load average: 1.72, 1.40, 1.09
12:05:24 up 291 days,  3:17,  0 users,  load average: 0.06, 0.56, 0.93
12:05:24 up 232 days, 15:26,  0 users,  load average: 0.97, 0.82, 0.51
12:05pm  up 522 day(s), 12:32,  1 user,  load average: 0.05, 0.05, 0.05
12:05:23 up 89 days, 18:00,  0 users,  load average: 0.03, 0.03, 0.00
12:05:23 up 89 days, 18:00,  0 users,  load average: 0.03, 0.03, 0.00
12:05pm  up 411 day(s), 17:14,  1 user,  load average: 0.11, 0.09, 0.08
12:05pm  up 311 day(s), 23:02,  0 users,  load average: 0.00, 0.01, 0.01
12:05pm  up 4 day(s),  5:02,  4 users,  load average: 0.79, 0.84, 0.81
12:05pm  up 59 day(s), 23:57,  6 users,  load average: 0.83, 0.70, 2.08
12:05pm  up 284 day(s),  3:44,  2 users,  load average: 0.21, 0.23, 0.22
12:05pm  up 150 day(s), 20:50,  3 users,  load average: 0.09, 0.09, 0.09
12:05pm  up 150 day(s), 20:47,  1 user,  load average: 1.76, 1.89, 2.00
12:05pm  up 52 day(s), 21:19,  9 users,  load average: 13.72, 13.29, 12.86
12:05pm  up 52 day(s), 21:18,  6 users,  load average: 19.36, 17.93, 16.74
12:05pm  up 284 day(s),  3:45,  0 users,  load average: 6.73, 5.39, 5.31
12:05pm  up 284 day(s),  3:45,  0 users,  load average: 0.00, 0.02, 0.07
12:05:25 up 490 days, 30 min,  1 user,  load average: 0.83, 0.46, 0.27
12:05:25 up 372 days, 18:19,  1 user,  load average: 0.15, 0.66, 0.56
12:05:25 up 29 days, 19:21,  1 user,  load average: 0.01, 0.01, 0.00
12:05:25 up 658 days, 21:05,  1 user,  load average: 0.26, 0.14, 0.11
12:05:25 up 722 days, 23:16,  3 users,  load average: 0.00, 0.00, 0.00
12:05:25 up 448 days, 22:41,  1 user,  load average: 0.00, 0.00, 0.00
12:05:25 up 83 days, 16:11,  0 users,  load average: 66.82, 74.28, 73.96
12:05:25 up 1084 days, 21:54,  0 users,  load average: 42.09, 41.04, 40.49
12:04pm  up 305 day(s), 18:26,  0 users,  load average: 0.07, 0.07, 0.04
12:05:24 up 251 days, 14 min,  0 users,  load average: 0.16, 0.11, 0.28
12:05:24 up 3 days, 12:05,  0 users,  load average: 0.04, 0.18, 0.39
12:05:24 up 4 days, 17:17,  0 users,  load average: 0.20, 0.15, 0.17
12:05:25 up 3 days,  7:02,  0 users,  load average: 0.42, 0.40, 0.37
12:05:25 up 4 days, 17:18,  0 users,  load average: 0.07, 0.08, 0.08
12:05:24 up 3 days,  7:03,  0 users,  load average: 0.19, 0.24, 0.22
12:05:25 up 4 days, 17:18,  0 users,  load average: 0.14, 0.13, 0.12
12:05:25 up 3 days,  7:03,  0 users,  load average: 0.04, 0.12, 0.12
12:05:25 up 4 days, 17:18,  0 users,  load average: 0.17, 0.05, 0.01
12:05:25 up 3 days,  7:03,  0 users,  load average: 0.14, 0.18, 0.17
12:05:25 up 2 days, 22:02,  0 users,  load average: 0.13, 0.15, 0.16
12:05pm  up 457 day(s), 16:01,  0 users,  load average: 0.00, 0.01, 0.07
12:05pm  up 550 day(s), 17:49,  0 users,  load average: 0.00, 0.00, 0.01
12:05pm  up 385 day(s), 21:31,  6 users,  load average: 0.40, 0.21, 0.18
12:05pm  up 46 day(s), 21:48,  1 user,  load average: 0.02, 0.02, 0.02
12:05:25 up 103 days, 22:42,  1 user,  load average: 0.05, 0.03, 0.00
12:05:25 up 162 days, 22:58,  0 users,  load average: 0.00, 0.01, 0.06
12:05pm  up 144 day(s), 17:54,  0 users,  load average: 0.01, 0.21, 0.38
12:05pm  up 115 day(s), 21:46,  0 users,  load average: 0.01, 0.01, 0.01
12:05pm  up 106 day(s), 16:13,  0 users,  load average: 0.05, 0.04, 0.04
12:05pm  up 652 day(s),  4:14,  0 users,  load average: 0.00, 0.00, 0.01
12:05am  up 257 day(s), 12:38,  0 users,  load average: 2.00, 2.00, 2.00
12:05:25 up 25 days, 19:17, 27 users,  load average: 1.48, 1.58, 1.61
12:05:25 up 25 days, 18:32, 25 users,  load average: 0.20, 0.22, 0.20
12:05pm  up 4512 day(s), 15:52,  0 users,  load average: 0.04, 0.04, 0.03
12:05:25 up 5 days, 22:35,  1 user,  load average: 6.96, 7.01, 6.47
12:05pm  up 396 day(s), 18:58,  0 users,  load average: 0.01, 0.01, 0.01
12:05pm  up 292 day(s), 16:50,  0 users,  load average: 0.07, 0.11, 0.14
[root@samojo biffstuff]#

to give you an idea how fast this runs, the above command does an uptime on 125 machines, It ran in just over 7 seconds

real    0m7.257s
user    0m4.686s
sys     0m2.949s
[root@samojo ~]# 


The other utilities in the package all work similiarly. If you're a busy sysadmin, I'd check out some of these tools