Parallel Computing with Shell Commands
There are some great parallel utilities that you can use on Linux and UNIX that will make work a lot easier for you
* Parallel ssh (pssh)
* Parallel scp (pscp)
* Parallel rsync (prsync)
* Parallel nuke (pnuke)
* Parallel slurp (pslurp)
You can download them at http://code.google.com/p/parallel-ssh/downloads/list/
I'll use pssh as an example of how all of these utilities work
1) First I'll create a file called myhosts. Each line of that file contains a hostname (one per line)
2) Now I can run this command:
pssh -h myhosts -o /tmp/biffstuff -t 5 "uptime"
-h myhosts is the filename with my list of hosts
-o is the directory to put the output (one file per host)
-t is timeout after n seconds
The command goes at the end between quotes
When I run the command the screen output will look like this:
[1] 12:05:17 [SUCCESS] ctnfs1
[2] 12:05:17 [SUCCESS] ctnfs2
[3] 12:05:17 [SUCCESS] ctnfs3
[4] 12:05:17 [SUCCESS] ctnfs4
[5] 12:05:17 [SUCCESS] ctnfs5
[6] 12:05:17 [SUCCESS] ctnfs6
[7] 12:05:17 [SUCCESS] elmnis1
[8] 12:05:17 [SUCCESS] flash
[9] 12:05:17 [SUCCESS] mlpnfs1
[10] 12:05:17 [SUCCESS] mlpnfs2
[11] 12:05:17 [SUCCESS] mlpweb1
[12] 12:05:17 [SUCCESS] monitor
[13] 12:05:17 [FAILURE] trdlogger Exited with error code 255
[14] 12:05:17 [FAILURE] imagrep1 Exited with error code 255
[15] 12:05:17 [FAILURE] rfprep2 Exited with error code 255
[16] 12:05:17 [FAILURE] tlmsybdev Exited with error code 255
[17] 12:05:17 [SUCCESS] bosdbatch1
[18] 12:05:17 [FAILURE] ssbatch1 Exited with error code 255
[19] 12:05:17 [FAILURE] imag9 Exited with error code 255
[20] 12:05:17 [SUCCESS] rfpnfs14
[21] 12:05:17 [SUCCESS] ctomsprod1
[22] 12:05:17 [SUCCESS] rfpnfs10
[23] 12:05:17 [SUCCESS] wpnfs1
[24] 12:05:17 [SUCCESS] oms666-2
[25] 12:05:17 [SUCCESS] bosdbatch2
[26] 12:05:17 [SUCCESS] bosdpm1
[27] 12:05:17 [SUCCESS] pm1
[28] 12:05:17 [SUCCESS] pm2
[29] 12:05:17 [SUCCESS] homemade-new
[30] 12:05:17 [SUCCESS] homemaker-new
[31] 12:05:18 [SUCCESS] xray1
[32] 12:05:18 [SUCCESS] omsprod1
[33] 12:05:18 [SUCCESS] mddl-666
[34] 12:05:18 [SUCCESS] rfpjumpstart
[35] 12:05:18 [SUCCESS] bosdbatch5
[36] 12:05:18 [SUCCESS] rs1
[37] 12:05:18 [SUCCESS] imaghist2010
[38] 12:05:18 [SUCCESS] mxrep1
[39] 12:05:18 [SUCCESS] bosdbatch3
[40] 12:05:18 [SUCCESS] bosdbatch4
[41] 12:05:18 [SUCCESS] mlpsyb
[42] 12:05:18 [SUCCESS] recap1
[43] 12:05:18 [SUCCESS] md1b1700
[44] 12:05:18 [FAILURE] netdata1-elm Exited with error code 255
[45] 12:05:18 [SUCCESS] nycprodsup01
[46] 12:05:18 [SUCCESS] md2b1700
[47] 12:05:18 [SUCCESS] 666-nb-master
[48] 12:05:18 [FAILURE] netdata2-elm Exited with error code 255
[49] 12:05:18 [SUCCESS] imag4
[50] 12:05:18 [FAILURE] netdata3-elm Exited with error code 255
[51] 12:05:18 [SUCCESS] 666-nb-media1
[52] 12:05:18 [SUCCESS] gr-nb-master
[53] 12:05:18 [SUCCESS] md5b1700
[54] 12:05:18 [SUCCESS] md2a1700
[55] 12:05:18 [SUCCESS] p2ps666d
[56] 12:05:18 [SUCCESS] trace1
[57] 12:05:18 [SUCCESS] 666-nb-media2
[58] 12:05:18 [SUCCESS] p2ps666a
[59] 12:05:18 [SUCCESS] md19b1700
[60] 12:05:18 [SUCCESS] md4a1700
[61] 12:05:18 [SUCCESS] md1a1700
[62] 12:05:19 [SUCCESS] batch6
[63] 12:05:19 [SUCCESS] gr-nb-media1
[64] 12:05:19 [SUCCESS] md5a1700
[65] 12:05:19 [SUCCESS] batch7
[66] 12:05:19 [SUCCESS] rfp-nb-master
[67] 12:05:19 [SUCCESS] batch8
[68] 12:05:19 [SUCCESS] rfp-nb-media1
[69] 12:05:19 [SUCCESS] wp-nb-master
[70] 12:05:19 [SUCCESS] p2ps666i
[71] 12:05:19 [SUCCESS] p2ps666j
[72] 12:05:19 [SUCCESS] p2ps666k
[73] 12:05:19 [SUCCESS] p2ps666f
[74] 12:05:19 [SUCCESS] rfprep1
[75] 12:05:19 [SUCCESS] md18b1700
[76] 12:05:19 [SUCCESS] mxdevsrv
[77] 12:05:19 [SUCCESS] netdata3-666
[78] 12:05:19 [SUCCESS] p2ps666c
[79] 12:05:19 [SUCCESS] p2ps666b
[80] 12:05:19 [SUCCESS] mxqasrv
[81] 12:05:19 [SUCCESS] mxprodsrv3
[82] 12:05:19 [SUCCESS] mxdrsrv
[83] 12:05:19 [SUCCESS] mxprodsrv
[84] 12:05:19 [SUCCESS] mlp-dacs1
[85] 12:05:19 [SUCCESS] mxprodsrv2
[86] 12:05:19 [SUCCESS] p2ps666e
[87] 12:05:19 [SUCCESS] netdata1-666
[88] 12:05:19 [SUCCESS] mxprodsrv1
[89] 12:05:19 [SUCCESS] sim23
[90] 12:05:19 [SUCCESS] netgraph1
[91] 12:05:19 [SUCCESS] cttrdcap7
[92] 12:05:19 [SUCCESS] cttrdcap4
[93] 12:05:19 [SUCCESS] netgraph2
[94] 12:05:19 [SUCCESS] cttrdcap1
[95] 12:05:19 [SUCCESS] netdata4-666
[96] 12:05:19 [SUCCESS] netdata2-666
[97] 12:05:19 [SUCCESS] lnnis2
[98] 12:05:19 [SUCCESS] netdata2-1700
[99] 12:05:19 [SUCCESS] sim20
[100] 12:05:19 [SUCCESS] netdata1-wp
[101] 12:05:19 [SUCCESS] lnnis1
[102] 12:05:21 [SUCCESS] siadmin001
[103] 12:05:22 [FAILURE] mlpnfs2-new Timed out, Killed by signal 9
[104] 12:05:22 [FAILURE] nis5 Timed out, Killed by signal 9
[105] 12:05:22 [FAILURE] nis1 Timed out, Killed by signal 9
[106] 12:05:22 [FAILURE] nis3 Timed out, Killed by signal 9
[107] 12:05:22 [FAILURE] nis6 Timed out, Killed by signal 9
[108] 12:05:22 [FAILURE] nis7 Timed out, Killed by signal 9
[109] 12:05:23 [FAILURE] rfpnis1 Timed out, Killed by signal 9
[110] 12:05:23 [FAILURE] bosdweb1 Timed out, Killed by signal 9
[111] 12:05:23 [FAILURE] bosdweb2 Timed out, Killed by signal 9
[112] 12:05:23 [FAILURE] ssbatch3 Timed out, Killed by signal 9
[113] 12:05:23 [FAILURE] imagrep2 Timed out, Killed by signal 9
[114] 12:05:23 [FAILURE] hanuman Timed out, Killed by signal 9
[115] 12:05:23 [FAILURE] md19a1700 Timed out, Killed by signal 9
[116] 12:05:23 [FAILURE] md19a666 Timed out, Killed by signal 9
[117] 12:05:23 [FAILURE] md3b1700 Timed out, Killed by signal 9
[118] 12:05:23 [FAILURE] md4b1700 Timed out, Killed by signal 9
[119] 12:05:23 [FAILURE] md6a666-new Timed out, Killed by signal 9
[120] 12:05:23 [FAILURE] sink2a666 Timed out, Killed by signal 9
[121] 12:05:23 [FAILURE] imag6 Timed out, Killed by signal 9
[122] 12:05:23 [FAILURE] md2e1700 Timed out, Killed by signal 9
[123] 12:05:23 [FAILURE] md3a1700 Timed out, Killed by signal 9
[124] 12:05:23 [FAILURE] rs2 Timed out, Killed by signal 9
[125] 12:05:24 [FAILURE] ladybug Timed out, Killed by signal 9
notice that it will show me the success or failure of each host.
3) I can then cd /tmp/biffstuff for the output
[root@samojo ~]# cd /tmp/biffstuff
[root@samojo biffstuff]# ls
666-nb-master ctnfs3 imag6 md2e1700 mxdevsrv netdata4-666 p2ps666f rs2
666-nb-media1 ctnfs4 imag9 md3a1700 mxdrsrv netgraph1 p2ps666i siadmin001
666-nb-media2 ctnfs5 imaghist2010 md3b1700 mxprodsrv netgraph2 p2ps666j sim20
batch6 ctnfs6 imagrep1 md4a1700 mxprodsrv1 nis1 p2ps666k sim23
batch7 ctomsprod1 imagrep2 md4b1700 mxprodsrv2 nis3 pm1 sink2a666
batch8 cttrdcap1 ladybug md5a1700 mxprodsrv3 nis5 pm2 ssbatch1
bosdbatch1 cttrdcap4 lnnis1 md5b1700 mxqasrv nis6 recap1 ssbatch3
bosdbatch2 cttrdcap7 lnnis2 md6a666-new mxrep1 nis7 rfpjumpstart tlmsybdev
bosdbatch3 elmnis1 md18b1700 mddl-666 netdata1-666 nycprodsup01 rfp-nb-master trace1
bosdbatch4 flash md19a1700 mlp-dacs1 netdata1-elm oms666-2 rfp-nb-media1 trdlogger
bosdbatch5 gr-nb-master md19a666 mlpnfs1 netdata1-wp omsprod1 rfpnfs10 wp-nb-master
bosdpm1 gr-nb-media1 md19b1700 mlpnfs2 netdata2-1700 p2ps666a rfpnfs14 wpnfs1
bosdweb1 hanuman md1a1700 mlpnfs2-new netdata2-666 p2ps666b rfpnis1 xray1
bosdweb2 homemade-new md1b1700 mlpsyb netdata2-elm p2ps666c rfprep1
ctnfs1 homemaker-new md2a1700 mlpweb1 netdata3-666 p2ps666d rfprep2
ctnfs2 imag4 md2b1700 monitor netdata3-elm p2ps666e rs1
[root@samojo biffstuff]#
[root@samojo biffstuff]#
[root@samojo biffstuff]#
[root@samojo biffstuff]# cat **
12:05:24 up 198 days, 13:02, 0 users, load average: 0.57, 0.15, 0.04
12:05:24 up 198 days, 13:08, 0 users, load average: 0.00, 0.01, 0.00
12:05:24 up 62 days, 17:37, 0 users, load average: 0.11, 0.13, 0.09
12:05pm up 4 day(s), 10:49, 4 users, load average: 0.04, 0.03, 0.06
12:05pm up 4 day(s), 10:34, 0 users, load average: 0.18, 0.43, 0.34
12:05pm up 4 day(s), 10:49, 0 users, load average: 0.03, 0.02, 0.04
12:05pm up 332 day(s), 8:57, 1 user, load average: 0.66, 0.55, 0.75
12:05pm up 332 day(s), 20:14, 3 users, load average: 6.55, 2.94, 1.67
12:05pm up 689 day(s), 22:25, 0 users, load average: 0.02, 0.07, 0.05
12:05pm up 26 day(s), 20:31, 0 users, load average: 0.00, 0.01, 0.00
12:05pm up 786 day(s), 19:12, 1 user, load average: 0.03, 0.04, 0.04
12:05pm up 332 day(s), 20:29, 0 users, load average: 0.12, 0.18, 0.18
12:05pm up 786 day(s), 19:19, 0 users, load average: 0.01, 0.01, 0.01
12:05pm up 77 day(s), 38 min(s), 0 users, load average: 0.31, 0.42, 1.13
12:05pm up 54 day(s), 16:41, 1 user, load average: 0.01, 0.01, 0.01
12:05pm up 786 day(s), 19:19, 0 users, load average: 0.01, 0.01, 0.01
12:05pm up 2 day(s), 12:52, 1 user, load average: 0.01, 0.01, 0.01
12:05pm up 146 day(s), 22:46, 0 users, load average: 0.01, 0.01, 0.01
12:05:23 up 92 days, 27 min, 0 users, load average: 0.04, 0.03, 0.00
12:05:25 up 4 days, 11:01, 7 users, load average: 1.48, 1.22, 1.28
12:05:25 up 4 days, 10:52, 4 users, load average: 0.22, 0.39, 0.59
12:05:25 up 4 days, 10:52, 6 users, load average: 0.22, 0.47, 0.51
12:05:23 up 658 days, 55 min, 0 users, load average: 0.35, 0.94, 0.47
12:05pm up 463 day(s), 13:11, 0 users, load average: 0.00, 0.00, 0.01
12:05:24 up 12 days, 21:51, 1 user, load average: 3.73, 3.79, 3.96
12:05:25 up 12 days, 21:51, 1 user, load average: 0.50, 0.38, 0.36
12:05pm up 1521 day(s), 3:22, 0 users, load average: 0.00, 0.00, 0.01
12:05pm up 1520 day(s), 23:28, 0 users, load average: 0.00, 0.01, 0.02
12:05pm up 37 day(s), 6:26, 0 users, load average: 0.02, 0.01, 0.01
12:05pm up 33 day(s), 18:01, 0 users, load average: 0.04, 0.04, 0.04
17:05:25 up 207 days, 20:29, 1 user, load average: 0.99, 0.85, 0.88
17:05:25 up 285 days, 21:04, 1 user, load average: 0.50, 0.46, 0.40
12:05:25 up 4 days, 6:31, 0 users, load average: 0.00, 0.00, 0.00
12:05:24 up 4 days, 7:02, 0 users, load average: 0.06, 0.01, 0.00
12:05:25 up 4 days, 7:01, 0 users, load average: 0.27, 0.55, 0.58
12:05:24 up 4 days, 7:01, 0 users, load average: 0.65, 0.74, 0.86
12:05:24 up 4 days, 7:00, 0 users, load average: 2.91, 2.99, 2.85
12:05:24 up 4 days, 7:01, 0 users, load average: 1.79, 2.08, 2.36
12:05:24 up 4 days, 6:31, 0 users, load average: 0.10, 0.09, 0.31
12:05:25 up 291 days, 3:16, 0 users, load average: 1.72, 1.40, 1.09
12:05:24 up 291 days, 3:17, 0 users, load average: 0.06, 0.56, 0.93
12:05:24 up 232 days, 15:26, 0 users, load average: 0.97, 0.82, 0.51
12:05pm up 522 day(s), 12:32, 1 user, load average: 0.05, 0.05, 0.05
12:05:23 up 89 days, 18:00, 0 users, load average: 0.03, 0.03, 0.00
12:05:23 up 89 days, 18:00, 0 users, load average: 0.03, 0.03, 0.00
12:05pm up 411 day(s), 17:14, 1 user, load average: 0.11, 0.09, 0.08
12:05pm up 311 day(s), 23:02, 0 users, load average: 0.00, 0.01, 0.01
12:05pm up 4 day(s), 5:02, 4 users, load average: 0.79, 0.84, 0.81
12:05pm up 59 day(s), 23:57, 6 users, load average: 0.83, 0.70, 2.08
12:05pm up 284 day(s), 3:44, 2 users, load average: 0.21, 0.23, 0.22
12:05pm up 150 day(s), 20:50, 3 users, load average: 0.09, 0.09, 0.09
12:05pm up 150 day(s), 20:47, 1 user, load average: 1.76, 1.89, 2.00
12:05pm up 52 day(s), 21:19, 9 users, load average: 13.72, 13.29, 12.86
12:05pm up 52 day(s), 21:18, 6 users, load average: 19.36, 17.93, 16.74
12:05pm up 284 day(s), 3:45, 0 users, load average: 6.73, 5.39, 5.31
12:05pm up 284 day(s), 3:45, 0 users, load average: 0.00, 0.02, 0.07
12:05:25 up 490 days, 30 min, 1 user, load average: 0.83, 0.46, 0.27
12:05:25 up 372 days, 18:19, 1 user, load average: 0.15, 0.66, 0.56
12:05:25 up 29 days, 19:21, 1 user, load average: 0.01, 0.01, 0.00
12:05:25 up 658 days, 21:05, 1 user, load average: 0.26, 0.14, 0.11
12:05:25 up 722 days, 23:16, 3 users, load average: 0.00, 0.00, 0.00
12:05:25 up 448 days, 22:41, 1 user, load average: 0.00, 0.00, 0.00
12:05:25 up 83 days, 16:11, 0 users, load average: 66.82, 74.28, 73.96
12:05:25 up 1084 days, 21:54, 0 users, load average: 42.09, 41.04, 40.49
12:04pm up 305 day(s), 18:26, 0 users, load average: 0.07, 0.07, 0.04
12:05:24 up 251 days, 14 min, 0 users, load average: 0.16, 0.11, 0.28
12:05:24 up 3 days, 12:05, 0 users, load average: 0.04, 0.18, 0.39
12:05:24 up 4 days, 17:17, 0 users, load average: 0.20, 0.15, 0.17
12:05:25 up 3 days, 7:02, 0 users, load average: 0.42, 0.40, 0.37
12:05:25 up 4 days, 17:18, 0 users, load average: 0.07, 0.08, 0.08
12:05:24 up 3 days, 7:03, 0 users, load average: 0.19, 0.24, 0.22
12:05:25 up 4 days, 17:18, 0 users, load average: 0.14, 0.13, 0.12
12:05:25 up 3 days, 7:03, 0 users, load average: 0.04, 0.12, 0.12
12:05:25 up 4 days, 17:18, 0 users, load average: 0.17, 0.05, 0.01
12:05:25 up 3 days, 7:03, 0 users, load average: 0.14, 0.18, 0.17
12:05:25 up 2 days, 22:02, 0 users, load average: 0.13, 0.15, 0.16
12:05pm up 457 day(s), 16:01, 0 users, load average: 0.00, 0.01, 0.07
12:05pm up 550 day(s), 17:49, 0 users, load average: 0.00, 0.00, 0.01
12:05pm up 385 day(s), 21:31, 6 users, load average: 0.40, 0.21, 0.18
12:05pm up 46 day(s), 21:48, 1 user, load average: 0.02, 0.02, 0.02
12:05:25 up 103 days, 22:42, 1 user, load average: 0.05, 0.03, 0.00
12:05:25 up 162 days, 22:58, 0 users, load average: 0.00, 0.01, 0.06
12:05pm up 144 day(s), 17:54, 0 users, load average: 0.01, 0.21, 0.38
12:05pm up 115 day(s), 21:46, 0 users, load average: 0.01, 0.01, 0.01
12:05pm up 106 day(s), 16:13, 0 users, load average: 0.05, 0.04, 0.04
12:05pm up 652 day(s), 4:14, 0 users, load average: 0.00, 0.00, 0.01
12:05am up 257 day(s), 12:38, 0 users, load average: 2.00, 2.00, 2.00
12:05:25 up 25 days, 19:17, 27 users, load average: 1.48, 1.58, 1.61
12:05:25 up 25 days, 18:32, 25 users, load average: 0.20, 0.22, 0.20
12:05pm up 4512 day(s), 15:52, 0 users, load average: 0.04, 0.04, 0.03
12:05:25 up 5 days, 22:35, 1 user, load average: 6.96, 7.01, 6.47
12:05pm up 396 day(s), 18:58, 0 users, load average: 0.01, 0.01, 0.01
12:05pm up 292 day(s), 16:50, 0 users, load average: 0.07, 0.11, 0.14
[root@samojo biffstuff]#
to give you an idea how fast this runs, the above command does an uptime on 125 machines, It ran in just over 7 seconds
real 0m7.257s
user 0m4.686s
sys 0m2.949s
[root@samojo ~]#
The other utilities in the package all work similiarly. If you're a busy sysadmin, I'd check out some of these tools