Monday, April 1, 2013

Using xargs or parallel to parallelize processes on linux

The command line:

find . -name '.txt' | xargs -n 1 -P 10 -I % sh -c 'cat % | perl command.pl > %.out'

The arguments:
-n number of arguments passed at each line, might not be useful here
-P number of processors used - enable parallelization of the processes
-I % get the argument passed by the command.

Equivalent command line with parallel:

find . -name '.txt' | parallel 'perl command.pl {} > {.}.out'
The arguments:
{} the argument
{.} the argument without extension

That's all folks!

No comments:

Post a Comment