cat webpages.txt | xargs wget
grep -l reddit *.*
grep -l reddit *.* | xargs -n1 ./process.sh
grep -l reddit *.* | xargs -n1 -P32 ./process.sh
#!/bin/bash
cat webpages.txt | xargs wget -P pages
grep -l reddit pages/*.* | xargs -n1 -P32 ./process.sh
Problems fall into two categories:
the effort expended on achieving high parallel processing rates is wasted unless it is accompanied by achievements in sequential processing rates of very nearly the same magnitude.