GPU's have been discussed, but not together with MPI like stuff. It appears that MPI has a lot of effort behind it. They have incorporated what (for me) were the superior features of PVM. they also support shared memory which gets you closer to the equivalent of multi-threading if desired. Your GPU reference looks promising.