Sequentially Splitting the load on a Parallel.Foreach Loop
I have a million elements in a List to process.
Dropping them crudely into a Parallel.ForEach would just saturate the CPU.
Instead I split the Elements Master Lists into pieces and drop the Sublists into a parallel loop.
List<Element> MasterList = new List<Element>();
Populate(MasterList); // puts a Million elements into list;
//Split Master List into 100 Lists of 10.0000 elements each
List<List<Element>> ListOfSubLists = Split(MasterList,100);
foreach (List<Element> EL in ListOfSubLists )
{
Parallel.ForEach(EL, E =>
{
// Do Stuff
}
//wait for all parallel iterations to end before continuing
}
What is the best way for waiting for all parallel iteration to end before continuing to next iteration of the upper loop ?
Edit :
as some answers stated, "saturate the CPU" is not an accurate expression.
Actually I just want to limit the CPU usage, and avoid excessive load on it coming from this processing.
Parallel.ForEach will not saturate the CPU; it uses some intelligence to decide how many parallel threads to run simultaneously, with a max of 63.
See: Does Parallel.ForEach limits the number of active threads?
You can also set the max degree of parallelism if you want, by supplying a ParallelOptions like new ParallelOptions { MaxDegreeOfParallelism = 5 } as the second argument to Parallel.ForEach .
As a last point, Parallel.ForEach blocks until all of the iterations have completed. So your code, as written, works. You do not need to wait for the iterations to complete.
What do you mean "Saturate the CPU"
You can still throttle the Parallel foreach loop by supplying it with ParallelOptions One property of which is a MaxDegreesOfParallelism
That will allow you to go back to your single collection eg
Parallel.ForEach(
collection,
new ParallelOptions { MaxDegreeOfParallelism = 5},
E => { DoStuff(E) }
);
上一篇: CPU绑定任务的并行化继续IO绑定