If anyone else might run into this issue, I would like to share the results and the status. There is in fact a solution - sort of.
As it turns out, the stutter experienced with multiple instances on the same process (application) was only present when streaming YouTube clips. It is NOT a problem when playing back local video files or when playing a rtsp stream over the network. This means the problem is YouTube is not caused by the network nor VLC's ability to handle several instances but something else.
I have tested running the following VLC instances simultaneously in the same WPF application on a Win7 computer:
- two 1080p video streams
- One YouTube video playing back 1080p source (I think)
- four mpeg4 video's playing back 1080p source
Notice that one YouTube video does not kill the system, but having two will. The CPU was pushed to 80% with 5 players and 100% with the 7 players above.
I will leave this one for now but hope that the findings can help others with knowing what works and what does not work.