Earlier this year, my worst experience with playing those HD video files (I was watching "Iris", yes, it's a Korean action drama), was that there would split-second freezes here and there but no green artefacts littering VLC's video window like I see now.
Perhaps it's CUDA driver related? Maybe. I should get some time to test whether that's the root of evil. It's either that or the 512MB rated for the Total VRAM is only eye-candy.
Update: Nope, it is not eye candy. Use Plex player for Mac OS X and you'll see that the Mini 311 can handle 720p .mkv quite smoothly.