This article explains the reason why some users are having issues with Blu-ray playback on WinDVD Pro 11.
WinDVD is usually trying to establish a digital connection with your monitor to determine if it meets the requirements for the AACS licensing demands. The software will not be able establish that connection properly if you are using a converter going from DVI to HDMI in your setup. The only solution is to try and remove the converter from the setup and use a straight DVI or straight HDMI connection between the video card and the monitor.
Also, please make sure that you are running the latest build version of WinDVD Pro 11: Patches and Updates