Make sure Standard Resolution (480i), 480, 720p and 1080i are checked. This will depend on the resolution of your TV supports. For example, if you have a 720 flat screen, you will not be able to capture in 1080i provided your game supports the resolution.
Make sure that the Xbox component cable is set to "HD" rather than "TV".
If the original connection from PS3 to TV is through HDMI, it is likely you have this setting checked on your PS3 if you have a more recent HDTV.
Roxio Game Capture - Getting Started with the PS3 (video)
Roxio Game Capture - Getting Started with the Xbox 360 (video)
Roxio Game Capture Setup Guide (pdf)