You are most likely trying to capture a game at a resolution higher than the video game supports. Roxio Game Capture only supports a resolution up to 1080i if the game does. However, most of modern HD games support 720 natively. You can generally see what resolution your game supports on the back of the game's packaging.
Make sure Standard Resolution (480i), 480, 720p and 1080i are checked. This will depend on the resolution of your TV supports. For example, if you have a 720 flat screen, you will not be able to capture in 1080i provided your game supports the resolution.
Make sure that the Xbox component cable is set to "HD" rather than "TV".
If the original connection from PS3 to TV is through HDMI, it is likely you have this setting checked on your PS3 if you have a more recent HDTV.
Roxio Game Capture - Getting Started with the PS3 (video)
Roxio Game Capture - Getting Started with the Xbox 360 (video)
Roxio Game Capture Setup Guide (pdf)