vlc (1.1.3-1squeeze6) cdg-heap-overflow.diff

Author: Dan Rosenberg <drosenberg@vsecurity.com>
Subject: Fix heap overflows in CDG decoder
 This patch resolves two heap corruption vulnerabilities in the CDG
 decoder for VLC media player.  In both cases, a failure to properly
 validate indexes into statically-sized arrays on the heap could allow a
 maliciously crafted CDG video to corrupt the heap in a controlled
 manner, potentially leading to code execution.
 .
 The patch is against v1.1.5 from vlc git, but this decoder hasn't been
 touched in awhile, so I'd expect it to cleanly apply to older versions.
 I've tested it and confirmed it resolves the heap corruption issues and
 does not break functionality.
Origin: upstream, http://git.videolan.org/gitweb.cgi?p=vlc/vlc-1.1.git;a=commit;h=d11fca8bf9dc058bcdf67d81c04f84f8905ad8b4

diff --git a/modules/codec/cdg.c b/modules/codec/cdg.c
index 31ecd0e..fe7b62d 100644
--- a/modules/codec/cdg.c
+++ b/modules/codec/cdg.c
@@ -254,7 +254,13 @@ static int DecodeTileBlock( decoder_sys_t *p_cdg, const uint8_t *p_data, int doX
         for( x = 0; x < 6; x++ )
         {
             const int idx = ( p_data[4+y] >> (5-x) ) & 0x01;
-            uint8_t *p = &p_cdg->p_screen[(sy+y)*CDG_SCREEN_PITCH+(sx+x)];
+
+            int index = (sy+y)*CDG_SCREEN_PITCH+(sx+x);
+            if( index >= CDG_SCREEN_PITCH*CDG_SCREEN_HEIGHT )
+                return 0;
+
+            uint8_t *p = &p_cdg->p_screen[index];
+
             if( doXor )
                 *p ^= p_color[idx];
             else
@@ -319,8 +325,8 @@ static int DecodeScroll( decoder_sys_t *p_cdg, const uint8_t *p_data, int b_copy
 
             if( b_copy )
             {
-                dy = ( dy + CDG_SCREEN_HEIGHT ) % CDG_SCREEN_HEIGHT;
-                dy = ( dy + CDG_SCREEN_WIDTH  ) % CDG_SCREEN_WIDTH;
+                dy %= CDG_SCREEN_HEIGHT;
+                dx %= CDG_SCREEN_WIDTH;
             }
             else
             {