Ok, you owe me one!!!! I have your answer.
The reason why it is doing this is because .mp4's are generally larger files and are hard to process. You need to convert it to a .avi for any movie longer than a minute or two. You need to set up your sony vegas correctly, I usually set mine to ntsd widescreen, this works well with my HD camera [which also shoots to .mp4, so we are in the same boat] but you'll just have to mess around. Make sure you set the settings TO MATCH THE FRAME RATE AND ASPECT RATIO OF YOUR ORIGINAL FILE. If your video camera shoots at 30 fps, then you need to set up your vegas file to be the same. You can do this under preferances.
Now to the conversion proces... I use PAZERA FREE MP4 TO AVI and it works FANTASTIC. Sometimes you will have to do it twice, but it works. Its 100 precent free. And its pretty fast. You can download it
http://download.cnet.com/Pazera-Free-MP4-to-AVI-Converter/3000-2194_4-10784027.html by clicking that link.
When you are setting it up, use the MAX bitrate, and the SAME FRAME RATE YOUR CAMERA SHOOTS IN. 2 pass encoding isn't necessary and can take FOREVER. Also make sure to adjust the RESOLUTION to match the original resolution YOUR CAMERA SHOOTS IN.
Set it up so that it dumps all the NEW AVI files into their own folder. It works really really well with vegas to do it that way.
I just saved you like 2 months worth of work... I'm not even lying. It TOOK ME SO LONG to figure this out. You owe me
If you need help, pm me. I'll post my answer here so others can read it.