You're not the only one. I've had my iPod Touch 4G since last August and have always had slight issues with the battery. It was the same with my previous touch 4G which I got replaced due to a faulty home button.
I think the issue is software based rather thank hardware. I found the same thing on a long coach journey yesterday. If you can still get about 6/7hrs of video watching from a full charge, the battery is fine. My reasoning this this. You play a game, probably the most battery intensive thing to do. When you play, the iPod knows it is doing something battery intensive and drops the meter. However, as soon as you stop and put it to sleep with the screen off, the device is in a very low power state. There is a big jump and the meter goes back up again. However, it doesn't stay up for as long as it normally does, it drops quickly.
I did a test. My battery dropped to 22% according to Battery Pro and the low battery alert appeared. I locked the device for two minutes. When I unlocked it and opened Battery Pro, it said 42%, the same as the built in meter. But, within about 20 seconds, the meter had dropped to 27%. I consider this to be the actual amount at the time.
Hope this helps