I've searched around now on Google about audiodg.exe and found another guy that had the same problem, but noone told him how to fix it.
Whenever I push my speakbutton in Ventrilo, the memory audiodg.exe uses increases by about 0,5-2 mb. This does not ever stop, if I stop the process that's it, no sound. If I close Ventrilo nothing happens.
This explains why my computer eventually gets slowed down like hell and sometimes makes my sound disappear when playing games.
My soundcard is a Creative SB X-Fi, I've got the proper drivers
Seriously, what is this process for and why is it buggy like hell?