If you dont need features like XP mode, domain join, bitlocker, MUI language support and imaging to network, you're just fine with home premium. So, basically this edition is for the typical home user who has a standalone machine and just does stuff like word processing, writing mail and watching movies.
I have one machine with home premium and the only thing I miss on that, at times, is the Group Policy Editor. But since gpedit is just a front-end for the registry, its fine, I can tinker directly via regedit.
I would suggest that if your profile fits the above, stick to the home premium. Save the Ultimate for another computer. However, if you do decide to switch, please image your current install using an app like like Macrium Free before changing, that way you can revert to Home Premium without the bother of activation/ reactivation.