Graphics card causing boot problem

Hello Everyone.
Well, coming to the issue, my pc doesn`t boots up in the first try. It gets stuck after the "Starting windows" logo with a black screen. However, when i press the restart button, it boots up correctly. Okay, i think that`s may be confusing, so, here`s my problem explained in steps-
1. I pushed the start button on the cpu case.
2. Things go nicely until the "Starting windows" screen.
3. After this, just a black screen (not the black screen of death, it just stops booting with a black screen and the red led on the case stops blinking)
4. I pressed the restart button. It booted nicely.
5. The problem doesn`t appear as long as i`m only restarting or logging on/off but as soon as i press shut down or sleep option, the same problem happens.
I`ve a XFX HD Radeon 4350 1GB card. I tried re-connecting the vga cable to the card`s slot to no avail. GPU-Z reads its location at PCIe which means its in the PCIe slot. I tried changing the bios settings, "Init display first" to PEG and "Onboard VGA" to Enable if no exist PEG" thus disabling the onboard VGA but it too doesn`t help. Oh yeah, i forgot to tell that the problem doesn`t appear when my monitor is hooked to the onboard graphics card.
One thing i know for sure now is that the issue has something to do with the add-in card. But i`m out of ideas now. Tried upgrading and downgrading it`s drivers to no avail. Played with bios sett., didn`t help. Its so annoying that I`ve to always wait for the black screen just to press restart button and again wait for it to boot . Thanks in advance.
My specs: Intel core 2 duo E7400 @ 2.8GHz
Motherboard- Gigabyte G31M-ES2L
BIOS- Latest version
3GB Memory

It has become 2 months old today. And considering the fact that i don't mess with my cpu casing and neither its overclocked, i don't think the card is bad.
About the power supply, i don't know. I just have an UPS for powering the machine. How do i find the wattage supply?
And yeah, thanks for replying atleast.

@liquidsnak
If a bad card's the case, then i'm fortunate that the card is under warranty.
Regarding the make, it's a HCL EzeeBee pc. On the CPU's carton, its written as "4097AZ155814 EZEEBEE TOP Z321 C2D"
And since i cant open the case, is there any way (s/w) to determine the PSU? Do i provide the details of the UPS? Will that be of any help?

thefabe said:

I just have an UPS for powering the machine

My suggestion is to try using a powersrtip and plug your system diirectly into your household current and see if the problem persists.Just wondering if it takes the UPS a bit to provide enough power to power it on correctly.Fabe

will try that too... btw, on the back of UPS, its written:
input-220V
output-220V
power-600VA (just if it could be of any help)
This problem emerged just few weeks ago. So, i don't think there's something wrong with the UPS.

Plugged the cables directly into the household sockets.. Didn't help.