VirusTotal: When is it a false positive, when is it new malware?

Hi everyone. Perhaps this is a stupid question, but I'm rather curious if there is any way to confirm that something is a false positive when it comes to malware scans?

Recently I've become interested in running a rather old program that's being support out-of-cycle by user generated updates. The problem is that the user-created patches seem to set off a few antivirus flags for certain scanning engines. Uploading to VirusTotal gives a result of 3/43 positives. Now, that could be three false positives... but how do I separate that from new malware that other engines simply haven't identified yet?

Additionally worrisome is the fact that the user-generated content comes with instructions to add exceptions for the firewall and antivirus to "avoid problems". I'm not sure if this is a legitimate recommendation to circumvent conflicts, or if it's a clever way to convince people to infect their own systems.

Is there any way I can take a closer look at the suspicious file to see if it's safe or not? Obviously I've already tried VirusTotal, but I don't know where to go from here.

I would be wary of using un-quality-controlled patches that want to add exceptions for themselves in security software, looks like a ticket to hell. Home brewed code can also play havoc with the stability and security of the OS, unless extensively tested and debugged by the community. If this is some hobbyist thing, ask questions on the specific user forums.

If you absolutely need to run this thing, do it in a virtual/sandboxed environment or on a test machine. Otherwise you are much better off using standard software.

Thanks very much for the input. I have tried Jotti's, and got much the same result as VirusTotal. ESET is my primary antivirus, as well as being contained in the VirusTotal results - ESET says the file is clean. I'd hate to bug experts on the forum over something so trivial, but I do admit to being curious as to what someone with more experience than I would say!

Bill, I agree with you completely with what you say. Sadly, there's no alternative for this particular program. It's well past the end of it's life cycle, so alternatives are null. Doing without it would be no crime, though, so if I can't verify it's safety I'll just not use it. Interestingly, I DID try running it in a Sandbox (Sandboxie, to be exact), but it came back with errors about files not being found - files which are clearly in the right locations. On the Sandboxie forums, tuzk said to use the latest beta, but I'm hesitant to install unsable betas.

Asking on the official user forums for this software is something I haven't done, but searching those forums reveals that the main user-developer said "Anyone with a disassembler, x86 assembler experience, and an afternoon could reverse engineer the DLLs added and verify there is nothing remotely malicious in them." Of course, this exceeds my experience, so I can't verify he's telling the truth.