Win 7 64 RAM filling up. Memory leak outside windows?

**cluberti** · 13 Dec 2010

Preliminary update, but it looks like (so far) that the AVG filter driver has put a ton of disk IRPs into a completion pending state, meaning your disk queue is backing up (thus being a likely culprit for a hang scenario). I still need a bit of time to go through all of these stuck IRPs and make sure it's not just a victim of something else backing up the queue, but so far, it looks like AVG as the cause of your "massive not responding"

. I'll post back with more in an hour or three - if you catch this before I update, it might be worth completely uninstalling AVG and retesting, as that is likely to restore some sanity to your system (this is one of the many reasons that most of us here do not recommend AVG antivirus software).

**cluberti** · 14 Dec 2010

So, ultimately, yes, it appears that your disk queue was growing. You have a few hundred threads on the system that look like this, waiting on NTFS flush routines to write data to the disk:

Code:

// This thread is doing non-cached I/O, meaning this *has* to be written to disk, not to the cache:
3: kd> !thread fffffa8006d27680 
THREAD fffffa8006d27680 Cid 0004.0024 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
fffff88003393258 NotificationEvent
IRP List:
fffffa800799a010: (0006,0550) Flags: 00060043 Mdl: fffffa800751a010
Not impersonating
DeviceMap fffff8a000008b30
Owning Process fffffa8006ca8b30 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 81474 Ticks: 4 (0:00:00:00.062)
Context Switch Count 7193 IdealProcessor: 0 
UserTime 00:00:00.000
KernelTime 00:00:00.280
Win32 Start Address nt!ExpWorkerThread (0xfffff80002cde850)
Stack Init fffff88003393db0 Current fffff88003392c80
Base fffff88003394000 Limit fffff8800338e000 Call 0
Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
fffff880`03392cc0 fffff800`02cd8992 : 00000000`0003aa00 fffffa80`06d27680 00000000`00000000 00000000`00000000 : nt!KiSwapContext+0x7a
fffff880`03392e00 fffff800`02cdacff : fffff880`033932f0 fffffa80`07d10950 00000000`00000000 fffff880`03393030 : nt!KiCommitThreadWait+0x1d2
fffff880`03392e90 fffff880`0123605f : fffffa80`07da9b00 00000000`00000000 fffff8a0`0bb2c100 fffffa80`07f5e100 : nt!KeWaitForSingleObject+0x19f
fffff880`03392f30 fffff880`0123ca12 : fffff880`033932f0 fffffa80`0799a010 fffffa80`07f5e180 fffff8a0`0bb2c140 : Ntfs!NtfsNonCachedIo+0x23f
fffff880`03393100 fffff880`01241413 : fffff880`033932f0 fffffa80`0799a010 fffff880`03393401 fffff880`03393401 : Ntfs!NtfsCommonWrite+0x872
fffff880`033932c0 fffff880`010e123f : fffffa80`0799a518 fffffa80`0799a010 fffffa80`079e7d30 00000000`00000000 : Ntfs!NtfsFsdWrite+0x1c3
fffff880`03393540 fffff880`010df6df : fffffa80`07d838c0 fffffa80`0b8a3dd0 fffffa80`07d83800 fffffa80`0799a010 : fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x24f
fffff880`033935d0 fffff880`03bd2803 : fffffa80`0799a010 fffff800`02cbdac2 f8a00514`7fd00400 f8a00514`7fd80400 : fltmgr!FltpDispatch+0xcf
fffff880`03393630 fffff800`02cc02ff : fffffa80`0a3af060 fffffa80`0799a010 fffffa80`0751a010 fffff800`02e4be80 : AVGIDSFilter+0x5803
fffff880`033936d0 fffff800`02cbf987 : 00000000`00000f00 fffff880`03393c58 fffffa80`035310f0 00000000`00000000 : nt!IoSynchronousPageWrite+0x24f
fffff880`03393750 fffff800`02cbd804 : fffff8a0`0cfbd7a0 fffff8a0`0cfbd9d8 fffffa80`071d5f90 fffffa80`071d5f90 : nt!MiFlushSectionInternal+0xa58
fffff880`03393990 fffff800`02cbd259 : 00000000`00013e48 00000000`00000000 00000000`00047000 00000000`00000000 : nt!MmFlushSection+0xa4
fffff880`03393a50 fffff800`02cc494b : fffffa80`079420c8 fffff800`00000001 00000000`00000001 fffff800`00047000 : nt!CcFlushCache+0x5e9
fffff880`03393b50 fffff800`02cc5520 : 00000000`00000000 fffff880`03393c58 00000000`00000000 00000000`00000000 : nt!CcWriteBehind+0x1eb
fffff880`03393c00 fffff800`02cde961 : fffffa80`06d21ab0 fffff880`0122d270 fffff800`02ed8140 fffff800`00000000 : nt!CcWorkerThread+0x1c8
fffff880`03393cb0 fffff800`02f75c06 : 00000000`00000000 fffffa80`06d27680 00000000`00000080 fffffa80`06ca8b30 : nt!ExpWorkerThread+0x111
fffff880`03393d40 fffff800`02cafc26 : fffff880`03164180 fffffa80`06d27680 fffff880`0316efc0 00000000`00000000 : nt!PspSystemThreadStartup+0x5a
fffff880`03393d80 00000000`00000000 : fffff880`03394000 fffff880`0338e000 fffff880`033939f0 00000000`00000000 : nt!KxStartSystemThread+0x16
 
// The IRP associated with this thread, showing pending an NTFS completion routine:
3: kd> !irp fffffa800799a010
Irp is active with 16 stacks 15 is current (= 0xfffffa800799a4d0)
Mdl=fffffa800751a010: Irp count=00000002: Thread fffffa8006d27680: Irp stack trace. 
cmd flg cl Device File Completion-Context
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000 
Args: 00000000 00000000 00000000 00000000
>[ 4, 0] 0 e0 fffffa8007f5e030 fffffa800b8a3dd0 fffff880012384f4-fffff88003393250 Success Error Cancel 
\FileSystem\Ntfs Ntfs!NtfsMasterIrpSyncCompletionRoutine
Args: 00047000 00000000 00ff4000 00000000
[ 4, 0] 0 0 fffffa8007f5e030 fffffa800b8a3dd0 00000000-00000000 
\FileSystem\Ntfs
Args: 00047000 00000000 00ff4000 00000000
 
// Waiting on a write of a file stream to the volmgr driver:
3: kd> !fileobj fffffa800b8a3dd0
Device Object: 0xfffffa8007ce0cd0 \Driver\volmgr
Vpb: 0xfffffa8007cdf9a0
Access: Read Write SharedRead SharedWrite SharedDelete 
Flags: 0x40100
Stream File
Handle Created
FsContext: 0xfffff8a00bb2c140 FsContext2: 0x00000000
CurrentByteOffset: 0
Cache Data:
Section Object Pointers: fffffa80079420c8
Shared Cache Map: fffffa800798e910 File Offset: 0 in VACB number 0
Data at offset 0 not mapped

All of the above threads on the system in that state are waiting on this IRP in the AVG filter driver:

Code:

// For some reason, the AVG TDI (network) driver is pending, holding up AFD, holding up volmgr, holding up NTFS:
3: kd> !irp fffffa80`06d5d300
Irp is active with 1 stacks 1 is current (= 0xfffffa8006d5d3d0)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
>[  f, 6]   0 e0 fffffa80080ce9f0 fffffa8007441f20 fffff88002f6c450-fffffa8006ec18b0 Success Error Cancel 
        \Driver\Avgtdia afd!AfdRestartAbort
   Args: 00000002 00000000 00000000 00000000
 
3: kd> !drvobj fffffa80`080d2060
Driver object (fffffa80080d2060) is for:
 \Driver\Avgtdia
Driver Extension List: (id , addr)
Device Object list:
fffffa80080f0040  fffffa80080f3b00  fffffa80080f3d10  fffffa80080ce9f0
fffffa80080cec00  fffffa80080d5040  fffffa80080f09f0  
 
3: kd> lmvm avgtdia
start             end                 module name
fffff880`02e62000 fffff880`02ec3000   avgtdia    (no symbols)           
    Loaded symbol image file: avgtdia.sys
    Image path: \SystemRoot\system32\DRIVERS\avgtdia.sys
    Image name: avgtdia.sys
    Timestamp:        Tue Nov 09 15:13:33 2010 (4CD9AB6D)
    CheckSum:         0006643C
    ImageSize:        00061000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

At this point, I'd remove AVG and see what happens - I'm not sure I'd replace it with anything just yet (you're still testing), but long term (post-testing) using MSE instead would be a good choice.

jemcgarvey · 15 Dec 2010

I had switched to MSE before and was under the impression that it had caused a memory problem, but I could just be mistaken in my memory. I have been watching I/Os and noticed that AVG keeps adding processes that send incessant I/O requests. I'll scrub it off and see what happens.

jemcgarvey · 17 Dec 2010

Well thanks cluberti, it seems that removing AVG has lightened my load quite a bit, as far as hang ups and "not responding". It's only been a few days but it's only hanging when the network is bogged down with other things.

This doesn't really have to do with the memory leak, but thanks nonetheless. I haven't had the memory problem since the increase in ram, and though it doesn't exonerate the memory controller in the least (usage is still high), it lets me breathe.

Anyway, if it manages to constrict me again, I'll be right back!

Cheers.

**cluberti** · 20 Dec 2010

If you want to continue tshooting the memory usage, that can be done. Let us know when you decide to go further with it. It will take some tools and debugging over time, of course.

jemcgarvey · 21 Dec 2010

Well I'm a student, so I don't have time when I want it, but I can work through it at a "chess by mail" pace.

I mentioned I haven't had any problems, but I notice this:

As I'm running chkdsk on the second 2TB in a row (one at a time). I know, I'm running two defrags as well, but the ram usage is the question, and that's definitely almost 7GB in use by one chkdsk routine.

**cluberti** · 21 Dec 2010

Yeah, that's a ton of usage by the explorer process (and it is committed, not just reserved). It'd be interesting to see a trace of explorer.exe at the time it looks like that (dumps over about a 30 minute period), although using DebugDiag on Win7 x64 is still hit or miss. If you could download/install it, create a memory/leak rule for explorer.exe, and take full dumps of that over about 30 minutes, that would be the best way to start.

logicearth · 21 Dec 2010

jemcgarvey said:

...As I'm running chkdsk on the second 2TB in a row (one at a time). I know, I'm running two defrags as well, but the ram usage is the question, and that's definitely almost 7GB in use by one chkdsk routine.

That is by design. There was a big whole stink about it before Win7 was released. But chkdsk will and does take as much memory as it possibly can. It does this to repair a disk much faster. It only does this when you use the "/r" switch (Repair) otherwise it doesn't utilize memory like that.

Not a memory leak for chkdsk.

jemcgarvey · 22 Dec 2010

Yeah in this instance I didn't run into any instability, and taskmanager is honest about how much ram is going to explorer, but I was a little surprised to see it, as I've never even had more than 4GB before.

chrysalis · 22 Dec 2010

baarod said:

What are you all basing these "leaks" on? What memory metric provided by what authority? Just because "Free" memory in Task Manager goes down does NOT mean you have a leak.

Here is how I define a leak.

eg. my IE8 leaks.

If my ram usage is higher then expected and I feel like diagnosing I will shutdown apps one at a time and wait for the big jump downwards. Often IE8 is leaking badly.

eg. task manager will report iexplore.exe using say 250meg ram, I shut it down and suddenly I have freed up a gig of ram.

The high ram usage also could be cache, disabling superfetch may help on that.