Too much memory on standby.....

[cluberti]

You still haven't told us what this server is used for, how it's working, and why all of the RAM is being consumed in the first place. Honestly, getting a trace with the Windows Performance Toolkit would be in order, but that would take some time (and require .net4 on the server).

 

[j0E]

:sarc: Hi guys, sorry for reviving an old thread, but I've been experimenting the same issue.
I have 8 GB of sniper DDR 3 2100 MHZ (OC to 2200), and standby memory occupies 60% of the memory sometimes and causes sluggish game behavior (dirt3) even though only 3 GB of ram are actually in use especially after long sessions of firefox browsing on the second monitor,
I tried multiple tricks like using firefox restart button, closing unwanted services and applications, memory clean up tricks etc, none of them ware able to restitute a" FRESH" feel and good responsiveness gaming experience EXCEPT puttng windows 7 to sleep ( hibernation) then waking it back :memory in use drops tp 1.7 o 2 GB (even with firefox loaded) and game play is snappy and great.

After more than a year looking for a solution I ended up using the hibernation trick which resets physically memory modules exactly like a fresh system boot: I think I am the first user to report this solution on a forum to the benefit of all users out there that are still looking.

Enjoy

 

[JoomSavvy]

Its not just cached data, its prioritized cached data.

Here is good description: Investigate memory usage with Windows 7 Resource Monitor - TechRepublic

quote from the link:

Standby

The Standby list, which is shown in blue, contains pages that have been removed from process working sets but are still linked to their respective working sets. As such, Standby list is essentially a cache. However, memory pages in the Standby list are prioritized in a range of 0-7, with 7 being the highest. Essentially, a page related to a high-priority process will receive a high-priority level in the Standby list.

For example, processes that are Shareable will be a high priority and pages associated with these Shareable processes will have the highest priority in the Standby list.

Now, if a process needs a page that is associated with the process and that page is now in the Standby list, the memory manager immediately returns the page to that process' working set. However, all pages on the Standby list are available for memory allocation requests from any process. When a process requests additional memory and there is not enough memory in the Free list, the memory manager checks the page's priority and will take a page with a low priority from the Standby list, initialize it, and allocate it to that process.

SO, if an application with a super high priority caches data (such as a VM), your regular applications may not be able to access that 'free memory' after all, and responses to this thread are incorrect.

StandBy will cache high priority data and then fail to release it if the subsequent requesting applications fail to meet the priority test, pushing the application to pagefile.

 

[LMiller7]

Welcome to the forum.

SO, if an application with a super high priority caches data (such as a VM), your regular applications may not be able to access that 'free memory' after all, and responses to this thread are incorrect.

StandBy will cache high priority data and then fail to release it if the subsequent requesting applications fail to meet the priority test, pushing the application to pagefile.

The linked to article does not support this conclusion. The Microsoft publication "Windows Internals" discusses the standby list is much more detail and there is no mention of this "problem".

I haven't seen documentation from any reputable source that would indicate that a process is unable to use memory from a higher priority standby list. It is just that the lower priority standby lists are accessed first. Even if true there is no evidence that it would cause a problem. Windows Task Manager does not show the individual standby lists but Processor Explorer and a number of other utilities do. The normal priority level of most processes is 5. I have never seen a situation where the priority levels 6 and 7 form any more than a small percentage of the total standby list except shortly after bootup when the standby list is quite small anyway.

The bottom line is do not suspect this as being a problem unless you have actual evidence from Process Explorer or similar utility.

 

[theoldwizard1]

... Windows does NOT release Standby Memeory

HP laptop, AMD A10-5745M, 16 GB memory, 500 GB SSD, more than 50% free, 15GB swap file.

This laptop is used to show slide shows that are distributed as .EXEs (application).

This is not my "first rodeo". I have been dealing with various virtual memory operating systems for over 30 years. Various OS's work slightly different or have different names for the same thing, but they all work similarly, except maybe Windows !

Immediately after booting the machine has over 11 GB of Free memory. The slide show runs perfect. Today I was copying a large number of files from a USB2 attach hard drive. I noticed that the Free memory was down to ZERO and Standby Memory was over 13GB ! Other VM systems will "close buffers" as demand for memory increases (note this is a "rate", pages demanded per unit of time). Windows 7 DID NOT !

I started the show while still copying files and the Standby memory DID NOT decrease. The hard fault rate was peaking at 800+ Faults/second, but the slide show played ... for a while. It crashed about 15 minutes into the show with no meaning full error message. There was still effectively ZERO Free memory. I closed the application window.

I walked away because I wanted the file copy to finish. Sometime later, well after the copies finished, there was still ZERO Free memory available. Nothing else was being run on this laptop.

WHY IS WINDOWS NOT RELEASING STANDBY MEMORY !

I rebooted, and have run the slide show end to end (about 30 minutes) several times in a row. Free memory is staying at 11 GB, even though I am running Firefox and writing this post,

 

[LMiller7]

WHY IS WINDOWS NOT RELEASING STANDBY MEMORY !

How do you know that it isn't?

Having zero free memory does not in any way shape or form indicate a problem. This is the normal state. On many occasions I have seen zero or close to zero free memeory and experienced no problems. I have seen many complaints about standby memory supposedly not being released. But how can you really know this is the cause of whatever problem you are having? I think in most cases this is only an assumption and one not based on facts.

 

[theoldwizard1]

The problem I am having (if you read my post) is an application (slide show) crashing, when there is (effectively) ZERO Free memory.

It does not crash when there is sufficient Free memory available.

Why would Windows have >10GB of Standby memory when there is nothing running except "normal" Windows back ground tasks.

If you this "normal" it is very different from every other VM system I have dealt with and IMHO is totally illogical !

 

[LMiller7]

The normal state of available memory is on the standby list. Free memory is the aberation, which unfortunately is often unavoidable in a modern OS. The ideal would be zero free memory at all times but we are not there yet. The NT platform has worked this way from the very beginning, although this wasn't readily apparent from Task Manager. To me this is totally logical.

 

[theoldwizard1]

The normal state of available memory is on the standby list. Free memory is the aberation, which unfortunately is often unavoidable in a modern OS. The ideal would be zero free memory at all times but we are not there yet.

Okay, I understand where you are coming from. Free memory is wasting a resource, if there is something better that it could be used for (I/O buffers/cache).

Then the issue is Windows NOT RELEASING STANDBY MEMORY QUICK ENOUGH when other application need it. Other OS's I have dealt with, have a "minimum" amount of Free memory >ZERO and will start releasing buffers/cache to stay above that minimum. If the Free memory minimum is still not met, non-running images are swapped out.


Now you could make the argument that the application may be poorly designed such that it crashes instead of just pausing, but it seems ridiculous to run into the situation I am having on a machine with 16 GB of memory.

 

[logicearth]

I can tell you right now, theoldwizard1. It is not the standby memory that is causing your issue. Right now you are just grasping at straws. Right now unless we know exactly what you are doing, it is impossible to help you.

 

[theoldwizard1]

Right now unless we know exactly what you are doing, it is impossible to help you.

Please re-read my posts. I do not know how to be more clear.

Running a slide show application (.EXE) crashes after about 15 minutes when there is ZERO Free memory.

Running the same slide show application, runs to completion, multiple time, when there is a "lot" of Free memory available.

 

[logicearth]

Then it is the application that is broken. Not Windows or its Standby Memory. The application must be doing something that is causing it to crash. Since I don't know want application it is, no idea what logs it makes.

And it certainly is not because Windows is not releasing Standby memory quick enough. That is not an issue, Windows doesn't release Standby it just over writes it. Clearly something is going wrong with your application.

 

[cluberti]

Ultimately, logicearth is correct - standby memory is effectively free memory if there's demand, as the contents of a "standby" page are pages that are not in a process' working set (actively touched memory), and that do not need a backing store to be removed from the working set either (memory that can go directly to the standby list can be temporary data, binaries from disk, freed memory, etc.). When a process makes an alloc request (not just a reservation), the memory manager initially looks to the zero list to fulfill the request (pages that have been freed and then completely zeroed out). If that list is empty, the memory manager then looks to the free list - if that list is empty, then standby pages are invalidated and used as free pages.

Standby pages are pages that have been allocated by a process (or multiple processes), but they did not contain "private" data before the memory manager deemed them to be no longer in use and yanked them from a process' working set. Standby pages are pages that do not need to be backed before they're cleared from RAM, because they're temporary, freed, or part of the "shared" portion of the working set, and as such do not need to be saved (they're either still in memory as part of the shared portion of another process' working set, or the memory is indeed no longer in use by any processes).

If the page was deemed able to be removed from the working set of a process, but was previously listed as part of the private portion of the working set (private working set memory is non-shared, as in it's data contents are unique to the process that used it), it's contents could be marked as not in use and pulled from a working set - but that page requires a backing store before being cleared. These pages are on the "modified" page list - they're still available to be used by other processes when the system is under memory pressure, but the contents of the pages would have to be put into a backing store (the paging file), then the page can be moved to the standby list where it could then be used as previously described (the modified page writer in the kernel is responsible for backing modified pages and then putting them on the standby list).

So, if you have a large standby list, but are having issues with performance or applications misbehaving, then it's not likely the OS that's at fault. We'd need to know more about *how* the process was requesting memory, and what types of requests for memory it was making (is it making a request for a reservation? Is it actually making an alloc/malloc call to acquire RAM? Is it actually trying to write data to those pages?). Something's not right here, and given that the only way an application can actually talk to physical RAM is to use AWE (32bit only) or use a request to lock pages via a driver (which will fail if there's not enough memory for the request, btw), we'd need to know more to understand why the application in question has issues when Windows starts getting low on free or zero pages. However, again, since by default a process has *zero* understanding of Physical RAM in a box, and can only work with virtual address space (VA), then we'd need to know more. However, if you have adequate standby memory, with little to no "free" or "zero" memory, then it's either a filter driver (not likely, but possible) that's preventing the memory manager from being able to access memory, or the application is trying to directly access RAM versus going through the regular memory manager APIs. In either case, the memory manager *should* be clearing the standby list if free/zero are empty. If it's not, then either there's something installed on the system that's preventing it, or the app itself is trying to do memory management (and failing).

Since we don't know why, we can only state that we don't know, but this is how it *should* work. If it isn't working that way, then we need to know more about how the app (or apps, in this case) are designed to request and allocate memory.

 

[theoldwizard1]

... When a process makes an alloc request (not just a reservation), the memory manager initially looks to the zero list to fulfill the request (pages that have been freed and then completely zeroed out). If that list is empty, the memory manager then looks to the free list - if that list is empty, then standby pages are invalidated and used as free pages.

Standby pages are pages that have been allocated by a process (or multiple processes), but they did not contain "private" data before the memory manager deemed them to be no longer in use and yanked them from a process' working set.

Well stated and I concur !

So, if you have a large standby list, but are having issues with performance or applications misbehaving, then it's not likely the OS that's at fault. We'd need to know more about *how* the process was requesting memory, and what types of requests for memory it was making ...
.
.
.
Since we don't know why, we can only state that we don't know, but this is how it *should* work. If it isn't working that way, then we need to know more about how the app (or apps, in this case) are designed to request and allocate memory.

We will never know just exactly what the application is doing !

I can only report what I have observed. Large Free list application runs to completion. Large Standby list no Free list, application crashes, but not immediately. I observed no changes in the size of the Standby or Free list while the application was running.

Even odder (from my perspective) is that well after the application crashed, and other tasks completed (that caused the Standby list to grow), there was still a large Standby list and ZERO Free list.


IMHO, an application programmer should not have to worry about how an OS manages memory. They should be confident that it will always "do the right thing" and provide them the virtual memory space needed to run. Or stated another way, a poor algorithm should cause an application to run poorly, not crash.

 

[cluberti]

The problem begins if the application developer starts to stray from the beaten path. If this was a poor algorithm, everyone in the world would be complaining of it. The fact that it happens so infrequently (unfortunately for you it's happening to you) would indicate a problem with the app code, and not the OS. It's *possible* that you're hitting a bug, but it's not very probable. The OS isn't going to magically move pages from standby to the free list unless the zero page list is also empty. Standby pages could conceivably still be requested by the process that initially requested them, so standby pages will NOT go from standby to ANY list unless there are applications making alloc requests for memory (again, not just reservations, but allocs and then make an attempt to write to that requested page). In your case, the fact that you can start applications that should very obviously be requesting memory (the simple act of starting a process would do this, for things like default heap, loading included modules, etc) but seemingly are not would tend to indicate an app programming issue, and not an OS issue. Again, there *are* ways to do your own memory management without using the inbuilt APIs to request memory (especially if the application is written in C or ASM). I'm not saying that's what's happening, but.... that's what appears to be happening.

 

[theoldwizard1]

35+ years administering medium and large VM systems. I stand by my position.

An application programmer should not have to worry about how an OS manages memory. They should be confident that it will always "do the right thing" and provide them the virtual memory space needed to run. Or stated another way, a poor algorithm should cause an application to run poorly, not crash.

 

[LMiller7]

IMHO, an application programmer should not have to worry about how an OS manages memory. They should be confident that it will always "do the right thing" and provide them the virtual memory space needed to run. Or stated another way, a poor algorithm should cause an application to run poorly, not crash.

Very true, provided some conditions are met.

1. The application must follow the rules as established by the OS. When an application breaks the rules it must suffer the consequences, including crashing.

2. The application must not ask for resources the system cannot provide. As an example, when allocating memory (virtual address space) it must not exceed the virtual address space or the commit limit.

I am not including RAM as a resource. Unless the application is using something like AWE it need have no knowledge of how much RAM is in the system, how much is available, etc. This is managed by the system without application involvement.

Unfortunately, sometimes application do violate riles #1 and #2 and the OS cannot always protect them from the consequences. The OS does have responsibility to see that application mistakes do not result in a system crash.

 

[Layback Bear]

I don't create programs or operating systems but to my understanding it is the program creators job to make their program work with the operating system they intend to use it with.
Microsoft is not going to change Windows 7 to suite a particular program nor should they.

Thousands if not millions of programs are made to work with Windows 7 so as just being a dummy like me I have to believe it can be done if the rules for making a program to be used on Windows 7 are followed.

 

[theoldwizard1]

Very true, provided some conditions are met.

I disagree with both

1. The application must follow the rules as established by the OS. When an application breaks the rules it must suffer the consequences, including crashing.

The application must follow the rules of THE COMPUTER LANGUAGE IT IS WRITTEN IN, which provides the run-time environment! The OS should have very little, if any affect on the application, unless the application is using OS-specific procedures.

If your statement was true, then what good would would application written in Java, Ruby or Python be ?

2. The application must not ask for resources the system cannot provide. As an example, when allocating memory (virtual address space) it must not exceed the virtual address space or the commit limit.

Only partially true. The application can ASK for any resource it wants. It is up to the OS to decide if the request and be fulfilled and in NOT somehow tell the application, "Hey, I can't do that !"



Sadly most programmers out there to day have grown up in either the Windows or Un*x environment typically using the C programming language. IMHO, these do NOT lend themselves to educating the programmer to check the status of errors. Even if they do the error code usually lack the "robustness" to allow the application to handle the error in an "intelligent" manner.

Sadly, many concept well defined and taught by now "extinct" OS also died with those OSs. Why is it next to impossible to open a file for exclusive access in Un*x without some application specific overhead ? Many other OSs handled this elegantly. The list goes on and on.



Opinions are like rectums ! Everybody has one !!

 

[LMiller7]

I stand by my statements 100%.

When a language like Java or Python is involved it takes over some of the responsibilities from the application. But to the OS it is all one and the same. The OS has no knowledge and doesn't care whether the application is written in C, Python, Ruby, or assembly language. The application, language combination must obey the rules established by the OS.

When a request is made of the OS (whether it comes form the application or the language makes no difference) that it cannot fulfill it has no reasonable alternative than to refuse the request. How an application or language handles this problem is up to it. Some handle this problem gracefully, others do not.

I have been programming a variety of systems since the 1970's so have some knowledge of these matters. I expect that some statements on both sides have been misunderstood. I do not wish this to become an argument so I probably won't be making any further replies to this thread.