32bit 4Gb Memory Limit Explained

Geoff Chappell · 12 Dec 2009

tcsenter said:

There was plenty coming from Microsoft (through developer channels) about this issue ...

There maybe was about what you want to count as "this issue", but where my paragraph that you quote says there is "strikingly little" from Microsoft, it is not talking of as wide an issue as you want, it is talking of "my article's claim" as described in the paragraph before your quote.

For the given context, then, your citations are irrelevant except for three that I do anyway cite in my article. Indeed, one of them is precisely what I meant by "some of what little it does say is absurd". Now, you may think I misunderstand or even misappropriate the articles that I cite, but since I do cite them, it's not enough that you just re-cite them.

As for the others, I'd have cited them too, had I depended on them, where I appraise how and what software must be recoded for PAE and the use of memory above 4GB. Instead, my only reason to cite them would be to say that Microsoft agrees with me.

That Microsoft presents your "well-documented driver issues in consumer space" as reason to limit consumer editions to 4GB certainly is repeated in the article, from one of the cited articles, but with the caveats that the stated reason does not itself explain the outcome and that the reasoning is not open to independent testing. If your citations cover both those points, then please say where.

By the way, for as much as you may say these citations show Microsoft was clear that consumer editions would be limited, one of them looks very much like a record that the reduction of memory for Windows XP SP2 was not very well publicised in advance.

If you don't like Microsoft's explanation why, maybe you'll better prefer Linus Torvald's explanation why Microsoft would do this....

The world may owe Linus a lot, but quoting that is no favour to him. I can only assume he's talking about other things or other contexts. He's at best out of topic when he wants to use normal pointers for physical memory addresses and thinks that memory above 4GB is limited to "special uses only". That said, you and I clearly have very different ideas about what my topic was when I asked my critic to explain himself.

mushroomboy · 12 Dec 2009

whs said:

An interesting article indeed. The way I read it is that 32bit and 64bit OS are essentially the same code and both are present when you buy 32bit. With some hacks and tweaks you can disable the 32bit blockage and make it run as 64bit. That's like taking the muffler out of the exhaust pipe of a mofa in order to make it run faster - and that is not legal either.

And that is VERY wrong. They are the same code, but the physical address space of 64bit can't ever be done by a 32bit cpu. You can't run 64bit code on a 32bit cpu without emulation of some kind. It'll have to chop up the data in parts that the 32bit cpu can run. It's like putting a square in a round hole. =(

With the others who talk about PAE it is an ugly hack, cpu's weren't ment to run like that. =S If we are going to use a car analogy it's like adding NOS to a 89 Mustang and saying "Oh that's how it's supposed to be run". Even though if you screw up on something simple there goes your Mustang! lol Yes we can do it, but that wasn't the intent when they started the NT kernel. So no they didn't lie when they said it wasn't supported, because a vanilla kernel wouldn't support it. =S

[edit] When I say the same code, I mean the C/C++ programming code they use to write with. Or whatever code they choose... But once it's compiled to machine, it's no longer the same code, 64bit has larger addressing than 32, that's just what happens after compile.

pallesenw · 12 Dec 2009

mushroomboy said:

but that wasn't the intent when they started the NT kernel. So no they didn't lie when they said it wasn't supported

That is the whole point. It was supported from the very beginning. In XP/SP2 they put an extra limitation into the memory manager, so all access to the address space above 4G was cut off.

mushroomboy · 12 Dec 2009

pallesenw said:

mushroomboy said:

but that wasn't the intent when they started the NT kernel. So no they didn't lie when they said it wasn't supported

That is the whole point. It was supported from the very beginning. In XP/SP2 they put an extra limitation into the memory manager, so all access to the address space above 4G was cut off.

But you forget the NT kernel was started in 95! I'm not talking about xp/sp2 did you not see the part when I stated a vanilla kernel? Don't quote part of a post just so you can word what you like with it.

pallesenw · 12 Dec 2009

What kind of relevancy does NT 4 and earlier have to do with this discussion?

PAE it is an ugly hack

I just love to see those quotes.

mushroomboy · 12 Dec 2009

pallesenw said:

What kind of relevancy does NT 4 and earlier have to do with this discussion?

PAE it is an ugly hack

I just love to see those quotes.

Because all the drivers are still based off the original memory management. Why would you re-write the driver code for something when you can just re-compile it for a new system and fix those bugs? So a lot of legacy code was in both the NT kernel and the drivers, and what does that mean? Buggy PAE, and what does that imply? Oh whatever, figure out marketing and how people write code for **** and then read this post over again. It'll give you a new light, because obviously people are supposed to write all new code for hardware support instead of re-use the old code with tweaks. Or am I just the stupid one and good marketing eludes me?

pallesenw · 12 Dec 2009

Oh, I wasn't aware that people still used drivers from the NT 4 generation.

mushroomboy · 12 Dec 2009

pallesenw said:

Oh, I wasn't aware that people still used drivers from the NT 4 generation.

No but the code is still used in XP... This is exactly why they started Vista, to re-build the kernel and driverspace. Or have you been paying attention to anything?

Windows NT - Wikipedia, the free encyclopedia

Code:

[edit] Driver models

 Windows NT introduced its own driver model, the Windows NT driver  model, and is incompatible with older driver frameworks. With Windows  2000, the Windows NT driver model was enhanced to become the Windows Driver Model, which was first introduced with Windows  98, but was based on the NT driver model.[10]  Windows Vista added native support for the Windows Driver Foundation, which  is also available for Windows XP, Windows Server 2003 and to an extent, Windows  2000.

They have many reasons for doing this, but we still see legacy code used in drivers. Daniel K's creative pack (hack I dare say?) gets ****ed up with larger memory, why do you suppose that is? Infact since it's still Creative's driver why do you suppose proprietary drivers should get messed up with larg ram? They still use legacy code, it doesn't mean Windows is using the legacy code, but if you add support for drivers with legacy code then your still loading legacy code into the kernel space. It's JUST like writing the legacy code, and then you get BSOD's.... GASP NO! People will use older code whenever possible, why would you re-write something that works, it's been going on for a long long long time.

Geoff Chappell · 14 Dec 2009

mushroomboy said:

And that is VERY wrong.

Yes, what you quote is a very wrong reading of the article.

But you forget the NT kernel was started in 95!

But even then it was made plain that drivers should account for 64-bit physical memory addresses and should use map registers when setting up DMA transfers. Although support for the latter was significantly revised for Windows 2000, this was mostly to allow that bus drivers could provide DMA support independently of the HAL. It doesn't change that the NT kernel always anticipated that physical memory might extend beyond 4GB. An NT 4.0 driver that was _correctly_ written won't have been tested for memory above 4GB on an x86 platform until Microsoft's introduction of PAE support, but it ought to work unchanged in later versions that actually can use physical memory above 4GB.

That some or even many drivers for the x86 platform were not written correctly but were instead written with shortcuts that happened to work with the x86 kernel as then implemented was a significant problem of backwards compatibility. But to say that memory above 4GB was not "the intent when they started the NT kernel" is quite a stretch.

So no they didn't lie when they said it wasn't supported, because a vanilla kernel wouldn't support it.

Perhaps it's just that I love (true) vanilla as a flavour, but I have to ask what do you mean by "vanilla kernel" for this point? You can't mean that accommodating PAE as an optional extra can't be done in the one kernel. That there's a separate kernel for PAE is just for better performance: it's an optimisation, not a necessity.

Because all the drivers are still based off the original memory management. Why would you re-write the driver code for something when you can just re-compile it for a new system and fix those bugs?

I imagine you would fix the bugs in the code and then re-compile the code for both systems. If those drivers "based off the original memory management" do not accommodate physical memory above 4GB on an x86 machine, then just recompiling them for an x64 machine is only rarely going to get you a driver that accommodates physical memory above 4GB on an x64 machine.

Remember that PAE does not require any particular coding in any driver. Use of memory above 4GB does. Whether the operating system supports such memory through PAE or through being natively 64-bit is essentially immaterial.

So a lot of legacy code was in both the NT kernel and the drivers, and what does that mean? Buggy PAE, and what does that imply?

I have no idea what relevant code you can dismiss as legacy code in the NT kernel, but let's leave that aside. You ask what's implied by "buggy PAE": not the treatment that Microsoft implemented.

Microsoft's ordinary solution to any feature looking buggy because of bad or old software, whether from Microsoft itself or from third parties, is a configurable setting. It could be something you enable if you have bad or old software, or you disable if you know you don't have bad or old software. Examples exist of both. A configurable setting to limit memory to the first 4GB, and thus prevent any effects from "buggy PAE", has long existed. Its use would have been the easiest solution consistent with Microsoft's explanations of limiting memory in Windows Vista.

I had perhaps better make clear that I'm not advocating PAE. Had there been no PAE code in Windows Vista, I would have nothing to say about it, I expect.

mushroomboy · 14 Dec 2009

Ok what I'm saying is, when XP was started and PAE was implamented they still had a lot of drivers using the 98 code, which didn't have the PAE implamentation. You do know that 90% of code is recycled if it still works. So if the code doesn't implament PAE they usually won't add it to current drivers unless it's new hardware. So that's what happens. And since the original XP code didn't even have PAE (SP2 implamentation) than you don't get it for a lot of the written drivers. 'Vanilla kernel' means original unmodified kernel, it's huge in unix and linux terms as The Linux Kernel Archives releases vanilla kernels. MY bad for being heavy into dev for linux and using those terms that aren't in windows. So next time you say I quoted wrong I didn't, I'm referring to the original plans for XP and Vista..... They re-wrote the driver code for Vista BECAUSE of said problems with XP drivers. Legacy code is ****ty and very poorly documented.

I have a question... Do you code C++? Do you code anything? Do you even know how code development stages go? Do you even know the terms for "vanilla kernel" or the different kernel run levels? Or that really drivers are a part of the kernel, just a module that's loadable/unloadable? So STFU if you are going to say PAE isn't buggy. It's an addition that wasn't planned very well and with 64bit so close it was never even developed very well for the public. So don't start this I quoted **** wrong, 64bit's been around for AGES and they knew this, it just wasn't as publically available. So why would a company even bother with PAE in it's drivers when 64bit exists? That's what most people would say if they made a driver, why? Because it's easier, safer, and is the natural course for 32bit. 32bit WAS NOT DESIGNED FOR MORE THAN 4G!!! PAE is more or less an emulated table for anything larger than 4G, so in all honesty even saying MS lied is still really a lie. Why? Because MS stated that 32bit has a physical limit of 3.5G, which in all honesty is TRUE. If you don't want to believe me then fine, but don't go saying I'm spouting bull****, because if you ran linux you don't even get PAE. A pure32 system cannot support over 3.5G, that's just it. Unix, Cisco, Linux, MAC all have the same limit, why? Because they didn't bother with PAE because they all have 64bit variants naturally, and have been running 64bit systems much smoother than MS, as far as 64bit support.

pallesenw said:

Oh, I wasn't aware that people still used drivers from the NT 4 generation.

Oh and as for this, you'd be suprised how much code has it's base in Win 3.1... Why? Because as I said earlier a lot of code is recycled. Infact, most code is already written, why would you bother to make something new when you can just use somebody else's work? With so much GPL code out there it's just too much work to make your own. Infact I'm going to guess a lot of Vista/7 code is XP based, just with the new memory management. Why? Why not!