32bit 4Gb Memory Limit Explained

pallesenw said:

Ok what I'm saying is, when XP was started and PAE was implamented they still had a lot of drivers using the 98 code, which didn't have the PAE implamentation.

[1]What is being said is, you don't implement anything. PAE is something between the OS doing memory management and the CPU.

And since the original XP code didn't even have PAE (SP2 implamentation)

But it was implemented. It was implemented back in the Windows 2000 days. Before XP/SP2, you also had support for memory above 4G.

32bit WAS NOT DESIGNED FOR MORE THAN 4G!!! PAE is more or less an emulated table for anything larger than 4G ... A pure32 system cannot support over 3.5G, that's just it.

[2]I don't think you understand PAE that well.

Oh and as for this, you'd be suprised how much code has it's base in Win 3.1... Why? Because as I said earlier a lot of code is recycled. Infact, most code is already written, why would you bother to make something new when you can just use somebody else's work? With so much GPL code out there it's just too much work to make your own. Infact I'm going to guess a lot of Vista/7 code is XP based, just with the new memory management. Why? Why not!

[3]You cannot expect to take code written for an ancient OS and just recompile to make it run on your new OS.

1st point) PAE and memory management is also part of the kernel, and since the drivers and kernel share memory it is completely relavant.

Writing device drivers in Linux: A brief tutorial

And if you think "well that's not windows" it isn't your right, but windows does do drivers in kernel space. That's why 64bit doesn't allow unsigned drivers, as they could disrupt kernel code and be a huge security risk. And if they are so closely intertwined then I'm guessing they probably need to be very close in memory management.... Which is why PAE is known for BSOD on drivers that weren't written for it. And if you read my 3rd point, MANY companies are scared drivers won't work on a new os....

2nd point) PAE was in the 2000 days? It seems as it only hit 2k pro, and then xp was released and it was only supported in xp sp2 as from the MS site itself.

3rd point) In a 32-bit operating system you'll never be able to address more than 4GB directly. A different addressing scheme may allow you to map more physical memory into that 4GB address space. That's what PAE does. (if this helps you think about how it works). You just cannot address it directly. Or here better yet

Code:

Memory Manager

                    The memory manager translates virtual memory addresses used  by
 the operating system and applications to actual physical memory  locations. The 
translation of virtual memory to physical memory is  transparent to the application. 
User mode processes are never able to  directly write to real memory and never 
actually know where their data  resides. A user mode process can request a block 
of memory and write to  it. The data written to the memory location might be 
written to real  memory, or might be written to a paging file. A paging file (also 
known  as a swap file) is a file on the hard disk that the memory manager uses  to
 hold data that does not fit in memory. The memory manager moves data  from the
 paging file to memory as needed and moves data from memory to  the paging file to
 make room for new data.

What Is PAE X86?: System Reliability

Actually, many companies fear a new os will make them re-write a lot of drivers/applications. Maybe that's why no XP drivers work in Vista? =S Or why Vista and XP are not interchangeable for drivers? Yet XP had a vast ability to run many different drivers for multiple machines. It's because it still was using some of the old driver standards, which are now gone in Vista (I think really SP2 changed a lot for XP).

PAE is an addon, it's all virtually addressed so you can "see" more ram, but infact 32bit can't actually see it. It's just an illusion, so when I say it wasn't ment to be, it really wasn't. We just kind of made an ugly way of doing it. Virtuilization. Though if I had said that w/o posting a ms link you'd say I'm retarded.

[edit] And before you go XP drivers will work with compat, yeah but I'm guessing those drivers completely met the newer standards, as the function calls weren't changed (obviously!) but as for XP itself, it didn't fully take on PAE because for consumers.

You really have to think about the market.... You think before Vista MS gave a crap about consumers and 4G of ram? Most of the people who use the "new" hardware is 10% of MS's market, and that 10% is probably reading this because they care about 4G or more of ram. A lot of the market IS letting companies use the same code over and over and over. Look around at most of the corp's, every fast food joint that runs survielance (still can't spell) is probably doing so under a Windows XP platform, using redicously old software.... How do I know this? A good friend of mine works for PEC in Iowa, and he happens to know the software that's run on the vast majority of cameras that they install. Which Paulson Electric isn't what I'd call small... And you know how much buisness MS would loose NOT supporting the drivers for these older cameras? Yeah Vista/Win7 doesn't, but even still SP3 isn't that old and I'm betting it made many companies happy.... This is how I know PAE is ugly, because for older drivers to not work for survielance it'd piss a lot of people off. And they don't want to re-write those drivers, many of the companies that work on the drivers won't update them anyways. So what do you do? MS has to taylor both ways to earn money... If they want to sell copies what do they do? Keep support for older stuff or tell people stfu? You can't always think of things moving ahead, a lot of the internet is running on not so new mainframes.

Example:
http://www.theregister.co.uk/2007/01...systems_part2/

Now if you really want to understand all this good for you, but I personally got annoyed with working on wiring for 20yr old equiptment. =( That's the bottom line though... You, the person who reads this, doesn't matter nearly as much as the corperation with money.

pallesenw said:

Let me ask you this: running a 64bit OS with only 32bit applications, is that also "emulation" of memory?

With the 64bit OS and 32bit apps you have a large physical address space and many smaller virtual address spaces. That gives you a great space to map all your 32bit applications to. If you install 32GB RAM and multitask many 32bit apps, you can do that without hitting the page file.

With a 32bit OS with PAE and 32bit apps: <insert here what I wrote above about the 64bit OS>

Edit: and just so you don't misunderstand: running a 64bit OS and 64bit applications still involves a page table where you "map" physical memory into your virtual address spaces.

Though I just have to wonder what this has to do with 32bit not supporting over 4G? I'm not going to argue that 32bit is emulated in 64bit space no, but what your saying just doesn't fit. It's like saying "oh since I can run 64bit apps in a 32bit kernel with emulation than a 32bit cpu was made to handle that?".... WTF No. I know 32bit is emulated in 64, but we aren't arguing that are we?

And about 64bit and memory, but the drivers can directly manage the memory, and the applications can too. Hence why any 32bit application maxes at 3.5G, go check it out. See the max memory any 32bit app can run at a time..... Then check 64bit apps. You'll have your answer.

pallesenw said:

mushroomboy said:

Though I just have to wonder what this has to do with 32bit not supporting over 4G? I'm not going to argue that 32bit is emulated in 64bit space no, but what your saying just doesn't fit. It's like saying "oh since I can run 64bit apps in a 32bit kernel with emulation than a 32bit cpu was made to handle that?".... WTF No. I know 32bit is emulated in 64, but we aren't arguing that are we?

You have misunderstood everything I wrote. I am not talking about how 32bit software is executed in a 64bit environment. I am talking about how memory access works (paging).

And about 64bit and memory, but the drivers can directly manage the memory, and the applications can too.

I am not sure what you mean by "directly". Your usage is only valid, if you mean a single 64bit application can use more than 4GB, because of its large VAS.

Hence why any 32bit application maxes at 3.5G, go check it out. See the max memory any 32bit app can run at a time..... Then check 64bit apps. You'll have your answer.

First, the 3.x number doesn't apply to the virtual address space. And that is the space you need to focus on when talking about how much an application can use.

Second, I am debating your "emulation" theory about how memory is used with PAE. Then the size of the VAS doesn't matter.

It's not the VAS, it's the fact that first they have to do virtual space, the vAS then within that it has to do another layer of virtual space for 32bit apps because they can't see the whole VAS because of the limit. If you didn't do another emulation layer making the 32bit applications think the VAS was only 3.5G you'd have problems. Hence why I say it's another virtual layer. If your in 64bit you only have the one layer, because a 64bit app doesn't need it sectioned into smaller 3.5G layers. Also, in 32bit apps they can't share the same memory, any memory outside the 3.5G section per app can't be seen by another application, though in 64bit that doesn't happen because they don't have the 3.5 limit. It's really just a layer on a layer. It's really wrong to say 32bit can address more than 3.5, because it can't. You can virtually set it up to address more but really your not doing that, it's an illusion as to where 64bit doesn't run into that.

[edit]
Basically once a single app reaches it's 3.5G limit, it can not and will not see anything outside that, as to where 64bit doesn't have that limit. That's the virtualization i'm talking about, because the space is set up that way. So lets say you want 2 applications to directly talk to each other, one is using 20MB the other is using it's limit, 3.5G what happens? The one using the 3.5G won't ever be able to see what that 20mb program is trying to share... Because they are in 2 different virtual spaces.

[side note]

And lets say you have 7G of ram ok? You can't have a program 'live' in the first 3.5G section and the seccond 3.5G section, why? Because if it sections both off in 3.5 incraments for apps to use they can't occupy both areas, you just can't have 2 different ram occupants with the same index. It would be like networking with IP conflicts, NO NO. And yes yes this is how it works.

[more thoughts]
See if you have hacking involved where you have to inject into memory (woah) you can't have the idea of each process having a virtualization of it's memory. If you did that then each app would get its own index (0-xxx) and only the OS would see it's memory and other programs couldn't see it. What does that imply? That would make hacking a LOT harder, as each thread would have it's own memory index. Fortunately that's not true, and there fore we are allowed program x to inject into program Y's memory. It's the same principal as the boot loader for the Xbox360 using the Video drivers graphix to overload the Hyperserver and cause memory addresses to overload. If what your saying is true, and it's all vituralized that wouldn't happen so easy. Now take that into affect that's the difference between PAE and non PAE. Because if a program uses 3.5G it can't see anything else, which would imply that it has used all of 0-xxx indexes.... Which means there are multiple indexes and multiple virtual spaces. THAT's what I'm talking about. As to where you only have one space, one area for physical mem. Does that make more sense?

This deserves a NEW reply just for this:

OWNED

Why?

Part 3: Memory Protection Technologies

Code:

Code:

         The primary driver compatibility concern is running Physical  Address Extension (PAE) mode on 32-bit systems. PAE mode enables  processors to address greater than 4 gigabytes (GB) of memory. The  primary difference between PAE memory paging and non-PAE memory paging  schemes is the extra level of paging that is required in PAE mode  (3 levels instead of 2).

THERE HAPPY!?!?!?!? There IS another layer of virtualization. BAH HUMBUG

[edit]

Oh and to make this so you understand where I'm comming from. That extra layer is an extra virtualization layer.

So with your logics, if I can boot and run a 64bit os in vmware on my 32bit PC does that mean my PC can boot 64bit linux? Think long and hard about that, because we all know the answer. Just because you make another layer for virtualization doesn't mean something was ment to do what we are doing now with it. Yeah my above post was wrong about memory but I was just using examples, I know each program gets it's own index, then the OS translates that into the index it sees.... But since the OS is using 64bit addressing and the application is using 32bit.... there is another virtualization layer that has to translate that. HAS TO WHAT?

See this:
http://www.boot-land.net/forums/inde...showtopic=9424
This is what we are doing, because 64bit register's are non-existant on a 32bit machine. So one really can not physically use more than the 32 (36) limit. We are using a 3rd layer, a what!?

What is it with all the "owned talk"? I could easily shoot everything down you just wrote. Because you have misunderstood all of it. But I wont.

mushroomboy said:

Hence why any 32bit application maxes at 3.5G, go check it out. See the max memory any 32bit app can run at a time..... Then check 64bit apps. You'll have your answer.

It's not the VAS, it's the fact that first they have to do virtual space, the vAS then within that it has to do another layer of virtual space for 32bit apps because they can't see the whole VAS because of the limit. If you didn't do another emulation layer making the 32bit applications think the VAS was only 3.5G you'd have problems. Hence why I say it's another virtual layer. If your in 64bit you only have the one layer, because a 64bit app doesn't need it sectioned into smaller 3.5G layers. Also, in 32bit apps they can't share the same memory, any memory outside the 3.5G section per app can't be seen by another application, though in 64bit that doesn't happen because they don't have the 3.5 limit. It's really just a layer on a layer. It's really wrong to say 32bit can address more than 3.5, because it can't. You can virtually set it up to address more but really your not doing that, it's an illusion as to where 64bit doesn't run into that.

[edit]
Basically once a single app reaches it's 3.5G limit, it can not and will not see anything outside that, as to where 64bit doesn't have that limit. That's the virtualization i'm talking about, because the space is set up that way. So lets say you want 2 applications to directly talk to each other, one is using 20MB the other is using it's limit, 3.5G what happens? The one using the 3.5G won't ever be able to see what that 20mb program is trying to share... Because they are in 2 different virtual spaces.

[side note]

And lets say you have 7G of ram ok? You can't have a program 'live' in the first 3.5G section and the seccond 3.5G section, why? Because if it sections both off in 3.5 incraments for apps to use they can't occupy both areas, you just can't have 2 different ram occupants with the same index. It would be like networking with IP conflicts, NO NO. And yes yes this is how it works.

[more thoughts]
See if you have hacking involved where you have to inject into memory (woah) you can't have the idea of each process having a virtualization of it's memory. If you did that then each app would get its own index (0-xxx) and only the OS would see it's memory and other programs couldn't see it. What does that imply? That would make hacking a LOT harder, as each thread would have it's own memory index. Fortunately that's not true, and there fore we are allowed program x to inject into program Y's memory. It's the same principal as the boot loader for the Xbox360 using the Video drivers graphix to overload the Hyperserver and cause memory addresses to overload. If what your saying is true, and it's all vituralized that wouldn't happen so easy. Now take that into affect that's the difference between PAE and non PAE. Because if a program uses 3.5G it can't see anything else, which would imply that it has used all of 0-xxx indexes.... Which means there are multiple indexes and multiple virtual spaces. THAT's what I'm talking about. As to where you only have one space, one area for physical mem. Does that make more sense?

mushroomboy said:

This deserves a NEW reply just for this:

OWNED

Why?

Part 3: Memory Protection Technologies

Code:

Code:

         The primary driver compatibility concern is running Physical  Address Extension (PAE) mode on 32-bit systems. PAE mode enables  processors to address greater than 4 gigabytes (GB) of memory. The  primary difference between PAE memory paging and non-PAE memory paging  schemes is the extra level of paging that is required in PAE mode  (3 levels instead of 2).

THERE HAPPY!?!?!?!? There IS another layer of virtualization. BAH HUMBUG

[edit]

Oh and to make this so you understand where I'm comming from. That extra layer is an extra virtualization layer.

So with your logics, if I can boot and run a 64bit os in vmware on my 32bit PC does that mean my PC can boot 64bit linux? Think long and hard about that, because we all know the answer. Just because you make another layer for virtualization doesn't mean something was ment to do what we are doing now with it. Yeah my above post was wrong about memory but I was just using examples, I know each program gets it's own index, then the OS translates that into the index it sees.... But since the OS is using 64bit addressing and the application is using 32bit.... there is another virtualization layer that has to translate that. HAS TO WHAT?

See this:
http://www.boot-land.net/forums/inde...showtopic=9424
This is what we are doing, because 64bit register's are non-existant on a 32bit machine. So one really can not physically use more than the 32 (36) limit. We are using a 3rd layer, a what!?