Detecting Memory (x86): Difference between revisions
[unchecked revision] | [unchecked revision] |
Content deleted Content added
m →Memory Map Via GRUB: Changed "real kilobytes"(=1024) to "kibibytes" (=1024) see wikipedia for further information |
m Fix lint errors |
||
(63 intermediate revisions by 29 users not shown) | |||
Line 1:
{{Bias}}
One of the most vital pieces of information that an OS needs in order to initialize itself is a map of the available RAM on a machine.
Fundamentally, the best way
There may be rare machines where you have no other choice but to try to detect memory yourself -- however, doing so is unwise in any other situation.
Line 20 ⟶ 21:
See [[Memory Map (x86)]] for a generic layout of memory.
==Detecting Low Memory==
"Low memory" is the available RAM below
INT 0x12: The INT 0x12 call will return AX = total number of
The AX value measures from 0, up to the bottom of the EBDA (of course, you probably shouldn't use the first 0x500 bytes of the space either --
i.e. the IVT or BDA).
Usage:
<
;
; Switch to the BIOS (= request low memory size)
int 0x12
; The carry flag is set
jc .Error
; AX = amount of continuous memory in KB starting from 0.
</syntaxhighlight>
Note: this function is supposed to be always present, and may not modify the carry flag. If an emulator doesn't support it, the carry flag will be set, indicating error.
Alternately, you can just use INT 0x15, EAX = 0xE820 (see below).
Line 55 ⟶ 51:
===BIOS Function: INT 0x15, EAX = 0xE820===
See also: [http://www.uruk.org/orig-grub/mem64mb.html]
By far the best way to detect the memory of a PC is by using the INT 0x15, EAX = 0xE820 command.
Line 61 ⟶ 59:
In reality, this function returns an unsorted list that may contain unused entries and (in rare/dodgy cases) may return overlapping areas.
Each list entry is stored in memory at ES:DI, and DI is <b>not</b> incremented for you. The format of an entry is 2
plus one additional
It is probably best to always store the list entries as 24 byte quantities -- to preserve
that last
* First
* Second
* Next
** Type 1: Usable (normal) RAM
** Type 2: Reserved - unusable
Line 74 ⟶ 72:
** Type 4: ACPI NVS memory
** Type 5: Area containing bad memory
* Next
** Bit 0 of the Extended Attributes indicates if the entire entry should be ignored (if the bit is clear). This is going to be a huge compatibility problem because most current OSs won't read this bit and won't ignore the entry.
** Bit 1 of the Extended Attributes indicates if the entry is non-volatile (if the bit is set) or not. The standard states that "Memory reported as non-volatile may require characterization to determine its suitability for use as conventional RAM."
Line 81 ⟶ 79:
Basic Usage:<br>
For the first call to the function, point ES:DI at the destination buffer for the list. Clear EBX. Set EDX to the magic number 0x534D4150. Set EAX to
0xE820 (note that the upper
If the first call to the function is successful, EAX will be set to 0x534D4150, and the Carry flag will be clear.
Line 119 ⟶ 117:
It is possible to get a fairly good memory map using Plug 'n Play (PnP) calls. {Need description and code.}
====SMBIOS====
SMBIOS is designed to allow "administrators" to assess hardware upgrade options or maintain a catalogue of what hardware a company current has in use (ie. it provides information for use by humans, rather than for use by software). It may not give reliable results on many computers -- see:
Line 129 ⟶ 126:
Detecting memory with these functions completely ignores the concept of memory holes / memory mapped devices / reserved areas.
====BIOS Function: INT 0x15, AX = 0xE881====
Should be available on everything made after 1994.
Since this is meant to be used in protected mode, the INT instruction cannot be used to call it. The correct calling procedure is unknown, but it might be the same as for protected mode EISA BIOS functions. Further information is probably buried in a Compaq technical reference manual somewhere.
====BIOS Function: INT 0x15, AX = 0xE801====
Line 152 ⟶ 151:
Linux Usage:
<
XOR CX, CX
XOR DX, DX
Line 169 ⟶ 168:
; AX = number of contiguous Kb, 1M to 16M
; BX = contiguous 64Kb pages above 16M
</syntaxhighlight>
====BIOS Function: INT 0x15, AX = 0xDA88====
This function returns the number of contiguous KiB of usable RAM starting at 0x00100000 in
If this function isn't supported it'll return "carry = set".
====BIOS Function: INT 0x15, AH = 0x88====
Note: This function may limit itself to reporting 15M (for legacy reasons) even if BIOS detects more
memory than that. It may also report up to 64M. It only reports contiguous (usable) RAM. In some BIOSes, it may not clear the CF flag on success.
Usage:
<
CLC ; CF bug workaround
MOV AH, 0x88
INT 0x15 ; request upper memory size
Line 190 ⟶ 189:
TEST AX, AX ; size = 0 is an error
JE SHORT .ERR
; AX = number of contiguous KB above 1M
</syntaxhighlight>
====BIOS Function: INT 0x15, AH = 0x8A====
Line 207 ⟶ 202:
If this function isn't supported it'll return "carry = set".
At least one Award BIOS has a bug where it pushes BX without a corresponding pop, resulting in a return to IP:BX with the flags still on the stack. The aforementioned BIOS also supports E820, so the simplest workaround is to avoid this function when E820 works.
====BIOS Function: INT 0x15, AH = 0xC7====
Although not widely supported, this function defined by IBM
<pre>
Size Offset Description
</pre>
The minimum number which can be returned by the first
* Local Memory on the system board or memory that is not accessible from the channel. It can be system or nonsystem memory.
Line 241 ⟶ 237:
Usage:
<
unsigned short total;
unsigned char lowmem, highmem;
Line 252 ⟶ 248:
total = lowmem | highmem << 8;
return total;
</syntaxhighlight>
====E820h====
Line 258 ⟶ 254:
There are a few other BIOS functions that claim to give you memory information. However, they are so widely unsupported that it is impossible to even find machines to test the code on. <b>All</b> current machines support E820 (above). If some user should happen to dig up such a dinosaur of a machine that its BIOS does not support any standard memory detection function -- they will not complain that your modern OS fails to support that machine. Just give an error message.
==== Manual Probing ====
{{Warning|This could possibly damage your computer.}}
Use BIOS to get a memory map, or use [[GRUB]] (which calls BIOS for you). Memory probing can have results that are '''unpredictable by nature''' because memory probing is unsupported by vendors.
===== Theoretical introduction =====
Direct memory probing is only useful for very old systems with buggy and/or non-updateable BIOSes, or maybe a system with modified hardware that does not match the firmware anymore. So if you don't intend to support this sort of computers, you don't need memory probing. Period. Even if you need, you may decide that it's safer to run will less memory than is available.
Anyway, don't think that this will save you from the effort of understanding how to call the complicated BIOS APIs. Before launching a probe, you will always need to detect '''if''' the computer where your OS is running really needs it, and '''which''' memory areas really need to be probed (because for other memory areas you should always consider the information provided by the BIOS as authoritative). You also need to take into account the appropriate memory holes and/or memory mapped devices, which may vary from system to system.
When perfectly implemented, directly probing memory may allow you to detect extended memory
However, the BIOS
Note: You will never get an error from trying to read/write memory that does not exist -- this is important to understand: you will not get valid results, but you won't get an error, either.
=====
There follows a list of technical difficulties involved in memory probing that may help to implement such an algorithm if you end up having to do it:
* There can be important data structures left in RAM by the BIOS (e.g. the ACPI tables) that you'd trash. Those structures may be anywhere and the only way to know their addresses is to "ask" the BIOS.
* There can be a memory mapped device from 15 MB to 16 MB (typically "VESA local bus" video cards, or older ISA cards that aren't limited to just video).
* There can also be
* On modern systems there can also be faulty RAM that INT 0x15, eax=0xE820 says not to use, that can be anywhere (except in the first 1 MB, probably).
* There can also be large arbitrary memory holes. (e.g. a NUMA system with RAM up to 0x1FFFFFFF, a hole from 0x20000000 to 0x3FFFFFFF, then more RAM from 0x40000000 to 0x5FFFFFFF.)
Line 283 ⟶ 280:
* There are also (typically older) motherboards where you can write a value to "nothing" and read the same value back due to bus capacitance; there are motherboards where you write a value to cache and read the same value back from cache even though no RAM is at that address.
* There are (older, mostly 80386) motherboards that remap the RAM underneath the option ROMs and BIOS to the end of RAM. (e.g. with 4 MB of RAM installed you get RAM from 0x00000000 to 0x000A0000 and more RAM from 0x00100000 to 0x00460000, which causes problems if you test each MB of RAM because you get the wrong answer -- either under-counting RAM up to 0x00400000, or over-counting RAM up to 0x00500000).
* If you write the code properly (ie. to avoid as many of the problems as you can), then it is <i>insanely</i> slow.
* Lastly, testing for RAM (if it actually works) will only tell you where RAM is - it doesn't give you a complete map of the physical address space. You won't know where you can safely put memory mapped PCI devices because you won't know which areas are reserved for chipset things (e.g. SMM, ROMs), etc.
Line 291 ⟶ 287:
==Memory Map Via GRUB==
[[GRUB]], or any bootloader implementing [http://www.gnu.org/software/grub/manual/multiboot/multiboot.html The Multiboot Specification] provides a convenient way of detecting the amount of RAM your machine has. Rather than re-invent the wheel, you can ride on the hard work that others have done by utilizing the multiboot_info structure. When GRUB runs, it loads this structure into memory and leaves the address of this structure in the EBX register. You may also view this structure at the GRUB command-line with the GRUB command <tt>displaymem</tt> and GRUB 2 command <tt>lsmmap</tt>.
The methods it uses are:
Line 300 ⟶ 296:
However, it does not take into account any bugs that are known to effect some BIOSs (see entries in [[RBIL]]). It does not check "E801" and/or "88" returning with carry set.
To utilize the information that GRUB passes to you, first include the file [https://www.gnu.org/software/grub/manual/multiboot/html_node/multiboot_002eh.html multiboot.h] in your kernel's main file. Then, make sure that when you load your _main function from your assembly loader, you push EAX and EBX onto the stack. Make sure to do this before calling any other initialization functions, such as [[Calling_Global_Constructors|global constructor initialization]], or the initialization function(s) will probably clobber the registers. Then, define your start function as such:
_main (multiboot_info_t* mbd, unsigned int magic) {...}
The key for memory detection lies in the multiboot_info struct. To determine the contiguous memory size, you may simply check <tt>mbd->flags</tt> to verify bit 0 is set, and then you can safely refer to <tt>mbd->mem_lower</tt> for conventional memory (e.g. physical addresses ranging between 0 and 640KB) and <tt>mbd->mem_upper</tt> for high memory (e.g. from 1MB). Both are given in kibibytes, i.e. blocks of 1024 bytes each.
To get a complete memory map, check bit 6 of <tt>mbd->flags</tt> and use <tt>mbd->mmap_addr</tt> to access the BIOS-provided memory map. Quoting [http://www.gnu.org/software/grub/manual/multiboot/html_node/Boot-information-format.html#Boot%20information%20format specifications],
{{Quotation
| If bit 6 in the flags
}}
Taking this into account, our example code would look like the following. Note that if you prefer a version that does not require the multiboot.h header downloaded from the link above, there is a version listed in the code examples section of this article.
<syntaxhighlight lang="c">
#include "multiboot.h"
void _main(multiboot_info_t* mbd, uint32_t magic)
{
/* Make sure the magic number matches for memory mapping*/
if(magic != MULTIBOOT_BOOTLOADER_MAGIC) {
panic("invalid magic number!");
}
/* Check bit 6 to see if we have a valid memory map */
if(!(mbd->flags >> 6 & 0x1)) {
panic("invalid memory map given by GRUB bootloader");
}
/* Loop through the memory map and display the values */
int i;
for(i = 0; i < mbd->mmap_length;
i += sizeof(multiboot_memory_map_t))
{
multiboot_memory_map_t* mmmt =
(multiboot_memory_map_t*) (mbd->mmap_addr + i);
printf("Start Addr: %x | Length: %x | Size: %x | Type: %d\n",
mmmt->addr, mmmt->len, mmmt->size, mmmt->type);
if(mmmt->type == MULTIBOOT_MEMORY_AVAILABLE) {
/*
* Do something with this memory block!
* BE WARNED that some of memory shown as availiable is actually
* actively being used by the kernel! You'll need to take that
* into account before writing to memory!
*/
}
}
}
</syntaxhighlight>
'''WARNING:''' If you downloaded the multiboot header from gnu.org (linked above) you probably got a version which defines the base address and length fields as one 64-bit unsigned integer each, rather than two 32-bit unsigned integers each. [https://forum.osdev.org/viewtopic.php?t=30318 This may cause gcc to pack the structure incorrectly] which can lead to nonsensical values when you try to read it.
* The linked forum post blames GCC for not properly packing the multiboot struct, however, the real error was the printf implementation/usage. When using the type uint64_t, you must specify %lx (instead of %x) so that printf will read all 64-bits as one argument rather than reading the high-32 as one argument and the low-32 as the next argument.
Alternatively, you can modify the multiboot header, specifically the multiboot_mmap_entry struct, to the following to get the correct values:
<syntaxhighlight lang="c">
struct multiboot_mmap_entry
{
multiboot_uint32_t size;
multiboot_uint32_t addr_low;
multiboot_uint32_t addr_high;
multiboot_uint32_t len_low;
multiboot_uint32_t len_high;
#define MULTIBOOT_MEMORY_AVAILABLE 1
#define MULTIBOOT_MEMORY_RESERVED 2
#define MULTIBOOT_MEMORY_ACPI_RECLAIMABLE 3
#define MULTIBOOT_MEMORY_NVS 4
#define MULTIBOOT_MEMORY_BADRAM 5
multiboot_uint32_t type;
} __attribute__((packed));
typedef struct multiboot_mmap_entry multiboot_memory_map_t;
</syntaxhighlight>
Each multiboot mmap entry is stored as the following:
:{| {{wikitable}}
|-
!
| size
|-
!
| base_addr_low
|-
!
| base_addr_high
|-
!
| length_low
|-
!
| length_high
|-
!
| type
|-
|}
* "size" is the size of the associated structure in bytes, which can be greater than the minimum of 20 bytes. base_addr_low is the lower 32 bits of the starting address, and base_addr_high is the upper 32 bits, for a total of a 64-bit starting address. length_low is the lower 32 bits of the size of the memory region in bytes, and length_high is the upper 32 bits, for a total of a 64-bit length. type is the variety of address range represented, where a value of 1 indicates available RAM, and all other values currently indicated a reserved area.
* GRUB simply uses INT 15h, EAX=E820 to get the detailed memory map, and does not verify the "sanity" of the map. It also will not sort the entries, retrieve any available ACPI 3.0 extended
* One of the problems you have to deal with is that
* Another problem is that the "type" field is defined as "1 = usable RAM" and "anything else is unusable". Despite what the multi-boot specification says, lots of people assume that the type field is taken directly from INT 15h, EAX=E820 (and in older versions of GRUB it is). However
==Memory Detection in Emulators==
Line 350 ⟶ 411:
that does not exactly match your expectations.
== What about on UEFI? ==
On UEFI, you have 'BootServices->GetMemoryMap'. This function is similar to E820 and is the only solution on new UEFI machines. Basically, to use, first you call it once to get the size of the memory map. Then you allocate a buffer of that size, and then call again to get the map itself. Watch out, by allocating memory you could increase the size of the memory map. Considering that a new allocation can split a free memory area into two, you should add space for 2 additional memory descriptors. It returns an array of EFI_MEMORY_DESCRIPTORs. They have the following format (taken from GNU EFI):
<syntaxhighlight lang="c">
typedef struct {
UINT32 Type; // EFI_MEMORY_TYPE, Field size is 32 bits followed by 32 bit pad
UINT32 Pad;
EFI_PHYSICAL_ADDRESS PhysicalStart; // Field size is 64 bits
EFI_VIRTUAL_ADDRESS VirtualStart; // Field size is 64 bits
UINT64 NumberOfPages; // Field size is 64 bits
UINT64 Attribute; // Field size is 64 bits
} EFI_MEMORY_DESCRIPTOR;
</syntaxhighlight>
To traverse them, you can use the NEXT_MEMORY_DESCRITOR macro.
Memory types are different to the E820 codes. For converting, see [https://github.com/tianocore/edk2/blob/70d5086c3274b1a5b099d642d546581070374e6e/OvmfPkg/Csm/LegacyBiosDxe/LegacyBootSupport.c#L1601 CSM E820 compatibility].
<syntaxhighlight lang="c">
typedef enum {
EfiReservedMemoryType,
EfiLoaderCode,
EfiLoaderData,
EfiBootServicesCode,
EfiBootServicesData,
EfiRuntimeServicesCode,
EfiRuntimeServicesData,
EfiConventionalMemory,
EfiUnusableMemory,
EfiACPIReclaimMemory,
EfiACPIMemoryNVS,
EfiMemoryMappedIO,
EfiMemoryMappedIOPortSpace,
EfiPalCode,
EfiPersistentMemory,
EfiMaxMemoryType
} EFI_MEMORY_TYPE;
</syntaxhighlight>
==Code Examples==
===Getting a GRUB Memory Map===
Declare the appropriate structure, get the pointer to the first instance, grab whatever address and length information you want, and finally skip to the next memory map instance by adding size+
<
typedef struct multiboot_memory_map {
unsigned int size;
Line 367 ⟶ 462:
unsigned int type;
} multiboot_memory_map_t;
// this is really an entry, not the entire map.
typedef multiboot_memory_map_t mmap_entry_t;
int main(multiboot_info* mbt, unsigned int magic) {
...
while(
// do something with the entry
}
...
}
</syntaxhighlight>
===Getting an E820 Memory Map===
<
; use the INT 0x15, eax= 0xE820 BIOS function to get a memory map
; note: initially di is 0, be sure to set it to a value so that the BIOS code will not be overwritten.
; The consequence of overwriting the BIOS code will lead to problems like getting stuck in `int 0x15`
; inputs: es:di -> destination buffer for 24 byte entries
; outputs: bp = entry count, trashes all registers except esi
mmap_ent equ 0x8000 ; the number of entries will be stored at 0x8000
do_e820:
mov di, 0x8004 ; Set di to 0x8004. Otherwise this code will get stuck in `int 0x15` after some entries are fetched
xor ebx, ebx ; ebx must be 0 to start
xor bp, bp ; keep an entry count in bp
Line 416 ⟶ 516:
je short .skipent
.notext:
mov ecx, [es:di + 8] ; get lower
or ecx, [es:di + 12] ; "or" it with upper
jz .skipent ; if length
inc bp ; got a good entry: ++count, move to next storage spot
add di, 24
Line 431 ⟶ 531:
stc ; "function unsupported" error exit
ret
</syntaxhighlight>
Sample in C (assuming we are in a bootloader environment, real mode, DS and CS = 0000):
<
// running in real mode may require:
__asm__(".code16gcc\n");
Line 442 ⟶ 542:
typedef struct SMAP_entry {
uint32_t BaseL; // base address
uint32_t BaseH;
uint32_t LengthL; // length
uint32_t LengthH;
uint32_t Type; // entry
uint32_t ACPI; //
}__attribute__((packed)) SMAP_entry_t;
// load memory map to buffer - note: regparm(3) avoids stack issues with gcc in
int __attribute__((noinline)) __attribute__((regparm(3))) detectMemory(SMAP_entry_t* buffer, int maxentries)
{
Line 492 ⟶ 592:
}
}
</syntaxhighlight>
===Getting an UEFI Memory Map===
<syntaxhighlight lang="c">
EFI_STATUS Status;
EFI_MEMORY_DESCRIPTOR *EfiMemoryMap;
UINTN EfiMemoryMapSize;
UINTN EfiMapKey;
UINTN EfiDescriptorSize;
UINT32 EfiDescriptorVersion;
//
// Get the EFI memory map.
//
EfiMemoryMapSize = 0;
EfiMemoryMap = NULL;
Status = gBS->GetMemoryMap (
&EfiMemoryMapSize,
EfiMemoryMap,
&EfiMapKey,
&EfiDescriptorSize,
&EfiDescriptorVersion
);
ASSERT (Status == EFI_BUFFER_TOO_SMALL);
//
// Use size returned for the AllocatePool.
//
EfiMemoryMap = (EFI_MEMORY_DESCRIPTOR *) AllocatePool (EfiMemoryMapSize + 2 * EfiDescriptorSize);
ASSERT (EfiMemoryMap != NULL);
Status = gBS->GetMemoryMap (
&EfiMemoryMapSize,
EfiMemoryMap,
&EfiMapKey,
&EfiDescriptorSize,
&EfiDescriptorVersion
);
if (EFI_ERROR (Status)) {
FreePool (EfiMemoryMap);
}
//
// Get descriptors
//
EFI_MEMORY_DESCRIPTOR *EfiEntry = EfiMemoryMap;
do {
// ... do something with EfiEntry ...
EfiEntry = NEXT_MEMORY_DESCRIPTOR (EfiEntry, EfiDescriptorSize);
} while((UINT8*)EfiEntry < (UINT8*)EfiMemoryMap + EfiMemoryMapSize);
</syntaxhighlight>
===Manual Probing in C===
Line 500 ⟶ 649:
* the assembly language manual probing code that follows this example is better
<
/*
* void count_memory (void)
Line 575 ⟶ 724:
outb(0xA1, irq2);
}
</syntaxhighlight>
===Manual Probing in ASM===
Line 592 ⟶ 741:
* it's better to assume that memory holes are present (and risk skipping some RAM) than to assume that memory holes aren't present (and risk crashing). This means assuming that the area from 0x00F00000 to 0x00FFFFFF can't be used and not probing this area at all (it's possible that some sort of ISA device is in this area, and that any write to this area can cause problems).
<
;Probe to see if there's RAM at a certain address
;
Line 604 ⟶ 753:
; ecx Number of bytes of RAM found
; esi Address of RAM
probeRAM:
Line 620 ⟶ 768:
sub edx,eax ;edx = number of bytes left to test
jc .done ; all done if nothing left after rounding
or esi,0x00000FFC ;esi = address of last
shr edx,12 ;edx = number of blocks to test (rounded down)
je .done ; Is there anything left after rounding?
Line 650 ⟶ 798:
pop eax
ret
</syntaxhighlight>
Further Notes:
Line 662 ⟶ 810:
===Threads===
* [http://www.osdev.org/phpBB2/viewtopic.php?t=11391 Grub's multiboot memory map], featuring real examples of GRUB/BIOS reported memory map.
* [https://forum.osdev.org/viewtopic.php?t=30318 QEMU Multiboot Invalid Memory Map], a potential pitfall when detecting memory with GRUB
[[Category:X86]]
[[Category:Physical Memory]]
[[Category:Hardware Detection]]
|