Security
Security in an OS is a very important issue. It not only includes security from outside threats, such as viruses and hacking, but internal security as well. For example, the OS is responsible for making sure that processes don't access memory outside of their own address space. If a program does so, it must be shutdown to protect other 'well behaved' programs. Operating system security can be divided into two basic categories, high-level and low-level security.
To see articles under this category, browse this list.
Low-level Protection Mechanisms
There are several different low-level protection mechanisms at the disposal of the operating system programmer. The first mechanism, called "CPU Rings" or more simply "rings", controls which CPU instructions are allowed to be executed. The second and third protection mechanisms are related to memory access. They are called "Paging" and "Segmentation" respectively. They control which areas of memory are allowed to be accessed and/or how those areas of memory are allowed to be accessed.
Rings
Rings offer a protection layer for programs. They allow certain levels of resource access to processes. This is good, because it keeps bad programs from messing things up. There are, however, several downsides: The more CPU rings you use, the more the OS is tied to the architecture. You can, however, have several architectures each with it's own ring switching code. Another issue with this is that you OS must have a TSS set up and several other features, making ring switching much more difficult than just running all programs in kernel mode. There are a total of 4 rings in most common architectures:
Ring 0
This is kernel mode or supervisor mode. This level has the least protection, and the most access to resources. When starting up, the OS runs in this mode unless it switches out. Interrupt handlers run in this mode.
Rings 1 and 2
These rings are mostly used for device drivers. They offer more protection, but not as much as ring 3.
Ring 3
This is the ring that most OS's use for applications. This ring is also called Userland, or Userspace. It has the most protection and the least resource access.
Most OS's use only Ring 0 and 3. This is because rings 1 and 2 are unneeded, as device drivers can run in either ring.
Sometimes applications need access to resources that their ring wont allow. If they try to access them, a General Protection Fault (int 13) will be triggered, and the application shutdown. The application must interface with the kernel somehow, and mostly this is done with System Calls.
Paging
See Paging
Segmentation
See Segmentation
High-level protection Mechanisms
High level security in an operating system can be accomplished in many different ways. One way would be through file permissions in a VFS. Files in *nix have a 'permission' value in their inode entry. This controls which users can read, write, execute or delete a file. These are mostly controlled by the File Systems.
Programming Language Specific Security
C/C++/ASM
Exploiting
Stack Based Overflows Stack-Based-Overflow
Heap Based Overflows Heap-Based-Overflows
Integer Overflows Integer-Overflows
Books on application Security Application-Security-Books
General Notes: Many operating systems around the osdev community trust user based input completely. While it is understood that most users are fighting to get a driver working/stable, it should also be understood that production level code should not contain fundamental security problems. For example, many system call implementations take values off the stack from users and store them into registers and them freely dereference them or use them for lengths/offsets etc. Using the above links, readers should be able to decipher why these are problems and how they can be used to gain elevated privledges inside an otherwise 'secure system'.
The weakest areas seen are by the current author are ELF parsers and network stack implementations. Many osdev projects trust integers and offsets taken from elf files inside their loaders and freely execute and store data at these addresses. Similarly, network protocol parsers are riddled with security related bugs mostly integer overflow/underflows that are then used to cause real security implications usually due to heap-based ramfications. ( For example: int size = int_from_user(); char *buf = malloc(size+1); memcpy(buf,data_from_user,size); ). As can be seen if the user sets size to (unsigned int)-1 then the allocation will allocate 0 bytes, which on most userland heap allocators will allocate space for the headers and no data, and cause a full blown heap overflow controllable by the user.
I will add more notes later if the above links do not help or if extra special attention is paid to this topic.
Prevention
While obviously these cannot be directly applied to hobby OSes, the ideas can certainly be used and ported to them. The best set of documents for securing systems can be found at PaX Design and Implementation
Pax provides almost all the operating system security you need. Reading the documents pertaining to its implementations should allow Hobby OS developers to use most of these ideas in their systems. Most current Windows/BSD (kernel) protections are based on these schemes. Userland Protections can be attributed to ASLR, Heap Cookies (along with general heap hardening), Stack Canaries, and NX.
Using these mechanisms does not prevent corruption of in memory data, but instead makes using the corruption for elevated privledges/advantage very difficult.
See Also
Articles
Threads
External Links
- Aug 2008: CPU Rings, Privilege, and Protection by Gustavo Duarte