OSDev.wiki

File Systems: Difference between revisions

[unchecked revision][unchecked revision]
← Older edit
Content deleted Content added
VisualWikitext
→‎Rolling your own: Neutralized this comment.
Undo revision 28897 by Drkeph (talk) duplicate line removed
 
(57 intermediate revisions by 28 users not shown)
Line 1:
{{Tone}}
Filesystems are the machine's way of ordering your data on readable and/or writable media. They provide a logical way to access the stuff that you have down on disk so that you can read or modify extit. Which file system you use depends upon what you want to do with it. For example, Windows uses the Fat32 or NTFS filesystem. If your disk is really huge, then there's no point using Fat32 because the FAT system was designed in the days when nobody had disks as big as we do now. At the same time, there's no point using a NTFS filesystem on a tiny disk, because it was designed to work with large volumes of data - the overhead would be pointless for, say, reading a 1.44m floppy disk.
 
{{Filesystems}}
 
File systems are the operating system's method of ordering data on persistent storage devices like disks. They provide an abstracted interface to access data on these devices in such a way that it can be read or modified efficiently. Which file system is convenient depends on the target application of the operating system. For example, Windows uses the FAT32 or NTFS file system. If a disk has a large capacity, FAT32 is inconvenient, because the FAT system was designed considering the smaller disks available at that time. At the same time, an NTFS file system is not convenient on a tiny disk, because it was designed to work with large volumes of data - there would be excessive overhead when using devices such as a 1.44 MB floppy disk.
 
For details on specific filesystems, browse [[:Category:Filesystems|this list]] of filesystems.
Line 7 ⟶ 11:
{{In Progress}}
 
A filesystem provides a generalized structure over persistent storage, allowing the low-level structure of the devices (e.g., disk, tape, flash memory storage) to be abstracted away. Generally speaking, the goal of a filesystem is allowing logical groups of data to be organized into ''files'', which can be manipulated as a unit. In order to do this, the filesystem must provide some sort of index of the locations of files in the actual secondary storage. The fundamental operations of any filesystem are:
=== Inodes ===
Inodes (information nodes) are a crucial design element in most Unix filesystems: Each file is made of data blocks (the sectors that contains your raw data bits), index blocks (containing pointers to data blocks so that you know which sector is the nth in the sequence), and one inode block.
 
* Tracking the available storage space
The inode is the root of the index blocks, and can also be the sole index block if the file is small enough. Moreover, as unix filesystems support hard links (the same file may appear several times in the directory tree), inodes are a natural place to store metadata such as file size, owner, creation/access/modification times, locks, etc.
* Tracking which block or blocks of data belong to which files
* Creating new files
* Reading data from existing files into memory
* Updating the data in the files
* Deleting existing files
 
(Perceptive readers will note that the last four operations - Create, Read, Update, and Delete, or CRUD - are also applicable to many other data structures, and are fundamental to databases as well as filesystems.)
 
Additionally, there are other features which go along with a practical filesystem:
 
* Assigning human-readable names to files, and renaming files after creation
* Allowing files to be divided among non-contiguous blocks in storage, and tracking the parts of files even when they are ''fragmented'' across the medium
* Providing some form of hierarchical structure, allowing the files to be divided into ''directories'' or ''folders''
* Buffering reading and writing to reduce the number of actual operation on the physical medium
* Caching frequently accessed files or parts of files to speed up access
* Allowing files to be marked as 'read-only' to prevent unintentional corruption of critical data
* Providing a mechanism for preventing unauthorized access to a user's files
 
Additional features may be found on some filesystems as well, such as automatic encryption, or journalling of read/write activity.
 
=== Indexing Methods ===
There are several methods of indexing the contents of files, with the most commonly used being ''i-nodes'' and ''File Allocation Tables''.
 
==== inodes ====
inodes (information nodes) are a crucial design element in most Unix file systems: Each file is made of data blocks (the sectors that contains your raw data bits), index blocks (containing pointers to data blocks so that you know which sector is the nth in the sequence), and one inode block.
 
The inode is the root of the index blocks, and can also be the sole index block if the file is small enough. Moreover, as Unix file systems support hard links (the same file may appear several times in the directory tree), inodes are a natural place to store Metadata such as file size, owner, creation/access/modification times, locks, etc.
 
==== FAT ====
The File Allocation Table ([[FAT]]) is the primary indexing mechanism for MS-DOS and it's descendants. There are several variants on FAT, but the general design is to have a table (actually a pair of tables, one serving as a backup for the first in case it is corrupted) which holds a list of blocks of a given size, which map to the whole capacity of the disk.
 
== Workings of File Systems ==
Line 36 ⟶ 69:
== Network File Systems ==
 
All these file systems are a way to create a large, distributed storage system from a collection of "back end" systems. That means you cannot (for instance) format a disk in 'NFS' but you instead mount a 'virtual' NFS partition that will reflect what's on another machine. Note that a new generation of Filefile Systemssystems is under heavy research, basing on latest P2P, cryptography and error correction techniques (such as the Ocean Store Project or Archival Intermemory.)
 
For details on various network file systems [[:Category:Network Filesystems|look here]]
Line 44 ⟶ 77:
 
=== "Beginners" filesystems ===
There are only fourfive filesystems that are both relatively easy to implement and worth to consider. There is no general recommendation as the choice depends largely on style and OS design. Instead you can read the comparison and make your own educated decision.
 
'''[[USTAR]]'''
* <code>+</code> Of these beginner "filesystems", this is the simplest by far to implement
* <code>+</code> Incredibly simple: a sector with metadata followed by data sectors
* <code>+</code> Widely used: utilities to create tar images are available for every mainstream OS and many minor ones
* <code>+</code> Supports special files (like devices and symlinks)
* <code>+</code> Supports Unix permissions
* <code>-</code> Not a filesystem in the common understanding of the term
* <code>-</code> Generally read-only, was never designed for in-place modifications
* <code>-</code> No support for fragmentation
* <code>-</code> No standard partition type for it (not that you should even consider using USTAR as a disk partition format)
* <code>-</code> Not actually the format used for ramdisks by things like Linux - that's CPIO
 
'''RAMFS/TMPFS'''
* <code>+</code> High flexibility of implementation
* <code>+</code> Fast
* <code>+</code> Will allow you to test out your VFS API without having to rely on filesystem specifics
* <code>+</code> *Highly* recommended as a starter filesystem to avoid morphing your VFS interface around a specific filesystem
* <code>+</code> Ideal to unpack a [[USTAR]] or [[CPIO]] [[initrd]] image into
* <code>-</code> Changes are, obviously, not persistent, and only in memory, to be wiped after a reboot
 
'''[[FAT]]'''
Line 50 ⟶ 103:
* <code>+</code> The 'standard' for floppies
* <code>+</code> Relatively easy to implement
* <code>-</code> Part of it involving long filenames and compatibility is [http://en.swpat.org/wiki/Microsoft_FAT_patents patented by Microsoft]
* <code>-</code> Patented by Microsoft. If you wish to use long file names you should pay them.
* <code>-</code> Large overhead
* <code>-</code> No support for large (>4GB4 GB) files
* <code>-</code> No support for unixUnix permissions
 
 
'''[[Ext2]]'''
* <code>+</code> Supports large files (with an extension)
* <code>+</code> Supports unixUnix permissions
* <code>+</code> Can be put on floppies
* <code>+</code> JournallingCan supportbe read and written from Linux
* <code>+-</code> Can not natively be read and written from linuxWindows (but [http://www.fs-driver.org/ drivers] are available)
* <code>-</code> Can not natively be read and written from windows
* <code>-</code> Very large overhead
* <code>-</code> Of these fourbeginner filesystems, this is the most complex filesystem
 
 
'''[[SFSBMFS]]'''
* <code>+</code> Supports large files
* <code>+</code> ByImplementation farAvailable theas easieststatic to implementlibrary
* <code>+</code> CanComes bewith pututility program for creating disk images on floppiesLinux and harddisksWindows
* <code>+</code> MinimalComes overheadwith FUSE bindings, allowing it to be mounted on Linux systems
* <code>+</code> Contains source code documentation
* <code>-</code> New, and therefore unsupported. The only operational utility is available for Windows.
* <code>-</code> NoDoes not support for unix permissionsfragmentation
* <code>-</code> Less control over the source code
 
 
'''[[ISO 9660]]'''
 
The defined standard for CDs. If you boot from CD then this is the way to go. If not, don't make it your first filesystem.
 
'''[[PureFS]]'''
* <code>+</code> Easy to implement
* <code>+</code> Supports large files
* <code>+</code> Supports nested directories
* <code>-</code> No support for Unix permissions
* <code>-</code> It can be journalable, provided that the OS takes care of this task itself
* <code>-</code> The maximum length of the file name is 255 characters
* <code>-</code> The maximum number of volumes is 40
* <code>-</code> Does not support Unicode names
* <code>-</code> Takes up a lot of space
 
=== Rolling your own ===
There are many different kinds of filesystems around, from the well-known to the more obscure ones. The most unfortunate thing about filesystems is that every hobbyist OS programmer thinks that the filesystem they design is the ultimate technology, when in reality it's usually just a copy of FAT with a change here and there, perhaps because it is one of the easiest to implement. The world doesn't need another FAT-like filesystem driver. Investigate all the possibilitespossibilities before you decide youto roll your own.
 
If despite of this warning you decide to create your own file system, then you should start that by implementing a [[FUSE]] driver for it. This gives the advantage that you can mount your file system image as any other storage device, and you can lists its contents, create new files and directories etc. with standard tools. FUSE is available for Linux, MacOSX and Windows as well.
 
==== Guidelines if you do decide to roll your own ====
* Consider carefully what it will be used for.
* Use a program to figure out the layout (e.g. a spreadsheet). The basic areas needed are:
** Bootsector. This is essential for booting on some systems such as BIOS-x86 and Atari ST, unnecessary for others such as UEFI and OpenFirmware. Even if you don't intend to boot on systems which require it, reserving the first sector will allow your OS to be ported to them at a later time. Note that reserving space for a MBR-like partition table is needed to allow the filesystem to work in "logical partitions".
** Partition metadata. This could fit into the first sector with the boot code, or be a separate group of sectors at a specific location. (FAT puts it in the first sector, calling it the FAT parameter block. ext* use a separate location, calling it the superblock.) At a minimum, this should contain the filesystem size, location of the file table, and a version number. Leave plenty of reserved space for features you don't think of. If you put it in the first sector, don't forget to leave space for a jmp instruction, the boot code, and a partition table!
** File table. Don't think of this as just a simple table containing a list of files and their locations. One idea is, instead of storing files, the system would store file parts, and the file table would list the parts in each file. This would be useful for saving space if many files on the disk are the same or similar (for example, license agreements).
** Data area. Files will be stored here.
* Create a program to read and write disk images with your filesystem. Parts of this will be portable into the fs driver.
* It is strongly recommended to create a [[FUSE]] driver for your file system.
* Implement the fs into your OS.
 
===Expert filesystems===
Once you have a beginner's file system under your belt you might want support for more advanced ones. Here are some:
*[[NTFS]] - (Windows) New Technologies File System. It's hard to find documentation. Try [http://www.opensource.apple.com/source/ntfs/ Apple NTFS (open source)].
*Btrfs - B-tree file system. It's a Linux file system with features such as copy-on-write and transparent compression.
 
== See Also ==
* [[I use a Custom Filesystem - What Bootloader Solution is right for me%3F]]
 
=== External links ===
* [https://blog.koehntopp.info/2023/05/05/50-years-in-filesystems-1974.html 50 years in filesystems] -- an approachable account of the history of file systems. Includes, among other information, some real-world examples of heuristics one can use to avoid fragmentation.
 
[[Category:OS theory]]
[[Category:Filesystems]]
Retrieved from "https://osdev.wiki/wiki/File_Systems"