NVMe
The NVMe Specifications can be found here.
Forum thread about this page here.
TLDR FAQ
Introduction
Examples in several programming languages can be found at GitHub
NVMe is a non-volatile memory(NVM) controller you access over PCI_Express.
NVMe has queues that are used to issue Administration-Commands and NVM-Commands(storage related).
Commands generally complete in any order, and operate simultaneously.
The NVMe specification allows for up to, of 64K(65536) submission queues, completion queues, queue sizes, and simultaneous outstanding commands.
Real world hardware often has limitations that allow far less, but that needs to be queried from the controller.
Submission Queue
Submission queues are used to submit commands to the controller.
Completion Queue
Completion queues are used by the controller to inform the driver of the completion of a previously submitted command.
Admin Commands
Admin commands are used to do non-storage related activities, such as identify the controller features, storage size, create and destroy general purpose queues.
NVM Commands
NVM commands consist of general purpose Read-Sectors and Write-Sectors, which is the initial focus here, as well as more complex commands.
High-Level Walk-Through
Set up registers
Create admin queue
Create NVM queues
Issue commands
Handle completions
Shutting down