NVMe
The NVMe Specifications can be found here.
Forum thread about this page here.
TLDR FAQ
Introduction
Examples in several programming languages can be found at GitHub
NVMe is a non-volatile memory(NVM) controller you access over PCI_Express.
NVMe has queues that are used to issue Administration-Commands and NVM-Commands(storage related).
Commands generally complete in any order, and operate simultaneously.
The NVMe specification allows for up to 64K(65536) of submission queues, completion queues, queue sizes, command sector count, and simultaneous outstanding commands.
Real world hardware often has limitations that allow far less, but that needs to be queried from the controller.
Submission Queue
Submission queues are used to submit commands to the controller.
Submission queues are associated with a completion queue.
Multiple submission queues may be associated with the same completion queue.
Completion Queue
Completion queues are used by the controller to inform the driver of the completion of a previously submitted command.
Many commands may be outstanding, and complete in an arbitrary order that may be influenced by setting priorities on submission queues.
Admin Commands
Admin commands are used to do non-storage related activities, such as identify the controller features, storage size, create and destroy general purpose queues.
Admin commands are issued from a special admin submission queue, and show completions in a special admin completion queue.
There is only one admin queue, always named queue zero.
NVM Commands
NVM commands consist of general purpose Read-Sectors and Write-Sectors, which is the initial focus here, as well as more complex commands.
High-Level Walk-Through
Set up registers
Create admin queue
Create NVM queues
Issue commands
Handle completions
Shutting down