NVMe

The NVMe Specifications can be found here.

Forum thread about this page here.

Introduction

Examples in several programming languages can be found at GitHub

NVMe is a non-volatile memory(NVM) controller you access over PCI_Express.

NVMe has queues that are used to issue Administration-Commands and NVM-Commands(storage related).

Commands generally complete in any order, and operate simultaneously.

The NVMe specification allows for up to 64K(65536) of submission queues, completion queues, queue sizes, command sector count, and simultaneous outstanding commands.

Real world hardware often has limitations that allow far less, but that needs to be queried from the controller.

Submission Queue

Submission queues are used to submit commands to the controller.

Submission queues are associated with a completion queue.

Multiple submission queues may be associated with the same completion queue.

Completion Queue

Completion queues are used by the controller to inform the driver of the completion of a previously submitted command.

Many commands may be outstanding, and complete in an arbitrary order that may be influenced by setting priorities on submission queues.

Admin Commands

Admin commands are used to do non-storage related activities, such as identify the controller features, storage size, create and destroy general purpose queues.

Admin commands are issued from a special admin submission queue, and show completions in a special admin completion queue.

There is only one admin queue, always named queue zero.

NVM Commands

NVM commands consist of general purpose Read-Sectors and Write-Sectors, which is the initial focus here, as well as more complex commands.

High-Level Walk-Through

Set up registers

Create admin queue

Identify

Create NVM queues

Issue commands

Handle completions

Shutting down