Designing for Density: A Guide to Storage-Focused OpenSFF Compute Nodes

Introduction

OpenSFF is a nonprofit organization developing open-source hardware specifications. We define three components: the Compute Node, the Enclosure, and the Management Module. These components can be combined to create a wide range of vendor-agnostic systems, from servers to edge devices.

Our specifications leave a lot of room for vendors to decide. In the case of the Compute Node, we are aware that some workloads call for more specialized nodes, including ones that are focused on providing plenty of storage. While we touched on the feasibility of creating storage-focused nodes in our node spinoff and rear I/O shield articles, we created this article to give vendors a better understanding of their options.

What the Compute Node Specification says

Despite its name, the Compute Node Specification is not prescriptive about compute. It mandates interoperability and serviceability, while leaving component-level decisions to vendors. That includes the type and quantity of storage media. This philosophy makes storage-focused nodes possible. The specification also mentions that vendors can add components to the underside of the node’s PCB, provided they stay within the 6mm height limit. That leaves more space for additional storage such as M.2 NVMe drives.

Selecting a storage form factor

M.2 NVMe

This is the easy default option. It is a great middle ground in terms of price and availability, and its compact footprint makes it easy to fit into a Compute Node’s PCB envelope, including, as we mentioned, on the underside of the board.

A Compute Node with at least two M.2 NVMe drives would be suited for developer workstations and workloads that involve small databases or those that require a fast local scratch drive.

The biggest tradeoff of M.2 NVMe drives is that their connectors are not designed for hot swapping and require specialized enclosures to support the feature. M.2 NVMe hot-swapping is also highly dependent on operating systems, drivers, and even filesystems. For instance, while ZFS supports hot-swapping, it expects drives to be replaced in a specific order or process. Hot-swap backplanes and the supporting circuitry also add cost and complexity.

Beyond hot-swapping considerations, vendors also have to make sure that their M.2 slots receive adequate power under sustained load.

EDSFF NVMe

EDSFF is the opposite of M.2 in terms of cost and serviceability. The standard is designed to natively support hot-swapping, but EDSFF drives are significantly more expensive than M.2 drives. As such, they are rarely seen outside hyperscale and enterprise deployments.

The Compute Node’s rear I/O shield has enough space for two to four E1.S drive bays. This configuration would be great for workloads that wear drives out relatively quickly, justifying the premium that users would pay to have fast, hot-swappable drives.

2.5”/3.5” SATA

SATA is the cheapest of the three form factors by a wide margin, and it remains the best choice for bulk storage. SATA hot-swap is a known quantity from a technical standpoint, but support does vary across operating systems.

A Compute Node with four 2.5” SATA drives would be highly suited for storing media files and archives, and other workloads where capacity matters more than speed.

When designing SATA-heavy nodes, space and vibration within mechanical drives are the prime considerations. SATA drives take up considerably more board space, and the layout and node design must compensate for the vibrations of spinning drives that sit close together. Vendors must also ensure that the drives are adequately cooled even when the OpenSFF-compatible system is installed in a poorly ventilated space.

Local storage vs. Enclosure-bound storage

Our Enclosure Specification is even more open-ended than the Compute Node Specification. Enclosures have no fixed form factor or layout, which means vendors are free to design an Enclosure with dedicated drive bays. The right storage architecture comes down to whether users need their data to remain on the node.

Local, node-based storage

Having all of the drives on the Compute Node is simpler and self-contained. It also relieves the network from accommodating storage traffic. Node storage is a great fit for boot drives, caching, scratch disks, and workloads involving local databases. Of course, vendors are limited to what a Compute Node can physically, electrically, and thermally accommodate.

Serviceability may also be an important factor. Vendors who want to offer user-replaceable drives must consider how the node’s cooling shroud can be repeatedly removed and replaced, or if the entire PCB should be removable for convenient access to the drive slots. Our specifications also strongly recommend using tool-less mechanisms wherever possible.

Enclosure-bound storage

Enclosures have more space for dedicated storage bays. This setup also facilitates swappable drives and shared storage in a multi-node system. Vendors do need to account how data will travel to and from nodes, and accessibility and serviceability must still be prime considerations in this architecture.

Considerations for multi-node systems with a storage node

Designing a storage node for a multi-node system adds considerations beyond the ones we already discussed. As with Enclosure-bound storage, the network can become the bottleneck when other nodes in an OpenSFF-compatible system rely on a storage node.

Vendors must also ensure that the storage node’s processor is up to the task, particularly when feeding multiple NVMe drives. This involves careful allocation of PCB space. A more powerful CPU needs larger passive cooling, which in turn competes with the drives for board space. The same is true for the RAM modules.

As for users, they must be aware that a storage node can be a single point of failure. Some deployments are better served by running multiple storage nodes for redundancy. They must then look into Enclosures that have enough node slots to support practical configurations.

For instance, a five-slot Enclosure can be the foundation for an OpenSFF-compatible system with redundant local storage. Three general purpose Compute Nodes can offload their storage to two storage nodes. In this case, users must ensure that the processors and network interfaces of the storage nodes are up to the task.

Build with OpenSFF

We do not include dedicated sections for storage nodes or any other purpose-built implementations in the Compute Node Specification. Instead, we inform vendors about our requirements for interoperability and serviceability. It is up to them to design Compute Nodes for their target use case, price tier, or audience, whether that is a budget-friendly SATA node, a dense M.2 database node, or a high-IOPS EDSFF cluster node.

We encourage you to read our specifications, and we would be grateful if you spread the word about OpenSFF. For technical clarifications, partnerships, and other inquiries, reach out to our development team at [email protected].