Org Chart: Exploring Server Management Topologies

Introduction

Servers have been modular for decades. Components such as memory, storage, and expansion cards are often replaceable, and in many designs the entire compute module can also be pulled and replaced. Meanwhile, management hardware has taken different paths, each one with its own serviceability, security, and engineering tradeoffs.

In this article, we are going to compare five ways that management hardware is attached to a server, including our standard’s solution. We are a nonprofit developing an open hardware standard for modular, small form factor systems, primarily multi-node servers. One of our hardware specifications defines the Management Module, a vendor-agnostic, user-replaceable management component that can be implemented in various ways. We will go over how those two characteristics and more make it a unique and necessary solution for millions of users toward the end of this article.

What a BMC does

In enterprise servers, the management hardware is called the baseboard management controller (BMC). It is an independent computer running separately from the host or node hardware and software stack. Therefore, it keeps working when the host’s operating system has crashed or even when the host is turned off.

Administrators connect to a BMC through its own network and peripheral interfaces, which is why this kind of management architecture is also known as out-of-band management, i.e., it has a dedicated network separate from workload traffic. A typical BMC allows administrators to:

• Provision new machines
• Manage its host’s power and boot cycle
• Access its host’s console remotely via IP-KVM, as well as mount disk images remotely
• View telemetry about its host, as well as save logs and even create alerts related to those data
• Manage its host’s firmware
• Manage the server’s users, including the permissions of those users

Most of these functions can be automated through an API. The modern standard is DMTF’s Redfish, which is built on common web standards. BMCs themselves, however, are connected to their hosts in different ways.

BMC embedded on the server mainboard

The traditional enterprise server design sees the BMC soldered onto the same board that holds the host’s CPU and memory. We can trace its origins all the way back to 1998, with the release of the Intelligent Platform Management Interface (IPMI) v1.0 specification. While Redfish has largely replaced IPMI, the embedded BMC is still found in servers designed for data centers, such as the Supermicro X14 and H14 platforms, and ASRock Rack’s Altra servers.

The main advantages of this straightforward architecture is that it does not require any connectors and retention mechanisms, and the BMC has direct access to its host’s buses. The most obvious disadvantage of an embedded BMC is that it is part of the host board. If something else on the board fails, the BMC is included in the replacement even when it is still functional. This integration also means that vendors cannot easily change the components or capabilities of the BMC without redesigning the whole board.

The embedded BMC is also a central point of failure and a central point of attack. While enterprise vendors and their hardware partners do take numerous significant security measures, vulnerabilities related to the BMC can still exist. Just last year, security researchers were able to install malicious firmware in Supermicro motherboards by exploiting flaws in the vendor’s BMC firmware. The attack can hand over “full and persistent control” of the BMC and its host’s OS.

As a result of these disadvantages and other motivations, enterprise servers are gradually moving away from this architecture.

Shared BMC with bridge controllers

Instead of a full BMC per node, this architecture uses one BMC, then has a bridge controller on each node that reports to the BMC. This idea has actually been around since the early days of the IPMI specification, but it has not seen widespread adoption. Meta is perhaps the most active organization that is pursuing this architecture. Its Yosemite specification, managed by the Open Compute Project (OCP), uses a BMC mounted on a side plane to manage four independent nodes that each have a bridge circuit. Meta published an open-source firmware for bridge ICs called OpenBIC, but that has not been updated in years.

This approach allows a BMC to manage multiple nodes, which theoretically can be adopted by multiple vendors. But the shared BMC still represents a single point of failure.

BMC per node plus a chassis manager

In this architecture, each server node has its own BMC, while one or two management modules on the chassis sit above them to centralize management and control chassis-wide aspects such as power and active cooling. Examples of this architecture can be found in the Dell PowerEdge MX7000 chassis, the PowerEdge MX9002m management module, and compatible PowerEdge MX compute sleds. HPE also has a similar architecture in the Synergy 12000 Frame, Frame Link module, and its long-running Synergy 480 compute modules.

This architecture adds more serviceability since the chassis managers are modular. There are also configurations, such as the Dell and HPE examples we cited, where up to two modules can be installed for redundancy. Meanwhile, the BMC in each node allows for direct access in multi-rack, multi-server environments, while the vendor’s management platform and the chassis managers also provide a central management point. The tradeoff is of course complexity. The vendor has to develop and support two kinds of controllers as well as the interface between them.

BMC per node only

There are also servers where the chassis has no network-facing manager, and each node is managed solely through its management port. Lenovo actually implements this in two different ways. Its ThinkSystem D3 chassis still has a microcontroller mounted on its midplane, but compatible nodes each have their own BMC. Administrators nominate a “caretaker” node to handle chassis functions such as power supply and firmware updates.

Then there’s the Lenovo ThinkEdge SE100. It is a compact edge server that can fit in compatible rack enclosures, but it can also be a standalone unit. This is a unique advantage that saves space and cost. On the other hand, wiring and managing numerous such units can be more cumbersome than ones with centralized management, since every node needs its own out-of-band connection.

Vendor-agnostic modular management

This is the newest management topology. The management hardware is moved entirely off the server mainboard and onto a replaceable module that connects via a standardized connector. The OCP’s Datacenter Secure Control Module (DC-SCM) is the primary example of this approach. It puts the BMC, firmware flash storage, and security hardware, if any, onto a small card that plugs into one or two host processor modules (HPM). The first version of the DC-SCM specification was released in 2021, and is currently at Revision 2.2 as of this writing.

Examples of DC-SCMs include the Flex SCM202B and the Axiado Smart SCM. While the architecture is designed primarily for hyperscalers, it is also finding its way into OEM enterprise servers such as the Dell PowerEdge R770 and the ASUS RS720-E12 servers.

This complete decoupling from the server board has several significant advantages. A DC-SCM is independently serviceable and reusable. It also allows vendors to customize implementations for different features or regions. Equally important, the module and the HPM do not have to come from the same vendor.

In terms of disadvantages, at the moment, the limit of two HPMs per DC-SCM is a major one that the OCP hopes to remove in future versions. The module also reads identifying data from the HPM. If the latter’s hardware changes, the module’s firmware may need to be updated. Finally, despite being around for only five years, there is already fragmentation in the specification, as DC-SCM 1.0 and 2.x modules are not interchangeable.

Finally, we have the OpenSFF Management Module. Instead of having one module per one or two host boards, it sits in a dedicated slot in the Enclosure and serves every Compute Node installed. Our Enclosure Specification states that Enterprise Enclosures must have a Management Module slot and supporting management infrastructure, whereas it is optional in Core Enclosures.

The biggest advantage of the Management Module is that it is designed to be vendor-agnostic, including its default management software. Its specification also defines a purely hardware, aka Pass-through design, which is a more affordable implementation that provides only local access. Like the other modular hardware we mentioned, it is also independently serviceable and reusable.

The tradeoff of the Management Module is that it is also a single point of failure similar to an embedded BMC. We are also defining it to allow access to only one node at a time for KVM console and power control. Further, we do not intend for the Management Module to have all the capabilities that enterprise BMCs have, such as detailed telemetry and boot management. We are defining the module and our standard for use cases outside the traditional data center, and as such we balance these tradeoffs with the cost and complexity appropriate for our intended users.

Together with the OpenSFF Compute Node and Enclosure, which are also vendor-agnostic, the Management Module can bring out-of-band management to a wide range of systems, from servers to edge devices and even workstations. Many of these devices rely on non-dedicated management hardware such as the tools integrated in CPUs or affordable IP-KVM devices.

Build with OpenSFF

Enterprise platforms can justify more complex management hardware because their customers have different priorities, such as independent node management, redundancy, or integration with larger management systems. These advanced capabilities require more hardware, more firmware, and in some cases, more management layers.

We are taking back management hardware back to a simpler, centralized architecture but in a replaceable module, and focus only on the functionality that smaller deployments prioritize.

We encourage you to read our specifications, and we would be grateful if you help spread the word about OpenSFF. For technical clarifications, partnerships, and other inquiries, reach out to our development team at [email protected].