BYO NOS: Building a maintainable, SRE-friendly switch
6 hours ago
- #mlxsw
- #whitebox-switching
- #networking
- The author previously used an Arista 7050QX2-32S switch, which is nearing end of life, and sought alternatives due to the vendor's unique support for third-party daemons like BIRD.
- Whitebox switching options like SONiC and DENT were considered; DENT uses switchdev for Linux kernel integration, allowing hardware offload similar to Arista.
- A Mellanox SN2010 switch with Spectrum ASIC was selected for its switchdev support, and Debian was installed after configuring the kernel and udev rules for mlxsw.
- Breakout cables were configured using devlink, and network interfaces were managed with systemd-networkd, though it required verbose configuration.
- Routing challenges included limited ASIC route capacity; solutions involved resource reallocation, ECMP default routing, and route compression via a custom Go tool to reduce routes from ~1.2 million to ~92k.
- ACLs were implemented using TC (Traffic Control) with features like blocks and chain templates to optimize hardware offload and resource usage, including actions like goto, trap, and pedit.
- Egress control was achieved through VLAN rewrites and additional routing tables, allowing traffic steering based on DSCP values without reconfiguring the router.
- Monitoring utilized standard tools like Node Exporter and hsflowd, with custom exporters for devlink metrics, ensuring comprehensive system and dataplane visibility.
- Maintenance is simplified as the system runs on Debian with upstream tooling, enabling easy updates and config management, exemplified by a quick kernel update for security vulnerabilities.