Automatic Horizontal Scaling With Gpu Aware Load Balancing

via “automatic model partitioning and load balancing”

Microsoft's distributed training library — ZeRO optimizer, trillion-parameter scale, RLHF.

Unique: Automatic partitioning based on layer FLOP analysis and parameter counts; uses communication-aware heuristics to minimize inter-GPU communication while balancing compute load

vs others: Eliminates manual partitioning effort; more sophisticated than naive layer-by-layer splitting