Transformer Block Level Routing And Sequencing

via “transformer-block-assembly”

A guide to building your own working LLM, by Sebastian Raschka.

Unique: Shows the complete assembly of transformer blocks with explicit tensor shape tracking and component ordering, making architectural decisions (pre-norm vs post-norm) explicit and modifiable

vs others: More transparent than using high-level framework modules, enabling practitioners to understand and experiment with architectural variants

via “transformer block-level routing and sequencing”

Unique: Implements block-level routing with per-block peer selection and failover, rather than static routing plans. RemoteSequenceManager dynamically constructs routing paths based on current peer availability and load, enabling adaptive inference across heterogeneous networks.

vs others: Provides dynamic block-level routing, whereas static inference pipelines require pre-configured peer assignments; Petals adapts routing to current network conditions.