Abstract
This survey reviews systems for efficient large language model training on distributed infrastructure. It covers accelerators, networks, storage, scheduling, parallelism, computation, communication, memory optimization, and reliability during long training runs.
Type
Publication
arXiv preprint arXiv:2407.20018
