1

Jet-Long is a tuning-free method for extending the context length of large language models. It combines a local RoPE-faithful window with a dynamically rescaled long-range window and an efficient attention implementation, preserving short-context behavior while supporting longer inputs without retraining.

Zeppelin balances variable-length workloads in data-parallel large model training. It combines hierarchical sequence partitioning, an attention engine with different parallel strategies, communication routing, and sequence-layout remapping to reduce communication overhead and balance computation.

FlexRL scales reinforcement learning for vision-language models by addressing multimodal data handling and workload imbalance. ShadowLoader distributes visual decoding and preprocessing to workers, while FlexUlysses adaptively shards sequences to balance computation and memory across GPUs.

PackMamba improves the efficiency of training Mamba models on variable-length sequences. It packs sequences and modifies the convolution and selective-scan operators to preserve sequence boundaries, reducing padding overhead while maintaining parallel execution.

Existing deep learning cluster schedulers make their scheduling decisions on partial job information, such as resource utilization …

Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to …