DeepSpeed Description
DeepSpeed is a deep learning optimization library that is open source for PyTorch. It is designed to reduce memory and computing power, and to train large distributed model with better parallelism using existing computer hardware. DeepSpeed is optimized to provide high throughput and low latency training.
DeepSpeed can train DL-models with more than 100 billion parameters using the current generation GPU clusters. It can also train as many as 13 billion parameters on a single GPU.
DeepSpeed, developed by Microsoft, aims to provide distributed training for large models. It's built using PyTorch which is a data parallelism specialist.