We are working hard to bring computation power together to support the open models ecosystem, while developing decentralized training algorithms for heterogeneous GPU hardwares connected over slow (1Gbps) internet links.

We are building state-of-the-art models that are much larger, that can outperforms many 100B+ parameter models on classification benchmarks. Training foundation models, such as GPT can be extremely expensive, often involving tens of thousands of GPUs running continually for months.

These models are typically trained in specialized clusters featuring fast, homogeneous interconnects and using carefully designed software systems that support both data parallelism and model/pipeline parallelism.