Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

Best practices & guides on how to write distributed pytorch training code

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

Repo for Qwen Image Finetune

Find slow PyTorch training bottlenecks: DataLoader stalls, low GPU utilization, rank stragglers, memory creep, and run regressions.

A PyTorch native library for training speculative decoding models

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Write once, run anywhere; ezpz 🍋

A comprehensive hands-on guide to building production-grade distributed applications with Ray - from distributed training and multimodal data processing to inference and reinforcement learning.

META LLAMA3 GENAI Real World UseCases End To End Implementation Guide

Research platform for model training, evaluation, and experimentation across architectures, benchmarks, and recipes.

Fast and easy distributed model training examples.

Billus 大模型技能库：面向 LLM、VL、多模态与图像生成模型的训练、调参、剪枝、量化与工程化技能库

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

A minimal, hackable pre-training stack for GPT-style language models

Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem

A minimal, hackable pre-training stack for GPT-style language models

Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP

Framework, Model & Kernel Optimizations for Distributed Deep Learning - Data Hack Summit

Implementations of some popular approaches for efficient deep learning training and inference