{"id":22788,"date":"2010-06-26T17:12:06","date_gmt":"2010-06-26T17:12:06","guid":{"rendered":"https:\/\/scannn.com\/github-nvidia-nemo-labs-molt-%c2%b7-github\/"},"modified":"2010-06-26T17:12:06","modified_gmt":"2010-06-26T17:12:06","slug":"github-nvidia-nemo-labs-molt-%c2%b7-github","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/github-nvidia-nemo-labs-molt-%c2%b7-github\/","title":{"rendered":"GitHub - NVIDIA-NeMo\/labs-molt \u00b7 GitHub"},"content":{"rendered":"\n
\n

Molt is agentic-first<\/strong> and PyTorch-native<\/strong>. The agent is the program;
\nthe trainer is a single actor; reward is any Python you write inside an Env<\/code>
\nor ChatAgent<\/code> \u2014 graders, multi-turn tools, VLM environments, LLM-as-judge.
\nThree components carry the rest \u2014 Ray<\/strong> for placement and async queues,
\nvLLM<\/strong> for rollout, NVIDIA AutoModel + FSDP2<\/strong> for training in pure
\nPyTorch. That is the whole stack: ~9.2K lines of RL code that scale to
\n1T-class MoE<\/strong> on vLLM with TP \/ EP \/ CP \u2014 think DeepSeek-V3 at
\n--fsdp.ep_size 256<\/code>, Adam CPU offload for the largest actors. One agent
\nAPI, one trainable actor, clean enough to read end-to-end.<\/p>\n

Three boxes. One async loop.<\/p>\n

\n <\/a>\n<\/p>\n

Ray<\/strong> owns placement and the async queue between the three boxes \u2014 that
\nis the entire runtime. The contract is token-first<\/strong>: token ids,
\nlogprobs, action ranges, rewards, and multimodal tensors stay aligned from
\nrollout to training. Anything you can compute in Python is a valid reward,
\nincluding LLM-as-judge calls back through the same vLLM engines that drive
\nrollout.<\/p>\n

<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
What you get<\/th>\nWhy it matters for research<\/th>\n<\/tr>\n<\/thead>\n
Agentic-first<\/strong><\/td>\nOne Gymnasium-aligned API \u2014 Env.step()<\/code> or ChatAgent.run()<\/code> \u2014 covers graders, multi-turn tools, VLM environments, and OpenAI\/Anthropic-compatible servers<\/td>\nThe agent is<\/em> the program \u2014 iterate on environments in plain Python, the trainer stays untouched<\/td>\n<\/tr>\n
Fully-async runtime<\/strong><\/td>\nRay placement, async rollout queues, vLLM engines, partial rollout, weight sync<\/td>\nRollout, training, and weight sync overlap \u2014 a DeepSeek-V3-class actor stays fed without bespoke infra<\/td>\n<\/tr>\n
PyTorch-native, AutoModel-first<\/strong><\/td>\nFSDP2 + NVIDIA AutoModel, pure PyTorch end-to-end<\/td>\nHack the model in the language you already write; no backend ceremony<\/td>\n<\/tr>\n
Single-actor simplicity<\/strong><\/td>\nOne actor, optional KL reference \u2014 the whole RL graph fits on a page<\/td>\nEvery gradient is explicit; every loss term is one file away<\/td>\n<\/tr>\n
Frontier-scale MoE<\/strong><\/td>\nAutoModel + FSDP2 + TP \/ EP \/ CP + Adam CPU offload, MoE-native \u2014 e.g. DeepSeek-V3 with --fsdp.ep_size 256<\/code><\/td>\nThe same script that trains 8B scales to 1T-class MoE \u2014 no rewrite between scales<\/td>\n<\/tr>\n
Token-first contract<\/strong><\/td>\nAligned token ids, logprobs, action ranges, rewards, multimodal tensors<\/td>\nMulti-turn, VLM, and tool-call traces share one format end-to-end<\/td>\n<\/tr>\n
Small, hackable surface<\/strong><\/td>\n~9.2K LOC of RL code across 3 thin layers<\/td>\nFork one layer without touching the others \u2014 read it in an afternoon<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/markdown-accessiblity-table><\/p>\n

The RL ecosystem optimizes for breadth. Molt optimizes for
\nagentic research velocity at scale<\/strong> \u2014 the smallest PyTorch-native stack
\nthat still drives fully-async agentic RL at frontier MoE scale on vLLM.<\/p>\n

<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Molt<\/strong><\/th>\nOpenRLHF<\/th>\nverl<\/th>\nslime<\/th>\n<\/tr>\n<\/thead>\n
Training backend<\/td>\nPyTorch \/ FSDP2 + NVIDIA AutoModel<\/strong><\/td>\nDeepSpeed ZeRO-3<\/td>\nFSDP \/ FSDP2 \/ Megatron<\/td>\nMegatron (FSDP exp.)<\/td>\n<\/tr>\n
Rollout engine<\/td>\nvLLM (Ray)<\/td>\nvLLM (Ray)<\/td>\nvLLM \/ SGLang \/ TRT-LLM<\/td>\nSGLang only<\/td>\n<\/tr>\n
RL topology<\/td>\nactor (+ optional PPO critic)<\/strong><\/td>\nactor + critic + RM<\/td>\nactor + critic + RM<\/td>\nactor + critic + RM<\/td>\n<\/tr>\n
Reward source<\/td>\nagent Python<\/strong><\/td>\nagent \/ endpoint \/ RM<\/td>\nagent \/ RM \/ endpoint<\/td>\nrollout fn \/ RM<\/td>\n<\/tr>\n
Parallelism<\/td>\nTP \/ EP \/ CP<\/strong>, MoE-native<\/td>\nZeRO-3 \/ FSDP<\/td>\nTP \/ PP \/ EP \/ SP<\/td>\nTP \/ PP \/ DP \/ CP \/ EP<\/td>\n<\/tr>\n
Multimodal<\/td>\nVLM RL, multi-turn tool calls<\/td>\nVLM RL (v0.10+)<\/td>\nQwen2.5-VL, Kimi-VL<\/td>\ngeo3k VLM<\/td>\n<\/tr>\n
Config surface<\/td>\nCLI flags only<\/strong><\/td>\nCLI + scripts<\/td>\nHydra + YAML<\/td>\nCLI + YAML<\/td>\n<\/tr>\n
RL code size\u00b9<\/td>\n~9.2K LOC<\/strong><\/td>\n~7.2K<\/td>\n~62K<\/td>\n~25K<\/td>\n<\/tr>\n
Design center<\/td>\nagentic-first research<\/strong><\/td>\nRLHF coverage<\/td>\nproduction breadth<\/td>\nMegatron throughput<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/markdown-accessiblity-table><\/p>\n

One framework, one job.<\/strong> Molt is the smallest PyTorch-native
\nstack that takes an NVIDIA AutoModel from SFT to frontier-scale agentic
\nRL on vLLM. Read every line that touches your gradients, in plain PyTorch.<\/p>\n

\n

\u00b9 RL code = every Python file the framework’s RL path uses \u2014 online
\ntrainer, rollout, Ray orchestration, experience\/advantage\/reward\/KL\/loss,
\nactor\/critic\/RM inference, plus shared models, utils, parallelism, and
\nkernels the RL training command depends on. Excludes pure SFT, DPO\/KTO\/IPO
\ntrainers, reward-model training<\/strong>, distillation, vendored third-party
\ncode, tests, examples, scripts, and docs. Counts code lines only (blank
\nand comment-only lines excluded). Measured by tracing the import graph
\nfrom each RL entry point (molt.cli.train_rl_ray<\/code>,
\nopenrlhf.cli.train_ppo_ray<\/code>, verl.trainer.main_ppo<\/code>); slime loads its
\nMegatron\/SGLang backends lazily, so its core slime\/<\/code> package plus its
\nslime_plugins\/<\/code> model-zoo (+~4.7K \u2014 the in-repo model code its RL path
\nuses, counted on the same basis as molt’s models\/<\/code>) are counted, minus
\nSFT\/distillation. Molt measured 2026-07-20 on this repo; the others
\nmeasured 2026-06-16 at each repo’s then-latest main HEAD
\n(verl 86e8123<\/code>, slime 243773c<\/code>, OpenRLHF b3d2927<\/code>).<\/p>\n<\/blockquote>\n

<\/p>\n\n\n\n\n\n\n\n\n\n\n
Area<\/th>\nSupport<\/th>\n<\/tr>\n<\/thead>\n
SFT<\/td>\nmolt.cli.train_sft<\/code><\/td>\n<\/tr>\n
RL<\/td>\nvLLM-backed online RL via molt.cli.train_rl_ray<\/code><\/td>\n<\/tr>\n
Runtime<\/td>\nRay placement, async rollout queues, vLLM engines, partial rollout sync<\/td>\n<\/tr>\n
Model scale<\/td>\nAutoModel + FSDP2 with TP \/ EP \/ CP, MoE-native \u2014 e.g. DeepSeek-V3 at --fsdp.ep_size 256<\/code><\/td>\n<\/tr>\n
Model backend<\/td>\nNVIDIA AutoModel is the primary path<\/strong> \u2014 native CP \/ EP \/ TP, custom MoE+EP parallelizer, TE fused attention; everything model-side aligns with AutoModel’s own recipes. The HF transformers path is a non-preferred fallback<\/strong> (AutoModel drops to it only when a model has no native class) supporting text + flash_attention_2 + packing only \u2014 no CP \/ EP \/ TP<\/strong><\/td>\n<\/tr>\n
Optimizer<\/td>\nadam<\/code> (default), with CPU offload for the largest actors (--fsdp.offload optimizer<\/code>). muon<\/code> (Newton\u2013Schulz via Dion: Muon for 2D weights and grouped MoE experts, AdamW for embeddings \/ head \/ norms) is experimental<\/strong> \u2014 runs distributed (FSDP \/ EP) but has shown no consistent win over adam<\/code> yet, which stays the recommended default<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/markdown-accessiblity-table><\/p>\n

<\/p>\n\n\n\n\n\n\n\n\n
Area<\/th>\nSupport<\/th>\n<\/tr>\n<\/thead>\n
Agent interface<\/td>\n--train.agent_path<\/code> with Env<\/code> or ChatAgent<\/code> subclass + an AgentRunner<\/code><\/td>\n<\/tr>\n
Reward source<\/td>\nResult(reward=...)<\/code> returned from Env.step<\/code> or ChatAgent.run<\/code><\/td>\n<\/tr>\n
Modalities<\/td>\nText and VLM prompts, including image payloads<\/td>\n<\/tr>\n
Chat templates<\/td>\nAssistant spans (SFT loss mask + multi-turn rollout stitching) are derived from the model’s own chat template \u2014 no hard-coded markers. Verified on ChatML (Qwen3.x, Nemotron omni3), Kimi-K2.6, GLM, Gemma and DeepSeek<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/markdown-accessiblity-table><\/p>\n

<\/p>\n\n\n\n\n\n\n\n\n\n
Area<\/th>\nSupport<\/th>\n<\/tr>\n<\/thead>\n
Estimators<\/td>\nreinforce<\/code>, reinforce_baseline<\/code>, rloo<\/code>, grpo<\/code>, dr_grpo<\/code>, gae<\/code> (PPO), on_policy_distill<\/code><\/td>\n<\/tr>\n
PPO critic<\/td>\n--algo.advantage.estimator gae<\/code> adds a value model: its own Ray group (CriticModelActor<\/code>), colocated on the actor’s GPUs by default or disaggregatable, GAE advantages (--algo.advantage.lam<\/code>) + clipped value loss (--critic.value_clip<\/code>), own optimizer\/LR (--critic.adam.lr<\/code>) and resumable _critic<\/code> checkpoint. Built on NeMoAutoModelForCausalLM<\/code> + a scalar value head, so it keeps the native TP \/ EP \/ CP path<\/td>\n<\/tr>\n
Distillation<\/td>\nOn-policy distillation \u2014 per-token reverse KL to a frozen teacher, via --algo.advantage.estimator on_policy_distill<\/code> + --ref.model_name_or_path<\/code><\/td>\n<\/tr>\n
IS correction<\/td>\nTrain\/rollout logprob-mismatch correction for off-policy \/ async rollout: is_correction_level {off,token,seq,geo}<\/code> \u00d7 is_correction_mode {mask,clip,trunc}<\/code> (covers TIS, IcePop, seq-mask-tis; see IS correction<\/em> below)<\/td>\n<\/tr>\n
KL<\/td>\nOptional reference workers when --algo.kl.init_coef > 0<\/code> (the reference doubles as the distillation teacher)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/markdown-accessiblity-table><\/p>\n

<\/p>\n\n\n\n\n\n\n
Area<\/th>\nSupport<\/th>\n<\/tr>\n<\/thead>\n
Router replay (R3)<\/td>\n--train.routing_replay<\/code> \u2014 vLLM’s per-token top-k selection replayed in the training forward; details in the MoE routing stability<\/em> section under Scaling Knobs<\/td>\n<\/tr>\n
Router freeze<\/td>\n--actor.freeze_moe_router<\/code> holds the gate\/router weights fixed so vLLM and the actor keep routing tokens to the same experts. Stabilizes MoE RL \/ distillation and shrinks the same rollout-vs-train logprob gap the IS-correction filters address \u2014 a router that drifts between refits is a large source of that gap<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/markdown-accessiblity-table><\/p>\n

\n

IS correction (train\/rollout logprob mismatch)<\/h3>\n

<\/svg><\/a><\/div>\n

Async and partial rollout make the FSDP actor’s recomputed pi_train<\/code> diverge from
\nvLLM’s gen-time pi_rollout<\/code> (different kernels, plus a mid-request weight swap the
\nHTTP router can’t observe). Molt corrects the resulting off-policy update with the
\nper-token importance ratio pi_train \/ pi_rollout<\/code>, gated by two knobs:<\/p>\n