Entity · model

Llama-3.2-1B-Instruct

modelactiveprovisionalllama-3-2-1b-instruct-4631b55e·1 events·first seen 22h ago

Aliases: Llama-3.2-1B-Instruct

Co-occurring entities

hh-rlhf OR Else: A Differentiable Trust Region for Policy Optimization Output Reset Meta Anthropic

More like this (12)

Llama 3.3 70B Instruct Llama-3 Llama3-8B-Instruct Llama3-8B Llama 3.2 90B Vision-Instruct Llama-3.1-8B Llama 3.2 Llama 1B Llama 3.2 11B Vision Llama 3 Llama 3.2 90B Vision TinyLlama-1.1B

Recent events (1)

4arXiv · cs.AI·22h ago·source ↗

OR Else: Output Reset as a smooth trust-region alternative to clipping in PPO and GRPO for LLM post-training

This arXiv paper proposes Output Reset (OR), a smooth one-sided saturation rule to replace the clipped surrogate objective in PPO and GRPO during LLM post-training. Experiments on Llama-3.2-1B-Instruct with the Anthropic hh-rlhf dataset show PPO-OR achieves a 0.305 higher mean reward-model score than PPO-clip under GAE, while GRPO-OR shows reduced variance but no reward gain at group size G=2. The work identifies a meaningful behavioral difference between the two optimization regimes but leaves open whether larger group sizes change GRPO-OR's effectiveness.

Evaluation and Benchmarking Alignment and RLHF Llama-3.2-1B-Instruct hh-rlhf OR Else: A Differentiable Trust Region for Policy Optimization +3 more