Rethinking the Trust Region in LLM Reinforcement Learning

Rethinking Trust Regions in Large Language Model Reinforcement Learning

Recent research highlights significant limitations in the Proximal Policy Optimization (PPO) algorithm for fine-tuning Large Language Models (LLMs) in reinforcement learning (RL) frameworks. The study proposes Divergence Proximal Policy Optimization (DPPO) to enhance training stability and efficiency for LLMs.

PPO’s ratio clipping based on the probability of sampled tokens is inadequate for LLMs with large vocabularies, leading to problematic learning dynamics. Low-probability tokens face excessive penalties, and updates to high-probability tokens are insufficiently constrained, creating instability in training.

Proposed Solution: Divergence Proximal Policy Optimization

DPPO replaces the heuristic clipping approach with constraints derived from direct estimates of policy divergence, such as Total Variation or Kullback-Leibler (KL) divergence, aiming to provide a more accurate representation of policy updates.

DPPO also incorporates efficient Binary and Top-K approximations to capture essential divergence information while maintaining low computational overhead, ensuring practicality for large-scale applications.

Empirical Evaluations

Empirical evaluations demonstrate that DPPO consistently outperforms existing methods in training stability and efficiency, significantly enhancing the robustness of reinforcement learning applications for LLMs.