市值: $2.1169T -2.34%
成交额(24h): $82.6304B 15.78%
  • 市值: $2.1169T -2.34%
  • 成交额(24h): $82.6304B 15.78%
  • 恐惧与贪婪指数:
  • 市值: $2.1169T -2.34%
加密货币
话题
百科
资讯
加密话题
视频
热门新闻
加密货币
话题
百科
资讯
加密话题
视频
bitcoin
bitcoin

$61605.656648 USD

-3.04%

ethereum
ethereum

$1741.712974 USD

-3.34%

tether
tether

$0.999452 USD

0.06%

bnb
bnb

$569.840001 USD

-3.10%

usd-coin
usd-coin

$1.000291 USD

0.05%

xrp
xrp

$0.962638 USD

-1.69%

solana
solana

$72.392737 USD

-3.10%

tron
tron

$0.309467 USD

-1.00%

hyperliquid
hyperliquid

$65.780663 USD

-3.44%

zcash
zcash

$850.819478 USD

-9.16%

monero
monero

$479.362672 USD

-4.57%

dogecoin
dogecoin

$0.057347 USD

-3.31%

unus-sed-leo
unus-sed-leo

$9.228306 USD

-1.21%

chainlink
chainlink

$8.839656 USD

-3.37%

cardano
cardano

$0.149404 USD

-2.68%

加密货币新闻

Genarm:自回归奖励模型的奖励引导产生

2025/02/11 03:46

大型语言模型(LLM)必须与人类的偏好等人的喜好保持一致,但是传统的一致性方法需要昂贵的重新训练和与动态或冲突的偏好斗争。

Genarm:自回归奖励模型的奖励引导产生

Large language models (LLMs) must align with human preferences like helpfulness and harmlessness, but traditional alignment methods require costly retraining and struggle with dynamic or conflicting preferences. Test-time alignment approaches using reward models (RMs) avoid retraining but face inefficiencies due to reliance on trajectory-level rewards, which evaluate full responses rather than guiding token-by-token generation.

大型语言模型(LLM)必须与人类的偏好等人的喜好保持一致,但是传统的一致性方法需要昂贵的重新训练和与动态或冲突的偏好斗争。使用奖励模型(RMS)进行测试时间对齐方法避免进行重新训练,但由于依赖轨迹级别的奖励而面临效率低下的效率,该奖励评估了全部响应,而不是指导代币的一代。

Existing alignment techniques fall into two categories: training-time methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), which fine-tune LLMs on preference datasets but demand significant computational resources and lack flexibility for new preferences. Test-time methods use RMs to guide frozen LLMs but rely on trajectory-level RMs that assign a single reward to complete responses. This creates a mismatch during autoregressive generation, where next-token decisions require partial response evaluations. For instance, ARGS approximates token-level rewards by applying trajectory RMs to incomplete responses, leading to inaccuracies since these RMs are trained only on full responses. Other methods like Transfer-Q generate multiple full responses per token candidate, multiplying inference costs. These inefficiencies limit scalability and real-time adaptability.

现有的对齐技术分为两类:训练时间方法,例如从人类反馈中学习(RLHF)和直接偏好优化(DPO),这些方法在偏好数据集中微调LLM,但要求大量的计算资源,并且缺乏对新偏好的灵活性。测试时间方法使用RMS指导冷冻的LLM,但依赖于轨迹级别的RMS,这些RMS分配了单个奖励以完成响应。这在自回归生成期间造成了不匹配,下一步的决策需要部分响应评估。例如,通过将轨迹RMS应用于不完整的响应,导致不准确的ARG近似值级别的奖励,因为这些RMS仅在完整的响应中受过培训。其他方法(例如Transfer-Q)每个令牌候选者会产生多个完整响应,从而增加推理成本。这些效率低下限制了可伸缩性和实时适应性。

To address these issues, researchers from the University of Maryland, College Park and JPMorgan AI Research propose GenARM (Reward Guided Generation with Autoregressive Reward Model), a test-time alignment framework combining a novel autoregressive RM with guided decoding. The key innovation is the Autoregressive Reward Model, which decomposes trajectory-level rewards into token-level components. Instead of assigning a single reward to a full response, it predicts the reward for each token conditioned on prior tokens, enabling dense, step-by-step guidance, allowing rewards to directly influence each token choice without evaluating partial responses inaccurately.

为了解决这些问题,马里兰大学,大学公园和JPMorgan AI研究的研究人员提出了Genarm(奖励引导产生带有自回调奖励模型),这是一个测试时间对齐框架,结合了新颖的自动回归RM与指导解码。关键创新是自回归奖励模型,该模型将轨迹级别的奖励分解为令牌级组件。它没有将单一的奖励分配给全面响应,而是预测在代币之前以代币为条件的每个令牌的奖励,实现了密集的,逐步的指导,从而允许奖励直接影响每个代币选择,而不会不准确地评估部分响应。

During generation, GenARM integrates the autoregressive RM’s token-level rewards with the base LLM’s logits. The next token is sampled from a modified distribution. Unlike prior methods, this requires only one forward pass through the base and reward models per token, avoiding costly candidate expansions.

在世代相传的过程中,Genarm将自动回归RM的令牌级别的奖励与Base LLM的ligits集成在一起。接下来的令牌是从修改的分布中采样的。与先前的方法不同,这仅需要一个前向通过的基础和每个令牌奖励模型,避免了昂贵的候选扩展。

Experiments demonstrate GenARM’s advantages across three scenarios:

实验证明了Genarm在三种情况下的优势:

1. General Human Preference Alignment: On the HH-RLHF dataset, GenARM outperforms test-time baselines like ARGS and Transfer-Q in helpfulness and harmlessness, matching the performance of training-time methods like DPO based on evaluations using GPT-4.

1。一般的人类偏好一致性:在HH-RLHF数据集上,Genarm在有益和无害性方面优于测试时间基准,例如ARGS和Transfer-Q,与使用GPT-4的评估相匹配DPO(例如DPO)的性能。

2. Weak-to-Strong Guidance: A 7B autoregressive RM effectively guides larger base models (13B, 70B) without fine-tuning them. It surpasses DPO at the 7B scale and nearly matches DPO at the 13B scale. At the 70B scale, GenARM recovers more than 70% of the performance gap in both raw and LC win rates between Tulu2-70B and Tulu2-DPO-70B, all without the need to train the 70B LLM, demonstrating that smaller RMs can steer larger LLMs efficiently.

2。弱到较强的指导:7B自回归RM有效地指导较大的基本模型(13b,70b),而无需微调它们。它以7B量表超过DPO,几乎与DPO匹配13B量表。在70B量表上,Genarm在TULU2-70B和TULU2-DPO-70B之间在RAW和LC赢得率中恢复了70%以上的性能差距,而无需训练70B LLM LLM有效。

3. Multi-Objective Alignment: GenARM balances conflicting preferences (e.g., helpfulness vs. harmlessness) by combining rewards from multiple autoregressive RMs. On the PKU-SafeRLHF-10K dataset, it achieves a Pareto frontier superior to Rewarded Soups and matches multi-objective RL without retraining.

3。多目标对准:Genarm平衡了相互冲突的偏好(例如,有助于无害的偏好),通过结合来自多个自回归RMS的奖励。在PKU-SAFERLHF-10K-10K数据集上,它在不奖励的汤中获得了优越的帕累托边境,并与多目标RL进行了匹配,而无需重新培训。

The autoregressive RM’s design ensures it can express any reward function achievable by traditional RMs within the KL-regularized reinforcement learning framework. This theoretical guarantee, combined with token-level factorization, makes GenARM both expressive and efficient. Unlike trajectory-level RMs, which struggle with partial contexts, autoregressive RMs provide accurate, incremental feedback, preventing reward hacking or incoherent outputs during long generations.

自回归RM的设计可确保它可以表达在KL规范化的增强学习框架内通过传统RMS获得的任何奖励功能。这种理论保证与令牌级别的分解相结合,使Genarm既表现力又有效。与轨迹级别的RMS不同,在部分环境中挣扎,自回归的RMS提供了准确的,增量的反馈,可以在长期内预防奖励黑客入侵或不一致的输出。

In summary, GenARM bridges the gap between training-time and test-time alignment by introducing autoregressive reward models that enable precise, token-level guidance. It eliminates the need for costly LLM retraining, supports dynamic adaptation to diverse preferences, and efficiently scales to larger models. By addressing the inefficiencies of trajectory-level rewards and enabling weak-to-strong guidance, GenARM offers a practical solution for aligning LLMs in resource-constrained scenarios. Future work could extend this approach to tasks like mathematical reasoning or code generation, where token-level rewards might enhance performance without additional fine-tuning.

总而言之,Genarm通过引入自动回归奖励模型来确切的,令牌级别的指导,从而弥合了训练时间和测试时间对齐之间的差距。它消除了对昂贵的LLM再培训的需求,支持对各种偏好的动态适应,并有效地扩展到较大的模型。通过解决轨迹级别的奖励的低效率并实现弱到强调的指导,Genarm提供了一种实用的解决方案,可以使LLM在资源约束的情况下对齐。未来的工作可能会将这种方法扩展到数学推理或代码生成等任务,令牌级别的奖励可以在不进行其他微调的情况下增强性能。

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 75k+ ML SubReddit.

查看纸。这项研究的所有信用都归该项目的研究人员。另外,不要忘记在Twitter上关注我们,并加入我们的电报频道和LinkedIn组。不要忘记加入我们的75k+ ml subreddit。

? Recommended Open-Source AI Platform: ‘IntellAgent is a An Open-Source Multi-Agent Framework to Evaluate Complex Conversational AI System’ (Promoted)

?推荐开源AI平台:“ Intellagent是一个开源的多代理框架,可评估复杂的对话AI系统”(已晋升)

原文来源:marktechpost

免责声明:info@kdj.com

所提供的信息并非交易建议。根据本文提供的信息进行的任何投资,kdj.com不承担任何责任。加密货币具有高波动性,强烈建议您深入研究后,谨慎投资!

如您认为本网站上使用的内容侵犯了您的版权,请立即联系我们(info@kdj.com),我们将及时删除。

2026年11月20日 发表的其他文章