{"id":1484,"date":"2026-07-30T12:47:51","date_gmt":"2026-07-30T04:47:51","guid":{"rendered":"https:\/\/www.ndnlab.com\/?p=1484"},"modified":"2026-07-30T12:47:53","modified_gmt":"2026-07-30T04:47:53","slug":"multi2-hierarchical-multi-agent-decision-making-with-llm-based-agents-in-interactive-environments","status":"publish","type":"post","link":"https:\/\/www.ndnlab.com\/?p=1484","title":{"rendered":"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">1. \u6458\u8981\uff08Abstract\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u672c\u6587\u7814\u7a76\u7684\u662f\u957f\u7a0b\u4ea4\u4e92\u73af\u5883\u4e2d\u5927\u6a21\u578b\u667a\u80fd\u4f53\u7684\u51b3\u7b56\u7a33\u5b9a\u6027\u95ee\u9898\u3002\u73b0\u6709 LLM-based agents \u5df2\u7ecf\u5177\u5907\u8f83\u5f3a\u7684\u4e0a\u4e0b\u6587\u7406\u89e3\u548c\u63a8\u7406\u80fd\u529b\uff0c\u4f46\u5728\u591a\u8f6e\u4ea4\u4e92\u4efb\u52a1\u4e2d\u4ecd\u7136\u5bb9\u6613\u51fa\u73b0\u76ee\u6807\u6f02\u79fb\u3002\u4e5f\u5c31\u662f\u8bf4\uff0c\u6a21\u578b\u4e00\u5f00\u59cb\u80fd\u591f\u7406\u89e3\u4efb\u52a1\u76ee\u6807\uff0c\u4f46\u968f\u7740\u4ea4\u4e92\u8f6e\u6b21\u589e\u52a0\uff0c\u8ba1\u5212\u548c\u52a8\u4f5c\u4f1a\u9010\u6e10\u504f\u79bb\u539f\u59cb\u610f\u56fe\uff0c\u6700\u7ec8\u5bfc\u81f4\u65e0\u6548\u52a8\u4f5c\u3001\u5faa\u73af\u884c\u4e3a\u6216\u4efb\u52a1\u5931\u8d25\u3002\u9488\u5bf9\u8fd9\u4e00\u95ee\u9898\uff0c\u8bba\u6587\u63d0\u51fa\u4e86 Multi\u00b2\uff0c\u4e00\u4e2a\u5c42\u7ea7\u5316\u591a\u667a\u80fd\u4f53\u51b3\u7b56\u6846\u67b6\uff0c\u7528\u4e8e\u63d0\u5347\u667a\u80fd\u4f53\u5728\u957f\u7a0b\u4ea4\u4e92\u73af\u5883\u4e2d\u7684\u7a33\u5b9a\u6027\u3001\u9c81\u68d2\u6027\u548c token \u4f7f\u7528\u6548\u7387\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Multi\u00b2 \u7684\u6838\u5fc3\u601d\u60f3\u662f\u5c06\u667a\u80fd\u4f53\u884c\u4e3a\u62c6\u5206\u4e3a\u4e24\u4e2a\u4e92\u8865\u89d2\u8272\u3002\u9ad8\u5c42\u667a\u80fd\u4f53 System 1 \u8d1f\u8d23\u6839\u636e\u4efb\u52a1\u76ee\u6807\u548c\u73af\u5883\u89c2\u5bdf\u751f\u6210\u4e0a\u4e0b\u6587\u76f8\u5173\u7684\u5b50\u76ee\u6807\uff0c\u4e3b\u8981\u627f\u62c5\u89c4\u5212\u529f\u80fd\u3002\u4f4e\u5c42\u667a\u80fd\u4f53 System 2 \u8d1f\u8d23\u6839\u636e\u5f53\u524d\u5b50\u76ee\u6807\u6267\u884c\u5177\u4f53\u539f\u5b50\u52a8\u4f5c\uff0c\u4e3b\u8981\u627f\u62c5\u73af\u5883\u4ea4\u4e92\u4e0e\u52a8\u4f5c\u63a7\u5236\u529f\u80fd\u3002System 1 \u4f7f\u7528\u76d1\u7763\u5fae\u8c03\u8bad\u7ec3\uff0c\u4ee5\u4fdd\u8bc1\u5b50\u76ee\u6807\u751f\u6210\u7684\u7a33\u5b9a\u6027\u548c\u4e00\u81f4\u6027\uff1bSystem 2 \u5219\u91c7\u7528 offline-to-online reinforcement learning\uff0c\u4f7f\u6267\u884c\u7b56\u7565\u5148\u4ece\u79bb\u7ebf\u6570\u636e\u4e2d\u7a33\u5b9a\u521d\u59cb\u5316\uff0c\u518d\u901a\u8fc7\u5728\u7ebf\u4ea4\u4e92\u6301\u7eed\u6539\u8fdb\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5b9e\u9a8c\u7ed3\u679c\u8868\u660e\uff0cMulti\u00b2 \u5728 ScienceWorld\u3001ALFWorld \u548c TextCraft \u7b49\u591a\u4e2a\u4ea4\u4e92\u73af\u5883\u4e2d\u6574\u4f53\u4f18\u4e8e ReAct\u3001Reflexion\u3001ADaPT\u3001GRPO \u548c Glider \u7b49\u5f3a\u57fa\u7ebf\u65b9\u6cd5\u3002\u76f8\u6bd4\u53ea\u4f9d\u8d56 prompting \u6216\u666e\u901a\u5c42\u7ea7\u89c4\u5212\u7684\u65b9\u6cd5\uff0cMulti\u00b2 \u4e0d\u4ec5\u4efb\u52a1\u5b8c\u6210\u7387\u66f4\u9ad8\uff0c\u800c\u4e14\u5728\u957f horizon\u3001\u4efb\u52a1\u96be\u5ea6\u589e\u52a0\u548c OOD \u5206\u5e03\u4e0b\u8868\u73b0\u66f4\u7a33\u3002\u8bba\u6587\u8fd8\u53d1\u5e03\u4e86\u4e09\u4e2a\u5c42\u7ea7\u5316 benchmark datasets\uff0c\u7528\u4e8e\u8bad\u7ec3\u548c\u8bc4\u4f30 LLM \u667a\u80fd\u4f53\u7684\u5c42\u7ea7\u51b3\u7b56\u80fd\u529b\uff0c\u5f25\u8865\u4e86\u8be5\u65b9\u5411\u6570\u636e\u8d44\u6e90\u4e0d\u8db3\u7684\u95ee\u9898\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"922\" height=\"742\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-12.png\"  class=\"wp-image-1485\" style=\"aspect-ratio:1.242603550295858;width:441px;height:auto\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-12.png 922w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-12-300x241.png 300w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-12-768x618.png 768w\" sizes=\"auto, (max-width: 922px) 100vw, 922px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">2. \u7814\u7a76\u80cc\u666f\u4e0e\u95ee\u9898\u52a8\u673a\uff08Introduction\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u6784\u5efa\u80fd\u591f\u5728\u52a8\u6001\u73af\u5883\u4e2d\u6301\u7eed\u4ea4\u4e92\u3001\u89c4\u5212\u548c\u884c\u52a8\u7684\u667a\u80fd\u4f53\uff0c\u662f\u5f53\u524d agentic AI \u7684\u91cd\u8981\u76ee\u6807\u3002\u4e0e\u4e00\u6b21\u6027\u95ee\u7b54\u4e0d\u540c\uff0c\u4ea4\u4e92\u73af\u5883\u4e2d\u7684\u667a\u80fd\u4f53\u6bcf\u6267\u884c\u4e00\u4e2a\u52a8\u4f5c\uff0c\u73af\u5883\u72b6\u6001\u90fd\u4f1a\u53d1\u751f\u53d8\u5316\uff0c\u540e\u7eed\u51b3\u7b56\u5fc5\u987b\u57fa\u4e8e\u65b0\u7684\u89c2\u5bdf\u7ee7\u7eed\u8fdb\u884c\u3002\u56e0\u6b64\uff0c\u8fd9\u7c7b\u4efb\u52a1\u5929\u7136\u5177\u6709\u591a\u8f6e\u3001\u957f\u7a0b\u548c\u95ed\u73af\u53cd\u9988\u7684\u7279\u70b9\uff0c\u5bf9\u667a\u80fd\u4f53\u7684\u76ee\u6807\u4fdd\u6301\u80fd\u529b\u548c\u6267\u884c\u7a33\u5b9a\u6027\u63d0\u51fa\u4e86\u66f4\u9ad8\u8981\u6c42\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u76ee\u524d\u5f88\u591a LLM agent \u5728\u77ed\u4efb\u52a1\u4e2d\u8868\u73b0\u4e0d\u9519\uff0c\u4f46\u5728\u957f\u7a0b\u4efb\u52a1\u4e2d\u4ecd\u7136\u8106\u5f31\u3002\u4e00\u4e2a\u5178\u578b\u95ee\u9898\u662f objective drift\uff0c\u5373\u667a\u80fd\u4f53\u5728\u591a\u8f6e\u4ea4\u4e92\u8fc7\u7a0b\u4e2d\u9010\u6e10\u504f\u79bb\u539f\u59cb\u76ee\u6807\u3002\u8bba\u6587\u6307\u51fa\uff0c\u8fd9\u5e76\u4e0d\u53ea\u662f\u5355\u7eaf\u7684\u89c4\u5212\u5931\u8d25\uff0c\u800c\u662f\u4e00\u4e2a\u4ea4\u4e92\u5c42\u9762\u7684\u7d2f\u79ef\u8bef\u5dee\u95ee\u9898\u3002\u524d\u9762\u67d0\u4e00\u6b65\u7684\u5c0f\u9519\u8bef\u53ef\u80fd\u5bfc\u81f4\u72b6\u6001\u504f\u79fb\uff0c\u540e\u7eed\u6a21\u578b\u53c8\u57fa\u4e8e\u9519\u8bef\u72b6\u6001\u7ee7\u7eed\u63a8\u7406\uff0c\u6700\u7ec8\u4f7f\u6574\u4e2a\u8f68\u8ff9\u8d8a\u8d70\u8d8a\u504f\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5df2\u6709\u65b9\u6cd5\u901a\u5e38\u4ece\u4e24\u4e2a\u65b9\u5411\u5c1d\u8bd5\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\u3002\u4e00\u7c7b\u65b9\u6cd5\u5982 ReAct \u548c Reflexion \u4f9d\u8d56 prompt \u548c\u957f\u5386\u53f2\u4e0a\u4e0b\u6587\uff0c\u8ba9\u6a21\u578b\u901a\u8fc7\u63a8\u7406\u548c\u81ea\u6211\u53cd\u601d\u7ef4\u6301\u76ee\u6807\u3002\u4f46\u8fd9\u7c7b\u65b9\u6cd5 token \u6d88\u8017\u8f83\u5927\uff0c\u4e14\u957f\u5386\u53f2\u672c\u8eab\u5e76\u4e0d\u80fd\u4fdd\u8bc1\u52a8\u4f5c\u6267\u884c\u6b63\u786e\u3002\u53e6\u4e00\u7c7b\u65b9\u6cd5\u91c7\u7528\u5c42\u7ea7\u89c4\u5212\uff0c\u4f8b\u5982 ADaPT \u548c Glider\uff0c\u5c06\u590d\u6742\u4efb\u52a1\u62c6\u6210\u5b50\u76ee\u6807\uff0c\u4ece\u800c\u964d\u4f4e\u89c4\u5212\u96be\u5ea6\u3002\u4e0d\u8fc7\uff0c\u8bba\u6587\u8ba4\u4e3a\u4ec5\u6709\u4efb\u52a1\u5206\u89e3\u8fd8\u4e0d\u591f\uff0c\u56e0\u4e3a\u6267\u884c\u5668\u5982\u679c\u6ca1\u6709\u7ecf\u8fc7\u4e13\u95e8\u8bad\u7ec3\uff0c\u4ecd\u7136\u65e0\u6cd5\u6709\u6548\u4fee\u6b63\u957f\u7a0b\u4ea4\u4e92\u4e2d\u7684\u7d2f\u79ef\u9519\u8bef\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u56e0\u6b64\uff0c\u672c\u6587\u771f\u6b63\u6293\u4f4f\u7684\u95ee\u9898\u662f\uff0c\u957f\u7a0b\u4ea4\u4e92\u4e0d\u662f\u53ea\u9700\u8981\u66f4\u597d\u7684 planning\uff0c\u4e5f\u9700\u8981\u66f4\u7a33\u7684 execution\u3002\u4e00\u4e2a\u53ef\u9760\u7684 LLM agent \u5e94\u8be5\u540c\u65f6\u5177\u5907\u4e09\u70b9\u80fd\u529b\u3002\u9996\u5148\u662f\u660e\u786e\u7684\u4efb\u52a1\u5206\u89e3\uff0c\u7528\u6765\u7ef4\u6301\u5168\u5c40\u610f\u56fe\u3002\u5176\u6b21\u662f\u80fd\u591f\u901a\u8fc7\u73af\u5883\u53cd\u9988\u4e0d\u65ad\u6539\u5584\u7684\u52a8\u4f5c\u6267\u884c\u80fd\u529b\u3002\u6700\u540e\u662f\u8f83\u9ad8\u7684 token efficiency\uff0c\u907f\u514d\u6bcf\u4e00\u6b65\u90fd\u4f9d\u8d56\u8d8a\u6765\u8d8a\u957f\u7684\u4e0a\u4e0b\u6587\u3002Multi\u00b2 \u6b63\u662f\u56f4\u7ed5\u8fd9\u4e09\u4e2a\u9700\u6c42\u8bbe\u8ba1\u7684\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">3. \u6846\u67b6\u4e0e\u6574\u4f53\u8bbe\u8ba1\uff08System Overview\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Multi\u00b2 \u91c7\u7528\u5c42\u7ea7\u5316\u591a\u667a\u80fd\u4f53\u7ed3\u6784\uff0c\u5c06\u539f\u672c\u7531\u5355\u4e2a agent \u5b8c\u6210\u7684\u957f\u671f\u51b3\u7b56\u4efb\u52a1\u62c6\u89e3\u4e3a\u4e24\u4e2a\u89d2\u8272\u5206\u5de5\u660e\u786e\u7684\u5b50\u7cfb\u7edf\u3002System 1 \u662f\u9ad8\u5c42\u89c4\u5212\u5668\uff0c\u8f93\u5165\u4efb\u52a1\u63cf\u8ff0\u548c\u5f53\u524d\u89c2\u5bdf\uff0c\u8f93\u51fa\u4e0b\u4e00\u9636\u6bb5\u9700\u8981\u5b8c\u6210\u7684\u5b50\u76ee\u6807\u3002System 2 \u662f\u4f4e\u5c42\u6267\u884c\u5668\uff0c\u8f93\u5165\u5f53\u524d\u89c2\u5bdf\u548c System 1 \u751f\u6210\u7684\u5b50\u76ee\u6807\uff0c\u8f93\u51fa\u5177\u4f53\u7684\u73af\u5883\u52a8\u4f5c\u3002\u5f53\u5f53\u524d\u5b50\u76ee\u6807\u5b8c\u6210\u6216\u7ec8\u6b62\u540e\uff0c\u7cfb\u7edf\u4f1a\u518d\u6b21\u8c03\u7528 System 1 \u751f\u6210\u65b0\u7684\u5b50\u76ee\u6807\uff0c\u5982\u6b64\u5faa\u73af\u76f4\u5230\u4efb\u52a1\u7ed3\u675f\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u79cd\u8bbe\u8ba1\u7684\u5173\u952e\u4e0d\u53ea\u662f\u628a\u4efb\u52a1\u62c6\u6210\u4e0a\u4e0b\u4e24\u5c42\uff0c\u800c\u662f\u8ba9\u4e24\u4e2a\u5c42\u7ea7\u5206\u522b\u5bf9\u5e94\u4e0d\u540c\u7684\u5b66\u4e60\u76ee\u6807\u3002System 1 \u8d1f\u8d23\u8bed\u4e49\u5c42\u9762\u7684\u4efb\u52a1\u7406\u89e3\u548c\u5b50\u76ee\u6807\u89c4\u5212\uff0c\u56e0\u6b64\u66f4\u9002\u5408\u7528 SFT \u5b66\u4e60\u7a33\u5b9a\u3001\u7ed3\u6784\u5316\u7684\u5b50\u76ee\u6807\u751f\u6210\u3002System 2 \u76f4\u63a5\u4e0e\u73af\u5883\u4ea4\u4e92\uff0c\u9700\u8981\u5904\u7406\u72b6\u6001\u53d8\u5316\u3001\u52a8\u4f5c\u7ea6\u675f\u548c\u6267\u884c\u9519\u8bef\uff0c\u56e0\u6b64\u66f4\u9002\u5408\u7528\u5f3a\u5316\u5b66\u4e60\u63d0\u5347\u52a8\u4f5c\u5c42\u9762\u7684\u9c81\u68d2\u6027\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8bba\u6587\u5c06\u6574\u4e2a\u6d41\u7a0b\u5206\u4e3a\u4e09\u4e2a\u9636\u6bb5\u3002\u7b2c\u4e00\u662f offline training\uff0c\u5206\u522b\u8bad\u7ec3 System 1 \u548c System 2\u3002\u7b2c\u4e8c\u662f online training\uff0c\u8ba9 System 2 \u5728\u73af\u5883\u4ea4\u4e92\u4e2d\u7ee7\u7eed\u81ea\u6211\u6539\u8fdb\u3002\u7b2c\u4e09\u662f execution\uff0c\u63a8\u7406\u65f6 System 1 \u6309\u9700\u751f\u6210\u5b50\u76ee\u6807\uff0cSystem 2 \u5728\u6bcf\u4e2a\u5b50\u76ee\u6807\u4e0b\u8fde\u7eed\u6267\u884c\u539f\u5b50\u52a8\u4f5c\u3002\u8fd9\u6837\u7684\u9009\u62e9\u6027\u8c03\u7528\u673a\u5236\u4e5f\u51cf\u5c11\u4e86\u4e0d\u5fc5\u8981\u7684\u957f\u4e0a\u4e0b\u6587\u63a8\u7406\uff0c\u56e0\u6b64\u80fd\u63d0\u5347 token efficiency\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u76f8\u6bd4 Glider \u8fd9\u7c7b\u5df2\u6709\u5c42\u7ea7\u65b9\u6cd5\uff0cMulti\u00b2 \u7684\u5dee\u5f02\u5728\u4e8e\u5b83\u66f4\u5f3a\u8c03 role specialization\u3002Glider \u867d\u7136\u4e5f\u6709 planner \u548c controller\uff0c\u4f46\u5728\u5728\u7ebf\u9002\u5e94\u65f6\u4e3b\u8981\u66f4\u65b0\u9ad8\u5c42\u89c4\u5212\u5668\uff0c\u4f4e\u5c42\u6267\u884c\u5668\u76f8\u5bf9\u56fa\u5b9a\uff0c\u800c\u4e14\u4e0d\u540c\u89d2\u8272\u4e4b\u95f4\u7684\u53c2\u6570\u9694\u79bb\u4e0d\u591f\u5145\u5206\u3002Multi\u00b2 \u5219\u8ba9 System 1 \u548c System 2 \u4f7f\u7528\u72ec\u7acb LoRA adapter\uff0c\u5e76\u9488\u5bf9\u5404\u81ea\u89d2\u8272\u91c7\u7528\u4e0d\u540c\u8bad\u7ec3\u65b9\u5f0f\uff0c\u4ece\u800c\u907f\u514d\u89c4\u5212\u548c\u6267\u884c\u804c\u8d23\u6df7\u5728\u4e00\u8d77\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"590\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-13-1024x590.png\"  class=\"wp-image-1486\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-13-1024x590.png 1024w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-13-300x173.png 300w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-13-768x443.png 768w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-13-1536x886.png 1536w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-13.png 1894w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe1\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe1\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">4. \u6570\u636e\u96c6\u6784\u9020\u4e0e\u8bad\u7ec3\u65b9\u6cd5\uff08Method\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u4e3a\u4e86\u652f\u6301\u5c42\u7ea7\u5316\u8bad\u7ec3\uff0c\u8bba\u6587\u9996\u5148\u6784\u9020\u4e86\u4e24\u4e2a\u89d2\u8272\u4e13\u7528\u6570\u636e\u96c6\u3002System 1 \u7684\u6570\u636e\u96c6\u7531\u4efb\u52a1\u63cf\u8ff0\u3001\u89c2\u5bdf\u548c\u5b50\u76ee\u6807\u7ec4\u6210\uff0c\u4e3b\u8981\u7528\u4e8e\u8bad\u7ec3\u9ad8\u5c42\u89c4\u5212\u5668\u5b66\u4f1a\u5728\u4e0d\u540c\u73af\u5883\u72b6\u6001\u4e0b\u751f\u6210\u5408\u7406\u5b50\u76ee\u6807\u3002System 2 \u7684\u6570\u636e\u96c6\u5219\u56f4\u7ed5\u6bcf\u4e2a\u5b50\u76ee\u6807\u5c55\u5f00\uff0c\u5305\u542b\u89c2\u5bdf\u3001\u52a8\u4f5c\u3001\u5956\u52b1\u548c\u4e0b\u4e00\u6b65\u89c2\u5bdf\uff0c\u7528\u4e8e\u8bad\u7ec3\u4f4e\u5c42\u6267\u884c\u5668\u5728\u7ed9\u5b9a\u5b50\u76ee\u6807\u4e0b\u9009\u62e9\u5177\u4f53\u52a8\u4f5c\u3002\u8fd9\u6837\u7684\u6570\u636e\u5212\u5206\u4f7f\u4e24\u4e2a\u7cfb\u7edf\u7684\u8bad\u7ec3\u76ee\u6807\u66f4\u52a0\u6e05\u6670\uff0c\u4e5f\u907f\u514d\u4e86\u5355\u4e00\u6570\u636e\u683c\u5f0f\u540c\u65f6\u627f\u62c5\u89c4\u5212\u548c\u6267\u884c\u4e24\u7c7b\u4efb\u52a1\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">System 1 \u7684\u8bad\u7ec3\u76f8\u5bf9\u76f4\u63a5\uff0c\u4f5c\u8005\u4f7f\u7528 SFT \u5bf9\u9ad8\u5c42\u89c4\u5212\u5668\u8fdb\u884c\u884c\u4e3a\u514b\u9686\uff0c\u4f7f\u5176\u6a21\u4eff\u4e13\u5bb6\u8f68\u8ff9\u4e2d\u7684\u5b50\u76ee\u6807\u751f\u6210\u8fc7\u7a0b\u3002\u8fd9\u6837\u505a\u7684\u539f\u56e0\u662f\uff0c\u9ad8\u5c42\u89c4\u5212\u66f4\u504f\u8bed\u4e49\u7406\u89e3\u548c\u4efb\u52a1\u5206\u89e3\uff0c\u76ee\u6807\u662f\u4fdd\u6301\u8ba1\u5212\u7ed3\u6784\u7a33\u5b9a\uff0c\u800c\u4e0d\u662f\u901a\u8fc7\u73af\u5883\u63a2\u7d22\u4e0d\u65ad\u8bd5\u9519\u3002\u56e0\u6b64\uff0cSFT \u5728\u8fd9\u91cc\u66f4\u5408\u9002\uff0c\u4e5f\u66f4\u5bb9\u6613\u4fdd\u8bc1\u751f\u6210\u5b50\u76ee\u6807\u7684\u4e00\u81f4\u6027\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">System 2 \u7684\u8bad\u7ec3\u66f4\u590d\u6742\u3002\u7531\u4e8e\u4f4e\u5c42\u6267\u884c\u5668\u76f4\u63a5\u9762\u5bf9\u52a8\u6001\u73af\u5883\uff0c\u5355\u7eaf\u6a21\u4eff\u79bb\u7ebf\u8f68\u8ff9\u5bb9\u6613\u8fc7\u62df\u5408\uff0c\u4e5f\u96be\u4ee5\u5904\u7406\u73af\u5883\u53d8\u5316\u5e26\u6765\u7684\u72b6\u6001\u504f\u79fb\u3002\u56e0\u6b64\uff0c\u4f5c\u8005\u4f7f\u7528 actor critic \u5f62\u5f0f\u7684 offline RL \u521d\u59cb\u5316 System 2\u3002critic \u5b66\u4e60\u52a8\u4f5c\u4ef7\u503c\u548c\u72b6\u6001\u4ef7\u503c\uff0cactor \u5219\u7ed3\u5408 advantage \u4fe1\u606f\u66f4\u65b0\u7b56\u7565\uff0c\u4f7f\u6a21\u578b\u4e0d\u4ec5\u6a21\u4eff\u79bb\u7ebf\u6570\u636e\u4e2d\u7684\u52a8\u4f5c\uff0c\u4e5f\u503e\u5411\u4e8e\u9009\u62e9\u4f30\u8ba1\u56de\u62a5\u66f4\u9ad8\u7684\u52a8\u4f5c\u3002\u8bba\u6587\u8fd8\u52a0\u5165 policy anchored advantage term\uff0c\u7528\u6765\u51cf\u8f7b\u8fc7\u5ea6\u6a21\u4eff\uff0c\u63d0\u9ad8\u7b56\u7565\u5728\u79bb\u7ebf\u8f68\u8ff9\u4e4b\u5916\u7684\u8fc1\u79fb\u80fd\u529b\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5728 online training \u9636\u6bb5\uff0cSystem 2 \u4ece\u79bb\u7ebf\u8bad\u7ec3\u5f97\u5230\u7684\u7b56\u7565\u51fa\u53d1\uff0c\u7ee7\u7eed\u4e0e\u73af\u5883\u4ea4\u4e92\uff0c\u5e76\u628a\u65b0\u8f68\u8ff9\u52a0\u5165 replay buffer\u3002\u5728\u7ebf\u66f4\u65b0\u4e2d\uff0c\u4f5c\u8005\u5f15\u5165 KL regularization\uff0c\u4f7f\u65b0\u7b56\u7565\u4e0d\u4f1a\u8fc7\u5ea6\u504f\u79bb\u79bb\u7ebf\u7b56\u7565\u3002\u8fd9\u6837\u65e2\u5141\u8bb8 System 2 \u901a\u8fc7\u5728\u7ebf\u4ea4\u4e92\u4fee\u6b63\u6267\u884c\u9519\u8bef\uff0c\u53c8\u907f\u514d\u7b56\u7565\u66f4\u65b0\u8fc7\u731b\u5bfc\u81f4\u5d29\u6e83\u6216 mode collapse\u3002\u6574\u4f53\u6765\u770b\uff0c\u8fd9\u5957 offline-to-online RL \u8bbe\u8ba1\u7684\u91cd\u70b9\u4e0d\u662f\u5355\u7eaf\u8ffd\u6c42\u66f4\u5f3a\u4f18\u5316\uff0c\u800c\u662f\u8ba9\u4f4e\u5c42\u6267\u884c\u5668\u5728\u957f\u7a0b\u4ea4\u4e92\u4e2d\u66f4\u7a33\u5b9a\u5730\u81ea\u6211\u6539\u8fdb\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"823\" height=\"1024\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-14-823x1024.png\"  class=\"wp-image-1487\" style=\"aspect-ratio:0.8041283374467131;width:375px;height:auto\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-14-823x1024.png 823w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-14-241x300.png 241w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-14-768x955.png 768w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-14.png 936w\" sizes=\"auto, (max-width: 823px) 100vw, 823px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe2\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe2\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">5. \u5b9e\u9a8c\u8bbe\u7f6e\uff08Experimental Setup\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u8bba\u6587\u5728\u4e09\u4e2a\u4ea4\u4e92\u73af\u5883\u4e0a\u8bc4\u4f30 Multi\u00b2\uff0c\u5206\u522b\u662f ScienceWorld\u3001ALFWorld \u548c TextCraft\u3002ScienceWorld \u66f4\u5f3a\u8c03\u79d1\u5b66\u4efb\u52a1\u4e2d\u7684\u957f\u7a0b\u89c4\u5212\u548c\u72b6\u6001\u53d8\u5316\uff0cALFWorld \u66f4\u504f\u5bb6\u5ead\u573a\u666f\u4e2d\u7684\u7269\u4f53\u64cd\u4f5c\u548c\u7a00\u758f\u5956\u52b1\uff0cTextCraft \u5219\u5173\u6ce8\u7b26\u53f7\u5316 crafting \u4efb\u52a1\u4e2d\u7684\u7ec4\u5408\u4f9d\u8d56\u548c\u76ee\u6807\u4fdd\u6301\u3002\u8fd9\u4e09\u4e2a\u73af\u5883\u5171\u540c\u8986\u76d6\u4e86\u4e0d\u540c\u7c7b\u578b\u7684\u957f\u7a0b\u4ea4\u4e92\u573a\u666f\uff0c\u56e0\u6b64\u6bd4\u8f83\u9002\u5408\u68c0\u9a8c\u667a\u80fd\u4f53\u662f\u5426\u771f\u7684\u5177\u5907\u7a33\u5b9a\u7684\u591a\u8f6e\u51b3\u7b56\u80fd\u529b\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u6a21\u578b\u65b9\u9762\uff0c\u4f5c\u8005\u4f7f\u7528 Qwen 2.5 3B\u3001Mistral 7B v0.3 \u548c Llama 3.1 8B \u4f5c\u4e3a backbone\uff0c\u5e76\u901a\u8fc7 LoRA \u8fdb\u884c\u53c2\u6570\u9ad8\u6548\u5fae\u8c03\u3002System 1 \u548c System 2 \u5171\u4eab\u540c\u4e00\u4e2a\u9884\u8bad\u7ec3 backbone\uff0c\u4f46\u4f7f\u7528\u4e0d\u540c\u7684 LoRA adapter\u3002\u5728\u8bad\u7ec3\u548c\u63a8\u7406\u65f6\uff0c\u53ea\u6fc0\u6d3b\u5f53\u524d\u7cfb\u7edf\u5bf9\u5e94\u7684 adapter\u3002\u8fd9\u4e2a\u8bbe\u8ba1\u65e2\u8282\u7701\u8bad\u7ec3\u6210\u672c\uff0c\u53c8\u80fd\u4fdd\u8bc1\u89c4\u5212\u5668\u548c\u6267\u884c\u5668\u5177\u5907\u4e00\u5b9a\u53c2\u6570\u9694\u79bb\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8bc4\u4ef7\u6307\u6807\u91c7\u7528\u4e25\u683c\u7684 pass@1\uff0c\u5373\u6bcf\u4e2a\u4efb\u52a1\u53ea\u5141\u8bb8\u4e00\u6b21 rollout\uff0c\u4e0d\u8fdb\u884c\u591a\u6b21\u91cd\u8bd5\u6216\u91c7\u6837\u3002\u8fd9\u4e00\u70b9\u6bd4\u8f83\u91cd\u8981\uff0c\u56e0\u4e3a\u5b83\u66f4\u80fd\u53cd\u6620 agent \u7684\u4e00\u6b21\u6027\u4efb\u52a1\u5b8c\u6210\u80fd\u529b\uff0c\u800c\u4e0d\u662f\u4f9d\u9760\u591a\u6b21\u5c1d\u8bd5\u78b0\u8fd0\u6c14\u3002\u8bba\u6587\u8fd8\u533a\u5206 ID \u548c OOD splits\uff0cID \u4efb\u52a1\u548c\u8bad\u7ec3\u96c6\u5171\u4eab\u4efb\u52a1\u6a21\u677f\u6216\u4ea4\u4e92\u6a21\u5f0f\uff0c\u4f46\u5177\u4f53\u73af\u5883\u5b9e\u4f8b\u4ecd\u4e0d\u540c\uff1bOOD \u4efb\u52a1\u5219\u4f7f\u7528\u65b0\u7684\u4efb\u52a1\u6a21\u677f\uff0c\u66f4\u80fd\u8003\u5bdf\u6cdb\u5316\u80fd\u529b\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5bf9\u6bd4\u65b9\u6cd5\u5305\u62ec prompt-based \u548c fine-tuning-based \u4e24\u5927\u7c7b\u3002prompt-based \u65b9\u6cd5\u5305\u62ec ReAct\u3001Reflexion \u548c ADaPT\uff0cfine-tuning-based \u65b9\u6cd5\u5305\u62ec GRPO \u548c Glider\u3002\u8fd9\u6837\u7684\u57fa\u7ebf\u8bbe\u7f6e\u6bd4\u8f83\u5b8c\u6574\uff0c\u65e2\u8986\u76d6\u4e86\u975e\u5c42\u7ea7\u65b9\u6cd5\uff0c\u4e5f\u8986\u76d6\u4e86\u5df2\u6709\u5c42\u7ea7\u65b9\u6cd5\uff0c\u8fd8\u80fd\u533a\u5206 prompt \u65b9\u6848\u548c\u53c2\u6570\u66f4\u65b0\u65b9\u6848\u4e4b\u95f4\u7684\u5dee\u5f02\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"562\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-15-1024x562.png\"  class=\"wp-image-1488\" style=\"aspect-ratio:1.8227436088696005;width:632px;height:auto\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-15-1024x562.png 1024w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-15-300x165.png 300w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-15-768x421.png 768w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-15-1536x843.png 1536w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-15.png 1892w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe3\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe3\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">6. \u5b9e\u9a8c\u7ed3\u679c\u4e0e\u6027\u80fd\u5206\u6790\uff08Experiments\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u4e3b\u5b9e\u9a8c\u7ed3\u679c\u663e\u793a\uff0cMulti\u00b2 \u5728\u5927\u591a\u6570\u8bbe\u7f6e\u4e0b\u53d6\u5f97\u6700\u4f73\u8868\u73b0\u3002\u4ee5 Qwen 2.5 3B \u4e3a\u4f8b\uff0cMulti\u00b2 \u5728 ScienceWorld ID \u4e0a\u8fbe\u5230 60.68\uff0c\u5728 ScienceWorld OOD \u4e0a\u8fbe\u5230 29.04\uff0c\u5728 ALFWorld ID \u548c OOD \u4e0a\u5206\u522b\u8fbe\u5230 61.43 \u548c 49.29\uff0c\u5728 TextCraft \u4e0a\u8fbe\u5230 28.50\u3002\u4ee5 Mistral 7B \u4e3a\u4f8b\uff0cMulti\u00b2 \u5728 ScienceWorld ID \u4e0a\u8fbe\u5230 69.97\uff0c\u5728 TextCraft \u4e0a\u8fbe\u5230 44.50\uff0c\u4e5f\u660e\u663e\u8d85\u8fc7\u591a\u6570\u57fa\u7ebf\u3002\u4ee5 Llama 3.1 8B \u4e3a\u4f8b\uff0cMulti\u00b2 \u5728 ALFWorld ID \u548c OOD \u4e0a\u5206\u522b\u8fbe\u5230 57.86 \u548c 56.43\uff0c\u5728 TextCraft \u4e0a\u8fbe\u5230 35.60\u3002\u6574\u4f53\u6765\u770b\uff0cMulti\u00b2 \u5bf9\u4e0d\u540c\u6a21\u578b\u89c4\u6a21\u548c\u4e0d\u540c\u73af\u5883\u90fd\u5177\u6709\u8f83\u5f3a\u9002\u7528\u6027\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u4ece\u65b9\u6cd5\u7c7b\u522b\u770b\uff0cprompt-based \u65b9\u6cd5\u6574\u4f53\u8868\u73b0\u8f83\u5f31\uff0c\u5c24\u5176\u5728\u957f\u7a0b\u4efb\u52a1\u4e2d\u5bb9\u6613\u51fa\u73b0\u9519\u8bef\u7d2f\u79ef\u3002ADaPT \u7531\u4e8e\u5f15\u5165\u5c42\u7ea7 prompting\uff0c\u5728\u90e8\u5206\u4efb\u52a1\u4e0a\u6bd4 ReAct \u548c Reflexion \u66f4\u597d\uff0c\u8bf4\u660e\u4efb\u52a1\u5206\u89e3\u672c\u8eab\u786e\u5b9e\u6709\u4ef7\u503c\u3002GRPO \u4f5c\u4e3a\u5355 agent RL \u65b9\u6cd5\u5728\u4e00\u4e9b\u573a\u666f\u4e2d\u6548\u679c\u6709\u9650\uff0c\u8bf4\u660e\u7b80\u5355\u5f3a\u5316\u5b66\u4e60\u5e76\u4e0d\u80fd\u81ea\u7136\u89e3\u51b3\u957f\u7a0b\u76ee\u6807\u4fdd\u6301\u95ee\u9898\u3002Glider \u4f5c\u4e3a\u5c42\u7ea7\u5fae\u8c03\u57fa\u7ebf\u8868\u73b0\u660e\u663e\u66f4\u5f3a\uff0c\u4f46\u4ecd\u7136\u4e0d\u5982 Multi\u00b2\uff0c\u8fd9\u8bf4\u660e\u5c42\u7ea7\u7ed3\u6784\u4e4b\u5916\uff0c\u89d2\u8272\u4e13\u7528\u8bad\u7ec3\u548c\u4f4e\u5c42\u6267\u884c\u5668\u7684\u5728\u7ebf\u81ea\u6211\u6539\u8fdb\u540c\u6837\u91cd\u8981\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8bba\u6587\u8fd8\u7ed9\u51fa\u4e86\u5b9a\u6027\u9519\u8bef\u5206\u6790\u3002Figure 3 \u5c55\u793a\u4e86 ScienceWorld \u4e2d Find a plant \u4efb\u52a1\u7684\u5178\u578b\u5931\u8d25\u6848\u4f8b\u3002ReAct\u3001Reflexion \u548c ADaPT \u7b49\u65b9\u6cd5\u5bb9\u6613\u4ea7\u751f\u88ab\u73af\u5883\u62d2\u7edd\u7684\u975e\u6cd5\u52a8\u4f5c\uff0c\u6216\u8005\u9677\u5165\u91cd\u590d\u6267\u884c\u4f46\u6ca1\u6709\u8fdb\u5c55\u7684\u5faa\u73af\u3002GRPO \u548c Glider \u867d\u7136\u7ecf\u8fc7\u8bad\u7ec3\uff0c\u4f46\u4e5f\u53ef\u80fd\u6301\u7eed\u671d\u7740\u4e0d\u5408\u7406\u76ee\u6807\u6267\u884c\u3002\u76f8\u6bd4\u4e4b\u4e0b\uff0cMulti\u00b2 \u80fd\u901a\u8fc7\u660e\u786e\u5b50\u76ee\u6807\u7ea6\u675f\u6267\u884c\u65b9\u5411\uff0c\u540c\u65f6\u5229\u7528 RL fine tuning \u63d0\u9ad8\u52a8\u4f5c\u6709\u6548\u6027\uff0c\u56e0\u6b64\u66f4\u5c11\u51fa\u73b0\u76ee\u6807\u6f02\u79fb\u548c\u65e0\u6548\u5faa\u73af\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">token efficiency \u65b9\u9762\uff0cFigure 4 \u663e\u793a Multi\u00b2 \u5728 ScienceWorld ID split \u4e0a\u7528\u66f4\u5c11 token \u83b7\u5f97\u66f4\u9ad8\u6027\u80fd\u3002\u5176\u539f\u56e0\u5728\u4e8e\u5c42\u7ea7\u6846\u67b6\u5e76\u4e0d\u9700\u8981\u6bcf\u4e00\u6b65\u90fd\u628a\u5b8c\u6574\u5386\u53f2\u548c\u590d\u6742\u63d0\u793a\u585e\u7ed9\u540c\u4e00\u4e2a agent\uff0c\u800c\u662f\u6309\u9700\u8c03\u7528 System 1 \u751f\u6210\u5b50\u76ee\u6807\uff0c\u518d\u7531 System 2 \u6267\u884c\u5177\u4f53\u52a8\u4f5c\u3002\u8fd9\u6837\u65e2\u51cf\u5c11\u957f prompt \u4f9d\u8d56\uff0c\u4e5f\u8ba9\u4f4e\u5c42\u52a8\u4f5c\u8f93\u51fa\u66f4\u7a33\u5b9a\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5728 objective drift robustness \u5206\u6790\u4e2d\uff0cFigure 5 \u6309\u4efb\u52a1\u96be\u5ea6\u5c55\u793a\u4e86\u4e0d\u540c\u65b9\u6cd5\u7684\u6027\u80fd\u3002\u968f\u7740\u4efb\u52a1\u4ece easy \u5230 hard\uff0cprompt-based \u65b9\u6cd5\u9000\u5316\u660e\u663e\uff0cfine-tuning-based \u65b9\u6cd5\u867d\u7136\u66f4\u5f3a\uff0c\u4f46\u5728 hard \u4efb\u52a1\u4e0a\u4ecd\u6709\u4e0b\u964d\u3002Multi\u00b2 \u5728\u56f0\u96be\u4efb\u52a1\u4e0a\u4f18\u52bf\u66f4\u660e\u663e\uff0c\u8bf4\u660e\u5b83\u5bf9\u957f horizon \u548c\u6301\u7eed\u76ee\u6807\u8ddf\u8e2a\u66f4\u9c81\u68d2\u3002\u8fd9\u4e5f\u662f\u5168\u6587\u5f88\u91cd\u8981\u7684\u5b9e\u9a8c\u652f\u6491\uff0c\u56e0\u4e3a\u5b83\u76f4\u63a5\u56de\u5e94\u4e86\u8bba\u6587\u5f00\u5934\u63d0\u51fa\u7684 objective drift \u95ee\u9898\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u6d88\u878d\u5b9e\u9a8c\u8fdb\u4e00\u6b65\u8bf4\u660e\u5404\u6a21\u5757\u90fd\u6709\u5fc5\u8981\u3002Figure 6 \u663e\u793a\uff0cSystem 1 \u7528 SFT\u3001System 2 \u7528 RL \u7684\u89d2\u8272\u5339\u914d\u65b9\u5f0f\u8868\u73b0\u6700\u597d\u3002\u5982\u679c\u4ea4\u6362\u8bad\u7ec3\u65b9\u5f0f\uff0c\u8ba9 System 1 \u7528 RL\u3001System 2 \u7528 SFT\uff0c\u6548\u679c\u6700\u5dee\uff0c\u8bf4\u660e\u89c4\u5212\u548c\u6267\u884c\u4e0d\u80fd\u968f\u4fbf\u5957\u7528\u540c\u4e00\u79cd\u8bad\u7ec3\u8303\u5f0f\u3002Figure 6 \u8fd8\u663e\u793a\uff0c\u5c42\u7ea7\u7ed3\u6784\u4f18\u4e8e\u5355\u6a21\u578b\u7ed3\u6784\uff0crole-specific adapter \u4f18\u4e8e shared adapter\u3002Figure 7 \u5219\u8bf4\u660e\uff0cpolicy anchored offline objective \u548c KL regularized online objective \u90fd\u6709\u52a9\u4e8e\u63d0\u9ad8\u7a33\u5b9a\u6027\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u6700\u540e\uff0cFigure 8 \u548c Figure 9 \u5206\u522b\u5206\u6790\u4e86\u6a21\u578b\u89c4\u6a21\u548c\u5728\u7ebf\u9002\u5e94\u6548\u679c\u3002Figure 8 \u663e\u793a\uff0c\u5728 ID split \u4e0a\u4e0d\u540c Qwen \u89c4\u6a21\u5dee\u5f02\u4e0d\u5927\uff0c\u4f46\u5728 OOD split \u4e0a\u6a21\u578b\u8d8a\u5927\u8868\u73b0\u8d8a\u597d\uff0c\u8bf4\u660e\u66f4\u5927 backbone \u5bf9\u5206\u5e03\u5916\u6cdb\u5316\u66f4\u6709\u5e2e\u52a9\u3002Figure 9 \u663e\u793a\uff0conline RL \u80fd\u8fdb\u4e00\u6b65\u63d0\u5347\u6267\u884c\u53ef\u9760\u6027\uff0c\u5c24\u5176\u5728 OOD split \u4e0a\u6536\u76ca\u66f4\u660e\u663e\uff0c\u8bf4\u660e\u5728\u7ebf\u4ea4\u4e92\u786e\u5b9e\u80fd\u591f\u5e2e\u52a9 System 2 \u4fee\u6b63\u79bb\u7ebf\u8bad\u7ec3\u4e2d\u65e0\u6cd5\u8986\u76d6\u7684\u6267\u884c\u9519\u8bef\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"779\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-16-1024x779.png\"  class=\"wp-image-1489\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-16-1024x779.png 1024w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-16-300x228.png 300w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-16-768x584.png 768w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-16-1536x1169.png 1536w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-16.png 1572w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe4\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe4\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"677\" height=\"1024\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-17-677x1024.png\"  class=\"wp-image-1490\" style=\"width:348px;height:auto\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-17-677x1024.png 677w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-17-198x300.png 198w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-17.png 754w\" sizes=\"auto, (max-width: 677px) 100vw, 677px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe5\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe5\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"315\" src=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-20-1024x315.png\"  class=\"wp-image-1493\" srcset=\"https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-20-1024x315.png 1024w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-20-300x92.png 300w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-20-768x236.png 768w, https:\/\/www.ndnlab.com\/wp-content\/uploads\/2026\/07\/image-20.png 1528w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" title=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe6\" alt=\"Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments\u63d2\u56fe6\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">7. \u8d21\u732e\u4e0e\u7ed3\u8bba\uff08Contributions and Conclusion\uff09<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u672c\u6587\u7684\u4e3b\u8981\u8d21\u732e\u53ef\u4ee5\u6982\u62ec\u4e3a\u56db\u70b9\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u7b2c\u4e00\uff0c\u63d0\u51fa Multi\u00b2 \u5c42\u7ea7\u5316\u591a\u667a\u80fd\u4f53\u6846\u67b6\uff0c\u5c06\u957f\u7a0b\u4ea4\u4e92\u4efb\u52a1\u4e2d\u7684 planning \u548c execution \u5206\u522b\u5efa\u6a21\u4e3a\u4e0d\u540c\u4f18\u5316\u95ee\u9898\u3002System 1 \u8d1f\u8d23\u751f\u6210\u4e0a\u4e0b\u6587\u76f8\u5173\u5b50\u76ee\u6807\uff0cSystem 2 \u8d1f\u8d23\u5728\u52a8\u6001\u73af\u5883\u4e2d\u6267\u884c\u539f\u5b50\u52a8\u4f5c\uff0c\u4ece\u7ed3\u6784\u4e0a\u7f13\u89e3\u76ee\u6807\u6f02\u79fb\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u7b2c\u4e8c\uff0c\u8bbe\u8ba1\u4e86 role-specialized training pipeline\u3002System 1 \u4f7f\u7528 SFT \u4fdd\u6301\u9ad8\u5c42\u89c4\u5212\u7a33\u5b9a\uff0cSystem 2 \u4f7f\u7528 offline-to-online RL \u63d0\u5347\u52a8\u4f5c\u6267\u884c\u9c81\u68d2\u6027\u3002\u8fd9\u79cd\u8bad\u7ec3\u5206\u5de5\u4e0e\u4e24\u4e2a\u7cfb\u7edf\u7684\u529f\u80fd\u5b9a\u4f4d\u4e00\u81f4\uff0c\u6bd4\u7b80\u5355\u7528\u4e00\u4e2a agent \u6216\u4e00\u4e2a\u5171\u4eab adapter \u66f4\u5408\u7406\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u7b2c\u4e09\uff0c\u63d0\u51fa\u9762\u5411 System 2 \u7684 offline-to-online RL \u76ee\u6807\uff0c\u5728\u79bb\u7ebf\u9636\u6bb5\u901a\u8fc7 critic guided policy anchoring \u51cf\u8f7b\u8fc7\u5ea6\u6a21\u4eff\uff0c\u5728\u5728\u7ebf\u9636\u6bb5\u901a\u8fc7 KL regularization \u907f\u514d\u7b56\u7565\u66f4\u65b0\u4e0d\u7a33\uff0c\u4ece\u800c\u5b9e\u73b0\u66f4\u5b89\u5168\u7684\u81ea\u6211\u6539\u8fdb\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u7b2c\u56db\uff0c\u8bba\u6587\u53d1\u5e03\u4e86\u9762\u5411\u5c42\u7ea7\u667a\u80fd\u4f53\u8bad\u7ec3\u4e0e\u8bc4\u4f30\u7684\u6570\u636e\u96c6\uff0c\u5e76\u5728 ScienceWorld\u3001ALFWorld \u548c TextCraft \u4e0a\u9a8c\u8bc1\u4e86 Multi\u00b2 \u5728\u6027\u80fd\u3001token efficiency \u548c\u957f\u7a0b\u9c81\u68d2\u6027\u4e0a\u7684\u4f18\u52bf\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u6574\u4f53\u6765\u770b\uff0c\u8fd9\u7bc7\u8bba\u6587\u771f\u6b63\u6709\u4ef7\u503c\u7684\u5730\u65b9\u5728\u4e8e\uff0c\u5b83\u6ca1\u6709\u628a\u957f\u7a0b\u4ea4\u4e92\u5931\u8d25\u7b80\u5355\u5f52\u56e0\u4e8e\u6a21\u578b\u80fd\u529b\u4e0d\u591f\uff0c\u800c\u662f\u628a\u95ee\u9898\u62c6\u6210\u4e86\u89c4\u5212\u6f02\u79fb\u3001\u6267\u884c\u8bef\u5dee\u7d2f\u79ef\u548c token \u6548\u7387\u4f4e\u4e09\u4e2a\u5c42\u9762\u3002Multi\u00b2 \u7684\u8bbe\u8ba1\u4e5f\u5bf9\u5e94\u8fd9\u4e09\u4e2a\u5c42\u9762\u5206\u522b\u5904\u7406\uff0c\u7528 System 1 \u4fdd\u6301\u76ee\u6807\uff0c\u7528 System 2 \u7a33\u5b9a\u6267\u884c\uff0c\u7528\u6309\u9700\u8c03\u7528\u964d\u4f4e token \u6d88\u8017\u3002\u5b83\u7684\u7814\u7a76\u95ee\u9898\u6bd4\u8f83\u6e05\u695a\uff0c\u5b9e\u9a8c\u95ed\u73af\u4e5f\u8f83\u5b8c\u6574\uff0c\u9002\u5408\u4f5c\u4e3a LLM agent \u957f\u7a0b\u51b3\u7b56\u65b9\u5411\u7684\u4e00\u7bc7\u9605\u8bfb\u8bba\u6587\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>1. \u6458\u8981\uff08Abstract\uff09 \u672c\u6587\u7814\u7a76\u7684\u662f\u957f\u7a0b\u4ea4\u4e92\u73af\u5883\u4e2d\u5927\u6a21\u578b\u667a\u80fd\u4f53\u7684\u51b3\u7b56\u7a33\u5b9a\u6027\u95ee\u9898\u3002\u73b0\u6709 LLM-based agents \u5df2\u7ecf\u5177\u5907\u8f83\u5f3a\u7684\u4e0a\u4e0b\u6587\u7406\u89e3\u548c\u63a8\u7406\u80fd\u529b\uff0c\u4f46\u5728\u591a\u8f6e\u4ea4\u4e92\u4efb\u52a1\u4e2d\u4ecd\u7136\u5bb9\u6613\u51fa\u73b0\u76ee\u6807\u6f02\u79fb\u3002\u4e5f\u5c31\u662f\u8bf4\uff0c\u6a21\u578b\u4e00\u5f00\u59cb\u80fd\u591f\u7406\u89e3\u4efb\u52a1\u76ee\u6807\uff0c\u4f46\u968f\u7740\u4ea4\u4e92\u8f6e\u6b21\u589e\u52a0\uff0c\u8ba1\u5212\u548c\u52a8\u4f5c\u4f1a\u9010\u6e10\u504f\u79bb\u539f\u59cb\u610f\u56fe\uff0c\u6700\u7ec8\u5bfc\u81f4\u65e0\u6548\u52a8\u4f5c\u3001\u5faa\u73af\u884c\u4e3a\u6216\u4efb\u52a1\u5931\u8d25\u3002\u9488\u5bf9\u8fd9\u4e00\u95ee\u9898\uff0c\u8bba\u6587\u63d0\u51fa\u4e86 Multi\u00b2\uff0c\u4e00\u4e2a\u5c42\u7ea7\u5316\u591a\u667a\u80fd\u4f53\u51b3\u7b56\u6846\u67b6\uff0c\u7528\u4e8e\u63d0\u5347\u667a\u80fd\u4f53\u5728\u957f\u7a0b\u4ea4\u4e92\u73af\u5883\u4e2d\u7684\u7a33\u5b9a\u6027\u3001\u9c81\u68d2\u6027\u548c token \u4f7f\u7528\u6548\u7387\u3002 Mult &hellip; <a href=\"https:\/\/www.ndnlab.com\/?p=1484\">\u7ee7\u7eed\u9605\u8bfb <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":1486,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-1484","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-rengongzhineng"],"_links":{"self":[{"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/posts\/1484","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1484"}],"version-history":[{"count":1,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/posts\/1484\/revisions"}],"predecessor-version":[{"id":1494,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/posts\/1484\/revisions\/1494"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=\/wp\/v2\/media\/1486"}],"wp:attachment":[{"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1484"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1484"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ndnlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1484"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}