MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
- Thesis:
- MetaGPT use assembly line paradigm to decompose complex tasks into smaller and simpler subtasks that are assigned to agents. To avoid compounding errors that happen when we chain LLMs together when some of them hallucinate and the errors get compounded as they get cascaded through the chain, it uses specialized agents that act as validators similar to human experts to validate and check intermediate results.
- This is done by take SOPs and generate sequence of prompts for workflows
- Uses message pools and subscription for agents to communicate
- Depends heavily on Standardized Operating Procedures (SOP)
- There is an executable feedback between engineers and other agents during code generation
- MetaGPT use assembly line paradigm to decompose complex tasks into smaller and simpler subtasks that are assigned to agents. To avoid compounding errors that happen when we chain LLMs together when some of them hallucinate and the errors get compounded as they get cascaded through the chain, it uses specialized agents that act as validators similar to human experts to validate and check intermediate results.
- Methods:
- Use schema and structured output enforce roles and responsibilities for each agent, which reduces hallucinations when compared to dialogue messages
- Contribution:
- MetaGPT framework with well-defined roles and responsibilities between agents by integrating human-like SOPs and message passing between agents
- Executable feedback during code generation to debug and test code
- Takeaways: Schema and structured output is key to reduce hallucinations for complex and long-running projects. Also, decomposing complex tasks into smaller ones and strictly enforce roles through prompt constraint help with task completion for complex tasks along with assembly line paradigm where agents work sequentially
- Improvements:
- Not clear how agents would learn with time from doing different projects
- Not sure how do we validate the output of each agent before moving on to the next agent
- Comparisons using LLMs with different sizes as well as codeLLMs
- Notes:
- Using SOPs, agents are given specific responsibilities by decomposing complex tasks into smaller and simpler ones. This will create standards for each agent’s output that will be checked by expert agents at each step
- The standard output of each agent significantly improves the success of code generation such as structured product requirements and artifacts
- MetaGPT is more of SWE agentic framework than just code generation as in StarCoder -> Completion rate is key metric
- Agents work sequentially until reaching the Engineer agent
- Each agent has well-structured and schema for its role/output, mostly documents and diagrams that will be the input for the next agent in the assembly line
- Message pool is used as a collaboration tool between agents. Instead of sending one-to-one messages or sending irrelevant details to agents, messages will be published to the message pool and agents can subscribe to to the topics (messages) they need to do their work
- Executable feedback mechanism is mainly used by Engineer agent to check the executability of the code and its correctness (passing test cases). This is done iteratively in collaboration with QA Engineer agent until the code passes all the unit tests and run successfully before moving to next tasks
- Agents learn from previous projects by summarizing feedbacks and update their constraint prompts for next projects. This is done first step for each new project
#nlp #llm #agents