Goal2Skill: Adaptive Planning for Long-Horizon Manipulation

Goal2Skill is a framework for long-horizon manipulation that enables robotic systems to achieve complex objectives through adaptive planning and reflection. It employs a dual-system AI architecture that separates high-level strategic planning from low-level motor control, allowing it to continuously adjust its strategy based on real-time feedback and recover from errors. This approach is crucial for navigating the uncertainty inherent in complex, multi-step tasks.

The Goal2Skill Architecture: A Dual-System Approach

At its core, Goal2Skill operates using a dual-system architecture that divides the labor of a complex task. This separation of concerns is key to its ability to handle long-horizon manipulation challenges.

• High-Level Planner: A Vision-Language Model (VLM) acts as the "brain" of the operation. It is responsible for high-level semantic planning, which includes decomposing a broad goal (e.g., "clean the table") into a sequence of smaller, manageable sub-tasks. This planner also manages a structured memory system, allowing it to track progress, and uses reflection to learn from execution failures.
• Low-Level Executor: A Vision-Language-Action (VLA) model serves as the "hands" of the system. It receives a specific sub-task from the high-level planner and is responsible for the geometry-oriented action generation needed to execute it. This involves translating a command like "pick up the cup" into the precise motor controls required to perform the action in the physical world.

This structure allows the VLM to focus on the "what" and "why" of the task, while the VLA handles the "how," creating a robust and flexible system.

Adaptive Planning and Reflection in Goal2Skill

Adaptive planning and reflection are the core principles that allow Goal2Skill to operate effectively over long horizons where conditions are unpredictable. This contrasts with traditional static planning, which executes a fixed sequence of steps and often fails when reality deviates from the initial plan.

The mental model for this adaptive process is akin to a thermostat: it continuously measures the current state, compares it to the target goal, and adjusts its behavior. In Goal2Skill, this is achieved through a tight feedback loop between the planner and the executor. The VLM-based planner decomposes the task, the VLA-based executor attempts a sub-task, and a closed-loop verification process reports the outcome.

If a sub-task fails, the system doesn't just stop. The high-level planner reflects on the failure, diagnoses the potential cause, and generates an adaptive replan. This might involve retrying the step, trying an alternative sub-task, or even modifying the overall plan to circumvent the obstacle. This shifts the focus from mere execution to a continuous cycle of learning, evaluation, and iteration.

Long-Horizon Manipulation in Practice

In a practical setting, Goal2Skill's architecture enables it to tackle complex, long-horizon tasks that would stymie more rigid systems. Consider a multi-step instruction like tidying a room.

1. Task Decomposition: The VLM planner first breaks down the ambiguous goal "tidy the room" into concrete steps, such as "pick up the book from the floor," "place the book on the shelf," "find the dirty cup," and "put the cup in the sink."
2. Execution and Verification: The planner sends the first sub-task, "pick up the book from the floor," to the VLA executor. The VLA generates the motor commands to perform the action. The system then verifies if the book was successfully picked up.
3. Memory and Progress: Upon success, the planner updates its structured memory to note that the step is complete and moves to the next sub-task.
4. Failure Recovery and Replanning: If the robot fails to grasp the book, the verification loop reports a failure. The VLM planner then engages its reflection capability. It might hypothesize that the grip was wrong and instruct the VLA to try again with a different grasp. If that fails, it might generate a new plan, such as pushing the book to a better position before attempting to pick it up.

This ability to dynamically replan based on real-world outcomes makes the system robust and capable of completing long sequences of actions despite unforeseen difficulties.

Comparison with Traditional Planning Frameworks

Goal2Skill's adaptive nature stands in sharp contrast to traditional robotic planning methods, which often rely on static, pre-computed plans.

Challenges and Current Limitations

Despite its advanced design, the Goal2Skill framework has several challenges and limitations that point toward future areas of research.

• Input Modalities: The system currently relies on provided subtitles as its primary textual channel and does not process raw audio. Integrating an Automatic Speech Recognition (ASR) module is a necessary next step.
• Model Optimization: The vision and grounding modules are fixed during the learning phase. Jointly optimizing these components with the rest of the system could significantly enhance performance and accuracy.
• Performance and Robustness: The framework may see performance degrade on very long contexts. It also has known limitations in recognizing fine-grained entities (e.g., specific celebrities) and exhibits limited robustness against adversarial perturbations in visual inputs.
• Deployment Latency: Real-time deployment is constrained by the throughput of its visual encoders and available GPU hardware. Reducing latency will require hardware-aware model choices and optimization techniques like micro-batching.

Furthermore, like other multimodal systems, it must avoid common pitfalls such as "shortcut learning," where the model relies on one modality even when another provides conflicting information, or "time-blind" processing that ignores the sequence of events.

AI Foundations for Adaptive Systems

The effectiveness of Goal2Skill is built on a solid foundation of AI models that process vast amounts of data to inform planning and reflection. Understanding these foundations is critical for evaluating performance and anticipating failure modes.

Generative vs. Predictive Models for Foresight

Generative and predictive models serve distinct but complementary roles within the Goal2Skill architecture.

The VLM acts in a generative capacity, exploring the "what if" scenarios of a task and creating plausible plans. The VLA is more predictive, estimating the direct outcome of a physical action. Using them in tandem allows the system to both strategize creatively and act precisely.

Data, Feedback, and Evaluation

For the AI models to be effective, the data and feedback they receive are paramount. Goal2Skill's structured memory and closed-loop verification are designed to address this.

• Structured Memory: This acts like a shared schema, mapping raw perceptual data into consistent fields that track the state of the world, the task plan, and completed steps. This allows for human auditing and traceability.
• Closed-Loop Feedback: The verification process provides constant, real-time feedback on the success or failure of actions. This data is crucial for the reflection mechanism and for updating plans.
• Evaluation: Evaluating the system involves more than just final task success. It requires walk-forward validation that respects the chronological flow of a task, auditing outputs for biases, and ensuring that the system's internal model of the world does not drift significantly from reality.

Frequently Asked Questions

Conclusion

The Goal2Skill framework represents a significant step forward in long-horizon manipulation by embedding adaptive planning and reflection into its core architecture. By separating high-level semantic planning from low-level motor execution, it creates a robust system capable of decomposing complex goals, learning from real-time feedback, and recovering from failures. While challenges related to input modalities, model optimization, and deployment latency remain, Goal2Skill's dual-system approach provides a powerful blueprint for creating more intelligent, resilient, and capable robotic systems that can operate effectively in the uncertainty of the real world.

Sources & References

• Designing Machine Learning Systems
• Talks
• survey on multimodal large language models | National Science Review | Oxford Academic
• The State of AI Search in 2026: Complete Guide - aeoengine blog | AEO Engine Blog
• AgentGPT 🤖
• Autonomous LLM Agents: Real-World Capabilities and Current Limits
• [1912.00245] Scalable Graph Algorithms
• [2107.12045] How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• [2407.15452] GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs
• [2505.24726] Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning