SayCan by Google

PaLM-SayCan is the first implementation that uses a large-scale language model to plan for a real robot. This allows users to provide high-level instructions, such as “I spilled my drink, can you help?”, and have the robot break it down into smaller actions that are possible with its current abilities, state and environment. When querying existing large language models like GPT-3 with an instruction like this, it can produce reasonable sounding responses but they may not be feasible to execute on the actual robot. The main idea behind SayCan is to use the output of a language model combined with knowledge about what skills are available and how likely they are to succeed (obtained from affordance functions) in order to assess which skill should be used next. This process happens iteratively – each skill selected by SayCan is appended onto the response from the robot until all tasks needed for completion of the instruction have been completed or progress has ceased. In summary, PaLM-Saycan enables robots to understand complex instructions given by users and make decisions based on their abilities and environment so that they can carry out those instructions successfully!