Benchmarking is Part 1 of 4 in a series on Robotics+AI :
recent developments, and the surprising realities we grapple with.
The ThreeDWorld platform gives Robotics+AI developers a standardized virtual environment for training, benchmarking, and testing.
- Why needed:
- Duplicating the exact dynamics of the physical world is extremely difficult
- Safety and expense practicalities when training real robots in actual environments
- What it offers:
- “a more advanced virtual environment simulator to define a new embodied AI task requiring an agent to change the states of multiple objects under realistic physical constraints”
- Why important
- Robotics+AI is central to advanced manufacturing, which is strategic because
- 70% of world goods are manufactured by only 10 countries,
- The US and others face decreasing labor availability.
- TDW provides a standard environment to benchmark models and developments
- A standardized environment gives a clearer understanding of the state of the art in robotics development
- Robotics+AI is central to advanced manufacturing, which is strategic because
- What still keeps us awake at night
- TDW has exposed the limits of current learning models
- 10% success rates for RL (reinforcement learning) models
- 25% average success rates (with best at 50%) for Hybrid models
( RL agent is combined with a rule-based high-level planner)
- Reinforced-Learning models struggle even at tasks with defined goal and rules in a standardized and highly simplified world. Are we ready for fully-autonomous bots roaming the real world and its continuous, and far more complex choices.
- Now consider fully-autonomous agents (e.g. vehicles): How does it decide its hierarchy of higher and lesser goals?
- An autonomous vehicle must prioritize the safety of its passenger, but what if the choice involves sacrificing a pedestrian to save the passenger? Which pedestrian?
- We regard human intelligence as the ultimate modeling for AI. Think of the task means as analogous to “ethics”, and choices as to “morality”. How well do we humans agree on the hierarchy – or how we come to an agreement.
- TDW has exposed the limits of current learning models
LINKS:
This is Part 1 of 4 in a series on Robotics+AI :
recent developments, and the surprising realities we grapple with.
+ Don’t miss next in this series:
Part 2: DEVELOPMENT:
Google’s INTRINSIC reducing the time, cost & complexity of developing robots in their ability to sense, learn, and adapt as they complete their tasks in a wider range of settings and applications.
Part 3: SIMULATION:
Megaverse is a physics-based interactive simulation platform for Reinforcement Learning, allowing agents one million experiences per second– 70x faster than DeepMind.
Part 4: EDGES:
But what about the underlying model we hold as the ultimate intelligence – the human mind?