Could a robot one day tell you where you left your keys, wallet, or an unfinished part on a factory floor? Researchers at the Massachusetts Institute of Technology have developed a memory system that moves a step in that direction, giving robots a way to remember what they saw, where they saw it, and when it happened.
The system is called Describe Anything, Anywhere, Anytime, at Any Moment, or DAAAM. It lets a moving robot build a detailed map of its surroundings and then answer plain-language questions about objects in that space, fast enough for real-time use, according to MIT News.
A map that remembers
A regular robot map can show walls, doors, furniture, and paths. But that is not the same as remembering that the red bike with a flat tire was near a specific building, or that a tool was left in a bin yesterday evening.
DAAAM is designed for that second kind of memory. The researchers call it “spatiotemporal memory,” which means a memory that connects place with time, much like people do when they retrace their steps after losing something.
The team behind DAAAM
Luca Carlone, an associate professor in MIT’s Department of Aeronautics and Astronautics and director of the SPARK Laboratory, worked on the project with lead author Nicolas Gorlo, an MIT graduate student, and Lukas Schmid, a former MIT researcher who is now a professor at the University of Technology Nuremberg.
Carlone’s central point is simple enough. If robots are going to work beside people, they need to understand requests in the way people actually speak, not just through rigid commands or labels.
Why robot maps fall short
Computer vision systems can often describe objects in impressive detail. Robotic mapping systems can also build three-dimensional layouts of rooms, buildings, or campuses.
The trouble is that these strengths do not always meet in one fast system. A robot may know the shape of a room without knowing much about the objects inside it, or it may describe objects one by one so slowly that the method is not useful while the robot is moving.
A faster way to store details
DAAAM tries to solve that problem by grouping nearby objects as the robot travels. It also chooses key frames, which are the clearest views of several objects at once, so the system can describe many things in parallel instead of wasting time on every single camera frame.
That choice matters in the real world, where a robot may pass hundreds of objects in just a few minutes. The team reports that this approach can speed up the description process tenfold, and Gorlo summed up the idea with a short line, “We annotate every object only once.”
How it answers questions
Once DAAAM builds its memory, it still has to search through a large collection of objects and descriptions.
To do that, the researchers use a large language model, the same broad type of AI behind modern chatbots, but connected to tools that search the robot’s stored map.
In practical terms, that means a person could ask about “the sculpture near the building” or “the red bike with the flat tire.” The system can search by meaning, location, or both, which helps it reduce made-up answers and respond in only a few seconds.
What the tests showed
The results are not just a neat demo. When compared with other methods, DAAAM was between 21 percent and 53 percent more accurate, depending on the type of question, according to MIT.
The open-access paper also reports stronger results on tasks involving object questions, location errors, time errors, and step-by-step task grounding. One test setting covered travel sequences of about 36 minutes and just over one mile, which matters because a home, warehouse, or campus is not a single frozen snapshot.
Where it could matter
The factory example is easy to picture. A worker leaves a partly-assembled component in a storage bin, comes back the next day, and asks a robot to find it without digging through labels, cameras, or a checklist.
There are other possible uses too. Augmented reality systems could guide maintenance workers toward a part that looks unusual, while travelers might use a similar memory system to get directions through large, confusing spaces like stations, hospitals, or airports.
What comes next
The researchers want to expand DAAAM so it can record important events in a space, not just objects and their locations. They are also working on confidence levels, so a robot might say, in effect, that it is fairly sure but not completely certain.
That caution is important. A robot that claims too much could be annoying at home and risky in a workplace, so the next step is not just remembering more, but knowing when the memory may be weak.
Robots, keys, and limits
So, could AI tell you where you left your keys? Not as a household guarantee yet, but DAAAM shows how a robot could start building the kind of grounded memory needed for that task.
The research was funded in part by the U.S. Army Research Laboratory and the Office of Naval Research, and MIT News noted that Carlone is currently on sabbatical as an Amazon Scholar while the work described was performed at MIT and not associated with Amazon.
The official study has been published in the IEEE/CVF Proceedings of the Computer Vision and Pattern Recognition Conference.
