BRKMYR/ AI Builder Portfolio/ iCub Deep Learning Object Recognition

iCub Deep Learning Object Recognition

Master's thesis, TU Munich, 2016: "Learning to Recognize New Objects Using Deep Learning and Contextual Information." A convolutional network on the iCub humanoid, with attribute and affordance classifiers and a KnowRob ontology on top, so the robot could recognise objects it had never been trained on and learn new classes at runtime. Stand-alone CNN on unknown objects: 0%. With semantic reasoning: 65%.

Year

2016

Institution

TUM, Institute for Cognitive Systems

Vision

BVLC CaffeNet, FC7 features

Platform

iCub humanoid

Status

Archived artifact

Context

Robotics before the VLM era.

In 2016 the going assumption in service robotics was that a robot needed a knowledge base to reason about the world, because pattern-matching alone would not tell you what a "cup" was for or where it belonged. The KnowRob framework at TUM was one of the more serious attempts at that: an OWL ontology of household objects, backed by SWI-Prolog for reasoning, hooked into the ROS ecosystem so a real robot could query it.

The iCub humanoid was the target platform. This project was one contribution to that line of work: a Prolog reasoning layer plus a C++ ROS/YARP bridge that let the robot observe an object, classify its physical attributes, and then ask the ontology "what is this and what is it for."

The perception under it was deep learning, in the year CNNs had won ImageNet but were barely deployed on robots. A BVLC CaffeNet (an AlexNet variant) ran on the iCub's camera crops over YARP; its 4096-dimensional FC7 activations fed regularised least-squares classifiers (GURLS) for material, shape, and affordance, and a fine-tuned CaffeNet handled the 28 known classes of iCubWorld28. A network trained on N classes scores 0% on class N+1. The thesis question was what the robot could still know about an object beyond its pixels.

Pipeline

Vision to ontology, in one loop.

iCub cameras → YARP crops (256×256) ↓ CaffeNet forward pass, FC7 features (4096-d) GURLS classifiers: object class, material, shape, affordance ↓ four YARP ports (object, material, shape, affordance) C++ ROS bridge node ↓ json_prolog query KnowRob (SWI-Prolog + semweb + owl) ↓ OWL reasoning objects_affordances_attributes.owl ↓ result: known class, or "unknown" Interactive learning loop ↓ operator provides label Assert new facts, export new OWL class file

The interesting piece was the last step. If the ontology did not match the object, the operator could correct it, and the system would assert the new facts into KnowRob's RDF store and write a fresh OWL class definition to disk. The robot's ontology grew as it worked.

Results

Semantic context on top of the network.

Condition	Stand-alone CNN	CNN + semantic reasoning
Known objects	85%	98%
Unknown objects	0%	65%

Datasets: iCubWorld28 (28 classes in 7 categories, 25,831 training and 24,884 test images) and a self-collected TUM-ICS set. Code: caffe-python-Scripts (data preparation, FC7 extraction, fine-tuning, evaluation, live YARP classification), icub-src-imageProcessing (visual servoing and capture).

Highlights

What is worth noting from the code.

Deep learning in the loop, 2016. Caffe was the framework, PyTorch did not exist yet. FC7 features as the representation and small classifiers on top was the working recipe for robots with few labelled images, and the live path (camera to CaffeNet to MATLAB over YARP ports) ran on the real robot, not offline.
Affordance-based reasoning. Objects were classified by what they were for (drinking, eating, cleaning) via OWL restrictions on hasObjectUsedFor, not only by shape or material. That framing predates the affordance vocabulary that modern VLM-plus-robotics work has settled on.
Runtime ontology learning. The C++ bridge (computablesCpp.cpp) took an operator correction and wrote a new OWL class file, so the knowledge base was mutable at run time rather than a static asset compiled once.
Custom KnowRob computables. comp_onTopTable/2 did spatial reasoning against the semantic map to check whether an object was on the table. check_MaterialType, check_ShapeType, check_AffordanceType did attribute checks via OWL restrictions.
Open-world workaround. A separate module (missing_definitions.pl) asserted attribute knowledge that OWL's open-world semantics would not infer on its own, one of the recurring pain points of doing serious reasoning on top of OWL.
Full ROS/YARP integration. The system talked to the iCub's actual vision pipeline over YARP ports, not to a mock, and lived inside a ROS launch file so it could run alongside the rest of the iCub stack.

Why this matters now

Ontologies are quietly back.

The 2016 answer to "what should a robot know about a cup" was an OWL ontology and a Prolog reasoner. The 2026 answer is usually a VLM. The interesting thing is that the two are converging: modern work on robot affordance grounding, task planning with LLMs, and neuro-symbolic scene understanding is re-introducing structured knowledge on top of the perception model, for the same reason KnowRob existed in the first place. Explicit typed relations survive failure modes that vibes-based reasoning does not.

For the author it was also the start of a working pattern: a neural network does the perception, structured knowledge makes its answer auditable. That is the shape of every product since, from productionising deep learning at TerraLoupe to safety cases for automated driving. This page is here as a snapshot of what that looked like ten years ago, not as a runnable system. The original ROS Indigo/Jade stack, KnowRob, and iCub simulator no longer install cleanly on modern machines, and reviving them is not the point.