BRKMYR/Prototypes I built/iCub Deep Learning Object Recognition

iCub Deep Learning Object Recognition

Master's thesis, TU Munich, 2016. A humanoid robot looks at a household object, describes it by material, shape and use, and asks a knowledge base for its name. When the answer is missing, a person teaches it in one short dialogue and the robot keeps the new object.

Archived artifact · 2016 · hand written · results from the thesis, not re run

At a glance

What

A system on the iCub humanoid robot that names household objects and learns new ones while it works. A convolutional neural network (CNN, an image classifier trained on labelled photos) describes each object by material, shape and affordance, meaning what the object can be used for. A knowledge base then infers the name.

Why it matters

A network trained on 28 objects scores zero on the 29th. A robot in a kitchen meets new objects every day. This thesis showed that describing an object by attributes and reasoning over an ontology, a formal catalogue of things and their relations, lets the robot place objects it never trained on, with a person correcting it.

How it was built

Written by hand in 2016, before PyTorch existed. Python 2 scripts drive Caffe and export FC7 features, the 4,096 numbers from the network's second to last layer, into MATLAB classifiers. C++ modules on YARP, the iCub's message bus, pass the labels through a standard robotics software layer to KnowRob, which reasons over the object catalogue in Prolog.

Problem

A closed class list ends at the last training label.

In 2016 convolutional networks had just won ImageNet and were barely running on robots. They also had a hard edge. A network trained on 28 object classes returns one of those 28 for every image it sees, including the object it has never met. A service robot in a household meets that case constantly.

The going assumption in service robotics was that a robot also needs a knowledge base, because pattern matching alone does not say what a cup is for or where it belongs. KnowRob at TUM was one of the serious attempts at that, with a catalogue of household objects written in OWL, the Web Ontology Language, reasoning in Prolog, a language built for logical rules, and a standard robotics software layer on top. The thesis question was what a robot can still work out about an object once the network runs out of labels.

What it does

Attributes from the network, names from the ontology.

The system takes a 256 by 256 crop from the iCub's image stream, runs it through CaffeNet and reads out a 4,096 dimensional FC7 vector. Classifiers predict object, material, shape and affordance and publish them on four YARP ports. A C++ bridge turns those four labels into a logical query, written in Prolog, against a catalogue of seven household object classes. When nothing matches, the bridge asks the operator, asserts the corrected facts and calls KnowRob's export to write a class file for the new object.

Prepares Caffe training and test lists for five labelling schemes: 28 objects, 7 categories, shape, material and affordance, plus a 10 object variant for the self collected TUM-ICS set. prepareData_All.py, prepareDATASET.py
Extracts FC7 features for 25,831 training and 24,884 test images and writes them as MATLAB files for the GURLS classifiers. caffe_FC7_to_GURLS_train.py
Classifies live YARP frames with CaffeNet, prints the top five guesses and streams the FC7 vector to the classifier port. yarp2caffe_classify.py
Crops 640 by 480 frames to a centred square and tracks a coloured blob with the head, proportional gain 0.1 on two neck joints. cropImage.cpp, findLocation.cpp, lookAtLocation.cpp
Queries the ontology on material, shape and hasObjectUsedFor, and on no match runs a yes or no console dialogue, asserts three RDF triples and exports a new class. computablesCpp.cpp

Solution

The robot's knowledge grows while it works.

The decoupling is the whole idea. The network is never asked for a name. It is asked for properties, and the ontology carries the name. That is what lets a person add an object in one dialogue with no retraining, and it is what makes every answer traceable to explicit triples a reader can inspect.

The step that made the robot open ended. A person answers one yes or no question, three triples land in the knowledge base, and the next query resolves the object without any retraining of the network.

Two variants were built. Five fine tuned networks in parallel gave the headline numbers. The variant these repositories contain uses one ImageNet network and least squares classifiers on its features, and it scores lower on both halves. The unknown object rows are measured after one teaching dialogue.

Result

Reasoning carried the objects the network could not.

Known objects

85 → 98%

Unknown objects

0 → 65%

First contact

27 / 49

Training images

25,831

The first two cells compare the stand alone network with the same network plus ontology reasoning, measured in the thesis on the parallel CaffeNet variant. The unknown object figure is measured after the object has been taught once. On first contact the system succeeded in 27 of 49 trials, and the thesis records that it always fails on the very first attempt with a new object. None of these numbers were re run for this page.

Datasets: iCubWorld28, 28 objects in 7 categories, 25,831 training and 24,884 test images, and a self collected TUM-ICS set, reported as 11 objects in the thesis and implemented as 9 classes in these scripts. Code: icub-visual-recognition for data preparation, feature export, fine tuning and live classification, icub-visual-servoing for cropping and gaze control, and icub-semantic-reasoning for the ontology, the Prolog predicates and the ROS bridge.

Solution architecture

What it stands on, and what is new.

Inputs and dataimages and labels

iCubWorld28IIT dataset, 28 objects, 7 categories, 50,715 images

TUM-ICS setcollected on the robot, 33,000 images kept, 9 classes in the scripts

iCubSim camerasIIT platform, YARP video ports. Every camera port in these repositories is the simulator

label schemeshand assigned shape, material and affordance per object

256 by 256 crops over YARP

Models and algorithmswhat turns pixels into labels

BVLC CaffeNetAlexNet variant, ImageNet weights, FC7 features

fine tuned CaffeNets28 class and 7 class heads, weights not committed

GURLS in MATLABa MATLAB toolbox running a simple statistical classifier on the FC7 vectors

KnowRob and SWI-PrologOWL parsing, owl_has reasoning, class export

four labels on four YARP ports

Engineown code, Python 2, C++, Prolog

Caffe pipeline scriptsdata prep, feature export, evaluation, live classification

YARP image and gaze modulescrop, resize, blob detection, head velocity control

computablesCpp bridgereads the four ports, builds the query, runs the dialogue

Prolog predicatesattribute checks, table geometry, closed world asserts

object ontology21 classes, 18 individuals, materials, shapes, affordance events

a name, or a question for the operator

Outputs and interfaceswhere a person meets it

console dialogueyes or no teaching loop on the terminal

exported class filesone OWL file per newly learned object

yarpview and Matplotliblive image windows and filter visualisations

Built in this projectThird party

Sources: iCubWorld28 and the iCub platform are IIT work, and no dataset file is in these repositories. CaffeNet is Berkeley Vision under the BVLC model terms. GURLS is IIT and runs inside proprietary MATLAB. KnowRob is TUM and SWI-Prolog is BSD. The ontology reuses KnowRob and OpenCyc classes under CC BY 3.0. The three code repositories carry no licence file, and one Prolog file keeps an inline GPLv3 header inherited from KnowRob.

Why it matters

A perceiving model with an inspectable layer above it.

The project showed one working way past the closed class limit of a network. Asking the model for attributes and letting an ontology carry the name is what allowed a person to add an object in a single dialogue. It also made every answer traceable to explicit triples. The cost is visible in the thesis. The system succeeded in 27 of 49 first contact trials, queries returned ten to twenty candidates once many objects had been learned, and the five separate runtimes it depends on no longer install together on a current machine.

In five years

Vision language models already answer the open vocabulary question this thesis posed, and affordance words like hold, drink and clean now sit inside robot foundation models. What survives from 2016 is the division of labour. A learned model perceives, and a structured layer holds the facts a person can read, audit and correct. Household robots that a family teaches new objects to will need that second layer, for the same reason it existed here.

How it was built

Five runtimes, three languages, one loop.

Python

2,564lines

C++

380lines

Prolog and OWL

936lines

Tests

Commits

39on 13 days

Runtime deps

9listed below

All 2016 code, ontology and thesis written by hand. No agents existed. The three README files added later were written with Claude Code. Line counts exclude generated YARP bindings and committed build trees. The nine runtime components are Caffe, the CaffeNet weights, NumPy, SciPy, Matplotlib, YARP, GURLS on MATLAB, ROS with catkin, and KnowRob on SWI-Prolog. Measured from the repositories.

Both code repositories open with YARP exercises: a crop module, a blob detector and a head servo against the simulator. One day later the live path from camera through CaffeNet reached a first working version. Next came the seven category variant and label preparation for attributes and affordances. The reasoning package was later reinitialised and rewritten inside one week, closing with new Prolog code and the ontology structure. Discarded along the way: a resize module in favour of cropping, per object definitions in favour of per category ones, and a direct assertion query replaced by owl_has. The stack was harder than the model.

Algorithms and mathTransfer learning on FC7 features, network fine tuning, regularised least squares, and OWL reasoning with an explicit workaround for open world semantics. The contribution is the combination.

Systems integrationPython 2 Caffe, C++ YARP modules, MATLAB classifiers over ports, a ROS catkin node, and KnowRob on SWI-Prolog, all in one loop on the iCub simulator.

Data and domainHand labelled attributes and affordances across 28 classes, a self collected 33,000 image set of household objects, and an ontology of materials, shapes and affordance events.

Evaluation rigorThe thesis reports trial counts per table and separates first contact from post teaching accuracy. No automated test, no committed weights, and the results are not reproducible from these repositories.

Interface and productA yes or no console dialogue, live image windows and filter plots. No dashboard and no report.

Difficulty 16 of 25 · AdvancedRelative to one competent engineer building it alone

Success criteria

When this counted as working.

The robot names a known object more reliably with reasoning on top than the network manages alone.
An object outside the training classes gets a usable answer instead of a confident wrong label.
A person can teach a new object in one dialogue, with no retraining and no restart.
Every answer traces back to triples in the knowledge base that a reader can inspect.
The whole loop runs on the robot's own vision stack rather than on recorded files.

Metrics, KPIs, evals

Numbers from the thesis.

Condition	Network alone	With reasoning
Known objects, parallel CaffeNet variant	85%	98%
Unknown objects, parallel CaffeNet variant	0%	65%
Known objects, GURLS variant in these repos	14%	79%
Unknown objects, GURLS variant in these repos	0%	26%
First contact teaching trials	27 of 49 succeeded, 55%

Ranges in the thesis are reported per object group. The figures above take the upper end of each group. Unknown object accuracy is measured after the object has been taught once, so it describes retention rather than first sight.

What this does not do.

Run today. The ROS Indigo and Jade stack, KnowRob, Caffe and the iCub simulator of that era no longer install on a current machine, and reviving them is not the point of this page.
Reproduce its own numbers. Network weights, the MATLAB classifier side and the collected dataset are outside these repositories, and every figure here comes from the thesis.
Scale past a small ontology. Once many objects had been learned, a query could return ten to twenty candidates with no ranking between them.
Work without a person. The open ended part of the system depends on an operator answering a question at the console.