Every locator strategy assumes the application will tell the truth about itself. Mobile apps often do not. Resource ids change between builds, accessibility labels are missing, and some screens are drawn straight onto a canvas with no element tree at all. This project was built for the case where the properties are gone and the pixels are all you have.
The problem with properties
An Appium test finds an element by its internal properties: an id, an accessibility label, an XPath through the view hierarchy. Those are the first things to change when a team refactors, and canvas-drawn or game-engine interfaces never had them. When the properties go, the test fails with “element not found” on a screen where a person would have tapped the right button without hesitating. The gap between those two outcomes is the whole project: make the test see what the person sees.
The first version integrated deep-learning image classification with Appium, so a test could ask for “the search icon” and get a match from a screenshot rather than from the tree. It kept working through property changes that would have broken every conventional locator on the page, which was enough to prove the idea and enough to show its limits: classification tells you what something is, not where it is.
Building the dataset
No public dataset looked like real mobile interfaces, so I built one. More than a hundred classes of mobile UI element, curated and labelled by hand from screenshots: buttons, inputs, toggles, tabs, checkboxes, icons, and the states that matter for a test, because a checked box is not the same target as an unchecked one.
Labelling your own data teaches you things a downloaded dataset hides. Box consistency matters more than box count; a hundred tightly drawn examples beat three hundred loose ones. Ambiguous classes are a labelling problem before they are a model problem: if two people would disagree about whether something is a button or a link, the model will disagree with itself. Most of the accuracy gains came from tightening the labelling rules, not from touching the network.
Training on a small dataset
A hundred-odd classes with a few examples each is far too little to train a detector from nothing. Transfer learning closes that gap: start from a network that already knows edges, shapes and text from natural images, and retrain the head on interface elements. The move from classification to object detection with YOLO meant a single pass over the screenshot returned every candidate with a bounding box, and non-maximum suppression collapsed the overlapping boxes a detector produces around one target.
Evaluation used two numbers on purpose. Mean average precision says whether the detector finds the right things. Intersection over union says whether the box is tight enough to tap in the right place. A detector with high mAP and loose boxes still taps the wrong pixel, so IoU is the number that predicts whether a test will pass.
Fast enough to be worth using
A model that takes seconds per screen makes a suite slower than the one it replaces, and a slow safety net gets switched off. Improving Tap-by-Image detection speed was mostly about doing less work: scale the screenshot to the model’s input once, search where a target can plausibly be, and reuse what the previous step already computed when the screen has not changed.
Finding it twice
The hard case was never missing an element. It was finding it twice. A screen with five identical “Add” buttons produces five correct matches, and taking the first is a coin toss that fails silently. Duplicate-element handling disambiguates by position and context when it can, and when it cannot it refuses and reports the ambiguity instead of guessing. A test that stops and says “there are five of these” is more useful than one that taps the wrong one and passes.
Where it sits now
This vision layer is one of the ways the agent resolves a target when the markup offers nothing stable. The technique is written up in more depth in Finding UI elements when the DOM won’t cooperate, and what happens after an element moves is the subject of Self-healing locators.
What it taught
- Label the data yourself at least once. The labelling rules are where most of the accuracy lives, and you cannot write them without doing it.
- Evaluate on the metric that matches the action. For tapping, that is IoU. A detector can be right about what and wrong about where.
- Speed is part of the model. A detector nobody enables because it is slow has an accuracy of zero.
- Report ambiguity, do not resolve it by luck. Five matches is a finding, not a failure.