Every automation framework eventually meets an element it cannot find. Not because it is missing from the screen, but because nothing stable points at it: an ID that regenerates on every build, a class name minified into noise, a component that looks identical on iOS and Android and shares none of the same internals. The DOM is the map, and sometimes the map is wrong.

Pixels as the stable interface

What a user sees does not change when the internal properties do. That is the whole idea. If a model can find a button by what it looks like, the test survives the refactor that would have broken every selector.

The implementation is object detection with YOLO, integrated with Appium so a test step can identify its target visually and act on it. The model’s job is to return a bounding box. The framework’s job is to turn that box into a tap.

Small data, transfer learning

The dataset was a hand-labelled corpus of a little over a hundred mobile UI elements. That is small by vision standards, and training from scratch on it would not have worked. Transfer learning did: start from a detector that already understands edges, shapes and layout, then retrain it on the specific vocabulary of a mobile interface. The labelling was the slow part and worth doing myself, because the labels define what the model believes a button is.

Ambiguity is the real failure

The first working detector found elements. It also found too many. Two identical buttons, a target that matched in three places, overlapping boxes around a single control. Non-maximum suppression handles the overlapping boxes. It does not decide which of two legitimately identical elements the test meant. Duplicate-element handling had to be built on top, so that an ambiguous match fails loudly instead of tapping the wrong thing quietly. A wrong tap that passes is worse than a missed tap that fails.

Measuring it

mAP tells you how good the detector is overall. IoU tells you whether a particular box is good enough to act on. For a test step those are different questions: a box that overlaps the true element by half can score acceptably in aggregate and still land the tap on the wrong pixel. Set the IoU threshold from what a tap needs, not from what the benchmark rewards.

Speed matters as much as accuracy, because detection runs inside a test step and every step runs many times. Making tap-by-image detection faster on the page was as much a part of the work as making it right.

Where a vision model beats a selector, and where it does not

  • It wins when the internals are unstable and the visual is not: cross-platform components, generated IDs, obfuscated class names.
  • It loses on text-heavy targets. A label is easier to read than to recognise as a shape, so OCR is the better signal there.
  • It is unnecessary when nothing has moved. Pixel comparison and template matching are far cheaper for that case.

The self-healing layer combines all of these rather than betting on one: pixel comparison and template matching first, OCR where the target is anchored by text, and a weighted variant of longest-common-subsequence to re-anchor a locator once an element has moved. The detector is one signal among several.

The lesson is the same one the agent work taught: prepare the signal, use the cheapest method that clears the bar, and fail loudly when the answer is ambiguous.