Pokémon TCG Scanner
Photograph a trading card and get back which card it is, which set it was printed in, and how much that answer can be trusted. Entirely on the phone, with no network call at any stage.
kit-v1 · 83 MB · Core ML encoder, the 20 394 card index and three fixtures
18 / 18 correct on real iPhone photographs20 394 cards indexed, across 170 setsThe encoder runs in 5.5 ms on the Neural EngineNo classifier: a new set is an index rebuild, never a retrainFrench cards, English index, and it does not care
The problem
Identifying a Pokémon card is not reading its name. 92 % of the 20 394 cards in the index share their name with another card: Pikachu has been printed 99 times, Eevee 63. What a collector actually needs is the printing, and the only thing separating two printings is a set symbol and a number in small type along the bottom edge.
The obvious build is a classifier with one class per card. That is 20 000 classes, a training run, and another training run every time a set comes out, which happens several times a year. The other obvious build reads the name with OCR and asks a web service, which works right up until you are standing in a card shop with no signal.
What it does
You photograph a card. It comes back with the card, the set it was printed in, and one of three confidence levels. There is no account and no upload: the encoder and the index of 20 394 cards ship inside the app, and the whole chain runs with the phone in aeroplane mode.
The answer is not always a full one, and it says so. When the printed number reads cleanly, or the top match is far enough ahead of the second, it names the card and the set outright. When two printings of one artwork are neck and neck, it names the card and admits it cannot pick the set. When nothing is ahead of anything, it offers the top five instead of guessing.
- Runs on
- iPhone, Neural Engine, no network call
- Index
- 20 394 cards, 170 sets, 20 MB on the device
- Encoder
- MobileCLIP2-S2, 512 dimensions, Core ML
- Bench
- 18 iPhone photographs, French cards
- New set
- Rebuild the index. The model never changes.
Architecture
- PhotographiPhone camera
Card, flat and upright
- Card, flat and uprightVision, homography
Card encoder, Printed number
- Card encoderCore ML, 512 dims
Index, on the phone
- Printed numberVision, bottom edge
Index, on the phone
- Index, on the phone20 394 vectors, 20 MB
Detection and rectification
Two of Apple's detectors propose the card's quadrilateral, a homography flattens it, and the position of the text decides which way up. A twelve megapixel photograph produces up to fifteen crop hypotheses.
The encoder
MobileCLIP2-S2 exported to Core ML: 256 by 256 pixels in, 512 numbers out. The only piece that had to be right to the pixel, and the only one that never changes.
The index
20 394 float16 vectors, L2 normalised, 20 MB shipped inside the app. The search is a single matrix product against all of them, which is exact and takes under two milliseconds.
Number and confidence
A second Vision pass reads the collector number off the bottom edge. Two margins on the ranking decide whether the app may name a set, name only the card, or name nothing.
Built with
On device
- Vision
- Two document detectors, the orientation read and the collector number read: four fifths of the cost of an identification
- Core ML
- The encoder on the Neural Engine. Letting Core ML pick its own compute unit costs nearly double.
- Accelerate
- Cosine search over 20 394 vectors as one matrix product, with no vector database to install
- expo-card-encoder
- The Swift port, wired into an Expo app as a native module beside the existing OCR path
Model and index
- MobileCLIP2-S2
- The image encoder Apple designed for the Neural Engine, used frozen: nothing here is fine tuned
- open_clip and PyTorch
- Builds the reference embedding for every card scan, six minutes on an M3 Pro
- coremltools
- The export, and the parity check that compares the exported model against PyTorch on every fixture
- pokemon-tcg-data
- Card metadata and the high resolution scans the index is built from, 5.5 GB of them
Tooling
- integration-kit
- Model, index, a reference Python implementation and a porting guide, published as one release
- expected.json
- Three photographs with their expected card, confidence and margins, so a port is validated in stages
- refresh_index.py
- Reports the sets released since the index was last built, which is the only maintenance this has
Measured
Accuracy is 18 real iPhone photographs of French cards, run against the English index through the exported Core ML model. Conditions were deliberately ordinary: backlight, sleeves, cluttered backgrounds, tilted cards. Latency is an average over 10 to 50 iterations after warm up, measured per call rather than per scan, on an iPhone 13 Pro and a MacBook Pro M3 Pro.
One identification, stage by stage
| Stage | iPhone 13 Pro | Mac M3 Pro |
|---|---|---|
| Detection, both detectors | 24 ms | 26 ms |
| Rectification | 6.4 ms | 2.9 ms |
| Orientation OCR | 11 ms | 9.5 ms |
| Geometric preprocessing | 1.2 ms | 1.0 ms |
| Embedding, Neural Engine | 5.5 ms | 4.1 ms |
| Search over 20 394 vectors | 1.9 ms | 0.6 ms |
| Collector number OCR | 65 ms | 60 ms |
The model is about 3 % of an identification and Vision's two OCR passes about 79 %. Reaching for a lighter model optimises the thing that is not the constraint.
What the bench says
| Measurement | Result |
|---|---|
| Correct identifications, 18 photographs | 18 / 18 |
| Core ML parity against PyTorch, mean | 0.9958 |
| Core ML parity, worst fixture | 0.9891 |
| Cards in the index | 20 394 |
| Sets covered | 170 |
| Cards sharing a name with another card | 92 % |
| Cards identifiable by printed number alone | 38 % |
Those last two lines are why confidence has two levels rather than one: nearly all the confusion lives inside a single card name.
What the app is allowed to claim
| Condition | Level |
|---|---|
| The collector number was read off the card | edition |
| Edition margin at or above 0.03 | edition |
| Name margin at or above 0.04 | name |
| Neither margin clears | uncertain |
Edition names the card and the set outright. Name gives the card and says the set is unconfirmed. Uncertain offers the top five, or asks for a steadier shot. Edition margin is the first candidate minus the second, name margin the first minus the first one bearing a different card name.
What this does not prove
Eighteen photographs from one player, all of recent cards, while 47 % of the index predates Sun and Moon, where borders and footer layouts differ substantially. The confidence thresholds are calibrated on that same set. Apple's detectors also need to see four closed edges: on a photograph of five cards where four overlapped, only one was isolated. Coverage is a count, it is not evidence.
Under the hood
01There is no classifier, and that is the whole design
Detect, embed, search. The model turns a card into 512 numbers and the answer is whichever of 20 394 stored vectors is closest. A new set costs an index rebuild and no GPU time at all, where a 20 000 class classifier would cost a training run several times a year, forever. The price is that the system can only ever return a card that is in the index, so the pipeline was built to notice that case rather than hide it: a number that reads cleanly and belongs to nothing in the index is reported as an unknown card, not attributed to whatever ranked first.
02The absolute similarity score discriminates nothing
Every card in the game shares a layout, so two entirely unrelated cards already sit at about 0.79. The useful range is 0.79 to 1, not 0 to 1, and the worst failure on the bench carried the second highest absolute score of the whole set. What carries information is the gap between the first candidate and the next, which is why nothing in the app ever thresholds on a score or shows one to a user.
margin_id = hits[0].score - hits[1].score
top_name = hits[0].card["name"]
margin_name = next(
(hits[0].score - h.score for h in hits[1:] if h.card["name"] != top_name),
margin_id,
)
if band_verdict == "ocr":
return Confidence("edition", "numéro lu", margin_id, margin_name)
if margin_id >= FIRM_ID_MARGIN:
return Confidence("edition", "similarité", margin_id, margin_name)
if margin_name >= FIRM_NAME_MARGIN:
return Confidence("nom", "similarité", margin_id, margin_name)
return Confidence("incertain", "similarité", margin_id, margin_name)The levels read edition, nom and incertain in the source. Both margins are taken before the printed number promotes anything: on the reordered list they come out negative.
03The embedding must never decide which way up the card is
An upside down card still looks like a card, and the flipped crop can out-score the upright one against the wrong answer: measured at 0.834 for a wrong card against 0.788 for the right one. So orientation is settled before the encoder ever runs, on where the text sits, because a card the right way up keeps its attack text and its footer in the lower half. Simply discarding the flipped variant would be cheaper and would break the day someone photographs a card head to tail.
04The preprocessing geometry has to match to the pixel
The index was built by scaling each reference scan's shortest side to 256 with antialiased bilinear resampling, then cropping the centre, which deliberately trims the top and bottom off a portrait card. Squashing an image straight to 256 by 256 instead drops measured parity to 0.65, and the symptom is not a crash but plausible, confident, wrong answers. On the Swift side there is no exact equivalent of the Python filter, so the four candidates were measured against a card that is itself in the index: 0.9938 for medium, 0.9926 for high, 0.9854 for low, 0.9656 for none. Medium wins, and it is also the bilinear one.
Gallery
Try it
Not from a browser, and not from a store.
The iOS app that uses this is a development build in a separate repository, so there is nothing to install. What is published is the working part: the Core ML encoder and the index, in one archive, with a reference Python implementation of the whole pipeline and three photographs whose expected answers are written down. Clone the repository, build the environment, and the bench runs on your machine.
Rebuilding the index from scratch needs 5.5 GB of card images and about twenty minutes on an M3. The archive exists so you do not have to.
What broke
1.8 seconds of a 2.3 second scan were a build setting
The Swift port felt unusably slow and the profile blamed the hand written pixel loops, at 124 ms and 114 ms per crop. They were not slow: they were compiled unoptimised, because Debug is what you run while porting. Framework calls into Vision and Core ML are unaffected by that, so nothing else looked wrong. With -O the same loops fall under a millisecond. Any port with its own pixel arithmetic now gets -O in Debug too, from the first build rather than the day the numbers stop making sense.
EXIF rotation applied on one side and not the other
Whatever finds the card and whatever crops it have to work in the same frame. Ignoring the orientation flag on both sides is perfectly valid; applying it on one side only points the detected outlines at pixels that are not there. The result was black crops and similarity scores around 0.4, which reads exactly like a bad model, and sent me looking at the encoder for hours. The rule taken from it: when a score collapses, check the geometry before the model.
Shipping a reference implementation with the model
The release is not just an encoder and an index. It carries a Python implementation of the whole pipeline, three photographs, and the card, confidence and margins each one should produce. Writing those fixtures felt like overhead next to getting the thing working. They are what made the Swift port debuggable in stages: encode the reference card, confirm it comes back at about 0.995, and only then look at anything downstream. Without them a port fails as one opaque wrong answer.
A card the visual search misses stays missed
On one photograph the correct card was not in the top ten at all, 0.089 behind an unrelated Trainer, while its printed number read cleanly three times running. Promotion can only reorder candidates the search returned, so the port now resolves a uniquely printed number against the whole index instead. That reaches the 38 % of cards whose number and total pair is unique. For the other 62 % the pair belongs to up to nine cards and designates none of them, and there is nothing in this architecture to fall back on.