Pokémon TCG Scanner
Photograph a trading card and get back which card it is, which set it was printed in, and how much that answer can be trusted. Entirely on the phone, with no network call at any stage.
kit-v5 · 86 MB · Core ML encoder, the 20 512 card index and three fixtures
35 / 39 correct on real photographs, three collections20 512 cards indexed, 176 sets, none missing an imageThe encoder runs in 5.5 ms on the Neural EngineNo classifier: a new set is an index rebuild, never a retrainFrench cards, English index, and it does not carePublic on GitHub, MIT licensed
The problem
Identifying a Pokémon card is not reading its name. 92 % of the 20 512 cards in the index share their name with another card: Pikachu has been printed 99 times, Eevee 63. What a collector actually needs is the printing, and the only thing separating two printings is a set symbol and a number in small type along the bottom edge.
The obvious build is a classifier with one class per card. That is 20 000 classes, a training run, and another training run every time a set comes out, which happens several times a year. The other obvious build reads the name with OCR and asks a web service, which works right up until you are standing in a card shop with no signal.
What it does
You photograph a card. It comes back with the card, the set it was printed in, and one of three confidence levels. There is no account and no upload: the encoder and the index of 20 512 cards ship inside the app, and the whole chain runs with the phone in aeroplane mode.
The answer is not always a full one, and it says so. When the printed number reads cleanly, or the top match is far enough ahead of the second, it names the card and the set outright. When two printings of one artwork are neck and neck, it names the card and admits it cannot pick the set. When nothing is ahead of anything, it offers the top five instead of guessing.
- Runs on
- iPhone, Neural Engine, no network call
- Index
- 20 512 cards, 176 sets, 21 MB on the device
- Encoder
- MobileCLIP2-S2, 512 dimensions, Core ML
- Bench
- 46 photographs, three collections, several people
- New set
- Rebuild the index. The model never changes.
Architecture
- PhotographiPhone camera
Card, flat and upright
- Card, flat and uprightVision, homography
Card encoder, Printed number
- Card encoderCore ML, 512 dims
Index, on the phone
- Printed numberVision, bottom edge
Index, on the phone
- Index, on the phone20 512 vectors, 21 MB
Detection and rectification
Two of Apple's detectors propose the card's quadrilateral, a homography flattens it, and the position of the text decides which way up. A twelve megapixel photograph produces up to fifteen crop hypotheses.
The encoder
MobileCLIP2-S2 exported to Core ML: 256 by 256 pixels in, 512 numbers out. The only piece that had to be right to the pixel, and the only one that never changes.
The index
20 512 float16 vectors, L2 normalised, 21 MB shipped inside the app. The search is a single matrix product against all of them, which is exact and takes under two milliseconds.
Number and confidence
A second Vision pass reads the collector number off the bottom edge. Two margins on the ranking decide whether the app may name a set, name only the card, or name nothing.
Built with
On device
- Vision
- Two document detectors, the orientation read and the collector number read: four fifths of the cost of an identification
- Core ML
- The encoder on the Neural Engine. Letting Core ML pick its own compute unit costs nearly double.
- Accelerate
- Cosine search over 20 512 vectors as one matrix product, with no vector database to install
- expo-card-encoder
- The Swift port, wired into an Expo app as a native module beside the existing OCR path
Model and index
- MobileCLIP2-S2
- The image encoder Apple designed for the Neural Engine, used frozen: nothing here is fine tuned
- open_clip and PyTorch
- Builds the reference embedding for every card scan, six minutes on an M3 Pro
- coremltools
- The export, and the parity check that compares the exported model against PyTorch on every fixture
- pokemon-tcg-data
- Card metadata and the high resolution scans the index is built from, 5.5 GB of them
Tooling
- integration-kit
- Model, index, a reference Python implementation and a porting guide, published as one release
- expected.json
- Three photographs with their expected card, confidence and margins, so a port is validated in stages
- refresh_index.py
- Reports the sets released since the index was last built, which is the only maintenance this has
Plate 1: the space of cards
Every card becomes a point
The model turns a card image into 512 numbers. Two cards that look alike get numbers that sit close together. Recognising a photograph is then a search for the nearest point, not a choice between 20 512 labels.
This is that space, flattened to two dimensions. Colour follows the eras, from 1999 to today. Click a point to see the eight cards the model judges closest to it.
Loading 20 455 points
The eras do not separate. You would expect the years to cluster, and they do not: the model encodes the illustration, not the layout and not the date. A 1999 Charizard sits closer to a 2024 Charizard than to the card printed next to it.
The neighbourhoods shown are not the neighbourhoods of this 2D projection. They are computed on the original 512 dimensions and then carried here, without which the page would be showing the geometry of the drawing rather than the geometry of the model.
Plate 2: where the time goes
The model is not the bottleneck
Seven stages separate the photograph from the verdict. Measured on an iPhone 13 Pro, they do not cost what you would think.
- Find the card in the photograph24 msVNDetectDocumentSegmentation
- Flatten the quadrilateral6.4 msCIPerspectiveCorrection
- Decide which way up it reads11 msVNRecognizeText · fast
- Crop to the model's format1.2 msresize + centre crop
- Encode into 512 numbers5.5 msCore ML · Neural Engine
- Compare against 20 512 cards1.9 msmatrix product · Accelerate
- Read the printed number65 msVNRecognizeText · accurate
- 87.0 %
- Text recognition
- 8.3 %
- Geometry and search
- 4.8 %
- The model
The neural network is 4.8 % of the work. Apple's text recognition is 87 % of it. Reaching for a lighter model to go faster would be optimising the thing that constrains nothing.
It is this split, and not the speed of the model, that dictates the architecture of a live scan: detect on every frame, identify once per card, and only read the printed number in the background.
Plate 3: a real failure
When resemblance gets it wrong
One photograph of Inkay, taken three times in a row. All three times the similarity ranking is led by an unrelated Trainer card, and Inkay is nowhere in the top ten.
Ranked by similarity, before the number is read
verdict: uncertain
- 1Beginning DoorDP Black Star Promos · 54/561st
- 2Ultimate ZoneDP Black Star Promos · 55/56−0.014
- 3ExcadrillObsidian Flames · 147/197−0.031
- 4Arceus LV.XPlatinum: Arceus · 94/99−0.039
- 5GoldeenPitch Black · 13/84−0.044
- 6InkayPitch Black · 51/84not in the top ten
After the printed number is read
verdict: exact edition
- 1InkayPitch Black · 51/84number read
- 2Ultimate ZoneDP Black Star Promos · 55/56+0.097
- 3Beginning DoorDP Black Star Promos · 54/56+0.089
- 4ExcadrillObsidian Flames · 147/197+0.047
- 5GoldeenPitch Black · 13/84+0.042
The footer carries 051/084. Exactly one card out of 20 512 bears that number and total pair, and reading that tiny strip of text (1.7 % of the height of the card) is enough to settle it. The positive gaps read as an admission: those cards resembled the photograph more than Inkay did. Printed proof beats resemblance, and it is the only case where the visual ranking is deliberately overturned.
That rescue only covers 38 % of cards. The number and total pair has to be unique across the whole index, which is true of 7 773 cards. For the rest, a card the resemblance misses stays missed.
Measured
Accuracy is 18 real iPhone photographs of French cards, run against the English index through the exported Core ML model. Conditions were deliberately ordinary: backlight, sleeves, cluttered backgrounds, tilted cards. Latency is an average over 10 to 50 iterations after warm up, measured per call rather than per scan, on an iPhone 13 Pro and a MacBook Pro M3 Pro.
One identification, stage by stage
| Stage | iPhone 13 Pro | Mac M3 Pro |
|---|---|---|
| Detection, both detectors | 24 ms | 26 ms |
| Rectification | 6.4 ms | 2.9 ms |
| Orientation OCR | 11 ms | 9.5 ms |
| Geometric preprocessing | 1.2 ms | 1.0 ms |
| Embedding, Neural Engine | 5.5 ms | 4.1 ms |
| Search over 20 512 vectors | 1.9 ms | 0.6 ms |
| Collector number OCR | 65 ms | 60 ms |
The model is about 3 % of an identification and Vision's two OCR passes about 79 %. Reaching for a lighter model optimises the thing that is not the constraint.
What the bench says
| Measurement | Result |
|---|---|
| Correct identifications, of 39 photographs with an answer | 35 / 39 (90 %, CI95 76-96) |
| Correct card somewhere in the top five | 37 / 39 |
| Correct refusals, of 7 photographs with no answer | 6 / 7 |
| Core ML parity against PyTorch, mean | 0.9958 |
| Core ML parity, worst fixture | 0.9891 |
| Cards in the index | 20 512 |
| Sets covered | 171 |
| Cards sharing a name with another card | 92 % |
| Cards identifiable by printed number alone | 38 % |
Those last two lines are why confidence has two levels rather than one: nearly all the confusion lives inside a single card name.
What the app is allowed to claim
| Condition | Level |
|---|---|
| The collector number was read off the card | edition |
| Edition margin at or above 0.03 | edition |
| Name margin at or above 0.04 | name |
| Neither margin clears | uncertain |
Edition names the card and the set outright. Name gives the card and says the set is unconfirmed. Uncertain offers the top five, or asks for a steadier shot. Edition margin is the first candidate minus the second, name margin the first minus the first one bearing a different card name.
What this does not prove
Forty-six photographs, three collections, several people, in ordinary conditions: backlight, sleeves, busy backgrounds, tilted cards, a whole batch shot in landscape. Thirty-nine of them have a right answer, and thirty-nine cannot demonstrate more than their own lower bound: even a clean sweep would cap at 91 %, so any figure quoted here without its interval is an overstatement. The confidence thresholds were fixed before the second batch arrived and have not moved, which makes that batch a hold-out for them; the detection filters were tuned while looking at it, and the third batch gives them theirs. 47 % of the index still predates Sun and Moon, where borders and footer layouts differ substantially. Apple's detectors need to see four closed edges: on a photograph of five cards where four overlapped, only one was isolated. Coverage is a count, it is not evidence.
Under the hood
01There is no classifier, and that is the whole design
Detect, embed, search. The model turns a card into 512 numbers and the answer is whichever of 20 512 stored vectors is closest. A new set costs an index rebuild and no GPU time at all, where a 20 000 class classifier would cost a training run several times a year, forever. The price is that the system can only ever return a card that is in the index, so the pipeline was built to notice that case rather than hide it: a number that reads cleanly and belongs to nothing in the index is reported as an unknown card, not attributed to whatever ranked first.
02The absolute similarity score discriminates nothing
Every card in the game shares a layout, so two entirely unrelated cards already sit at about 0.79. The useful range is 0.79 to 1, not 0 to 1, and the worst failure on the bench carried the second highest absolute score of the whole set. What carries information is the gap between the first candidate and the next, which is why nothing in the app ever thresholds on a score or shows one to a user.
margin_id = hits[0].score - hits[1].score
top_name = hits[0].card["name"]
margin_name = next(
(hits[0].score - h.score for h in hits[1:] if h.card["name"] != top_name),
margin_id,
)
if band_verdict == "ocr":
return Confidence("edition", "numéro lu", margin_id, margin_name)
if margin_id >= FIRM_ID_MARGIN:
return Confidence("edition", "similarité", margin_id, margin_name)
if margin_name >= FIRM_NAME_MARGIN:
return Confidence("nom", "similarité", margin_id, margin_name)
return Confidence("incertain", "similarité", margin_id, margin_name)The levels read edition, nom and incertain in the source. Both margins are taken before the printed number promotes anything: on the reordered list they come out negative.
03The embedding must never decide which way up the card is
An upside down card still looks like a card, and the flipped crop can out-score the upright one against the wrong answer: measured at 0.834 for a wrong card against 0.788 for the right one. So orientation is settled before the encoder ever runs, on where the text sits, because a card the right way up keeps its attack text and its footer in the lower half. Simply discarding the flipped variant would be cheaper and would break the day someone photographs a card head to tail.
04The preprocessing geometry has to match to the pixel
The index was built by scaling each reference scan's shortest side to 256 with antialiased bilinear resampling, then cropping the centre, which deliberately trims the top and bottom off a portrait card. Squashing an image straight to 256 by 256 instead drops measured parity to 0.65, and the symptom is not a crash but plausible, confident, wrong answers. On the Swift side there is no exact equivalent of the Python filter, so the four candidates were measured against a card that is itself in the index: 0.9938 for medium, 0.9926 for high, 0.9854 for low, 0.9656 for none. Medium wins, and it is also the bilinear one.
Gallery
Try it
Not from a browser, and not from a store.
The iOS app that uses this is a development build in a separate repository, so there is nothing to install. What is published is the working part: the Core ML encoder and the index, in one archive, with a reference Python implementation of the whole pipeline and three photographs whose expected answers are written down. Clone the repository, build the environment, and the bench runs on your machine.
Rebuilding the index from scratch needs 5.5 GB of card images and about twenty minutes on an M3. The archive exists so you do not have to.
What broke
1.8 seconds of a 2.3 second scan were a build setting
The Swift port felt unusably slow and the profile blamed the hand written pixel loops, at 124 ms and 114 ms per crop. They were not slow: they were compiled unoptimised, because Debug is what you run while porting. Framework calls into Vision and Core ML are unaffected by that, so nothing else looked wrong. With -O the same loops fall under a millisecond. Any port with its own pixel arithmetic now gets -O in Debug too, from the first build rather than the day the numbers stop making sense.
EXIF rotation applied on one side and not the other
Whatever finds the card and whatever crops it have to work in the same frame. Ignoring the orientation flag on both sides is perfectly valid; applying it on one side only points the detected outlines at pixels that are not there. The result was black crops and similarity scores around 0.4, which reads exactly like a bad model, and sent me looking at the encoder for hours. The rule taken from it: when a score collapses, check the geometry before the model.
Shipping a reference implementation with the model
The release is not just an encoder and an index. It carries a Python implementation of the whole pipeline, three photographs, and the card, confidence and margins each one should produce. Writing those fixtures felt like overhead next to getting the thing working. They are what made the Swift port debuggable in stages: encode the reference card, confirm it comes back at about 0.995, and only then look at anything downstream. Without them a port fails as one opaque wrong answer.
A Japanese card can only ever be refused
The index is built on Western printings, so a Japanese card is unidentifiable by construction: its artwork is in there, but every entry carrying it is a different printing of it. Two separate paths still reached a firm verdict, and both were wrong. On a Japanese 033/100 read without a single error, the one digit tolerance promoted a Wigglytuff 13/100 from an unrelated English set, seven places down, because the total happened to match; that tolerance now applies only to the top five candidates, which is what it was for. And on three of seven Japanese photographs, similarity alone came back firm on a Western printing while the banner read 041/100 against a candidate numbered 159; a read total that contradicts the candidate now forbids a firm verdict outright. The card still cannot be named. It can no longer be named wrongly.
A card the visual search misses stays missed
On one photograph the correct card was not in the top ten at all, 0.089 behind an unrelated Trainer, while its printed number read cleanly three times running. Promotion can only reorder candidates the search returned, so the port now resolves a uniquely printed number against the whole index instead. That reaches the 38 % of cards whose number and total pair is unique. For the other 62 % the pair belongs to up to nine cards and designates none of them, and there is nothing in this architecture to fall back on.