How Chess Acuity reads a chessboard from a screenshot

This page explains how that works, end to end.

The short version

1. You frame the board. A crop box with an 8×8 grid overlay lets you line the frame up exactly with the board’s edges, and you tell the app which side is at the bottom of the picture.
2. The app slices the crop into 64 tiles, one per square.
3. A tiny neural network classifies each tile as one of 13 things: empty, or one of the six white or six black pieces.
4. The 64 answers are assembled into a FEN (the standard text notation for a chess position), checked for legality, and opened in a board editor so you can verify — and fix — anything before analysing.

Alongside this, the app computes an honest confidence score, and refuses to show you a guessed position when the crop doesn’t actually look like a chessboard.

Step 1: You do the framing

Many board-recognition systems try to find the board in the image automatically. Chess Acuity deliberately doesn’t: you drag a crop box over the screenshot instead. The box has draggable corners, pinch-to-resize, and a thin 8×8 grid drawn inside it, so you can align the grid lines with the board’s own square boundaries.

This matters because of what happens next: the app divides the crop into an 8×8 grid of equaltiles. If the frame is exactly on the board’s edges, every tile contains exactly one square. If the frame is off by half a square, every tile straddles two squares and recognition would be garbage. Letting you see and control the alignment is simpler, more transparent, and more reliable than guessing — and screenshots of digital boards have crisp, straight edges that are easy to line up by hand.

You also pick which side is at the bottom of the picture (White or Black). Screenshots taken while playing Black show the board “upside down”, and the recogniser needs to know this to label the squares correctly.

Step 2: Slicing into tiles

A small native module (written in Swift) crops the board region out of the screenshot, divides it into 64 cells, and resizes each cell to a 32×32-pixel RGB image. Cell boundaries are snapped to whole pixels so adjacent tiles fit together perfectly with no gaps or overlaps.

The output is just raw pixel data for 64 small images — the input the neural network was trained on.

Step 3: A small neural network names each square

Each 32×32 tile goes through a convolutional neural network (CNN) — the kind of model used for image classification — with 13 possible answers:

empty square, or white/black pawn, knight, bishop, rook, queen, king

The network is deliberately tiny: three convolutional blocks, about 100 KB once converted to TensorFlow Lite. That’s small enough to ship inside the app and to classify all 64 squares in a few milliseconds on the phone’s CPU. It can be this small because the task is heavily constrained: digital board tiles are axis-aligned, evenly lit, and drawn from a limited family of piece designs — nothing like general-purpose photo recognition.

How it was trained.The training data is synthetic: millions of tiles are generated by compositing openly-licensed chess piece sets (the classic “cburnett” set and other open-source sets from Lichess) onto a wide range of board colour themes, with realistic variations mixed in — move highlights, last-move markers, coordinate labels, slight colour jitter, and the anti-aliasing you get when a screenshot is scaled. Because the data is generated, every tile comes with a perfect label for free. The model is then evaluated against real screenshots that were labelled by hand, which is the accuracy number that actually matters. Only openly-licensed artwork is used for training.

Step 4: From 64 answers to a chess position

The 64 classifications form an 8×8 grid of piece letters. Turning that into a FEN is done in ordinary code, not by the model — the model only ever has to answer “what’s on this one square?”, which eliminates a whole class of transcription errors (miscounted empty squares, dropped ranks) by construction.

The assembly step also:

• Flips the board in codeif you said Black was at the bottom, so the final position is always in standard orientation. (Rotating a board “mentally” is exactly the kind of thing recognition models get wrong, so it is never asked to.)
• Validates the position with a chess library: exactly one king per side, and a position the rules can accept. If validation fails — say a king was misread — the app still shows you everything it recognised, flags the problem, and drops you into the editor to fix it, rather than showing an empty board.

Knowing when to say “I’m not sure”

A classifier will always give an answer, even for a photo of your cat. So Chess Acuity computes two independent trust signals and multiplies them:

Certainty — does the model trust its own answers?Each classification comes with a probability. The app combines the 64 probabilities using a geometric mean, so one very unsure square drags the score down instead of being averaged away. Individual squares below a threshold are highlighted in the editor as “double-check this square”.

Structure — does the crop even look like a chessboard? Neural networks are famously overconfident on inputs unlike anything they were trained on: a misaligned crop can produce confident nonsense. So the app also checks the crop itself, without the neural network. It estimates each tile’s background colour from its corners (corners are rarely covered by a piece), splits the 64 tiles into the two checkerboard colour groups, and measures how cleanly they separate into a light-vs-dark pattern. A real, well-framed board separates almost perfectly; off-board content or a frame offset from the grid does not. In testing this score is essentially binary — near 100% for a correctly framed board, near 0% otherwise — which makes it a reliable gate.

The headline confidence shown after import is the product of the two: a crop must both look like a board and be classified confidently to score high. If the structure score fails the gate, the app refuses to open a possibly-hallucinated position and asks you to adjust the frame instead, showing both sub-scores so you can see what went wrong.

Verify, then analyse

However confident the recogniser is, the final authority is you. Every import lands in the board editor with the recognised position on the board, any uncertain squares highlighted, and the side to move ready to confirm. One tap fixes a square; then Stockfish — also running entirely on your device — takes it from there.

Privacy, in one paragraph

Board recognition, engine analysis, and move explanations all run on your device. The screenshot you import is read locally, processed locally, and never transmitted. Chess Acuity has no accounts, no analytics, and makes no network requests. The app is open source, so all of this can be verified in the code.