How AI Food Recognition Works: From Pixel to Plate

When you point your phone at lunch and an app spits back '612 kcal, 32 g protein, 71 g carbs, 22 g fat,' it can feel like sleight of hand. It is not. The number is the output of a small assembly line of models, each doing one job, and each capable of being wrong in its own characteristic way. If you understand the assembly line, you can spot when the answer is suspicious without having to weigh your food.

This piece is a walkthrough of that assembly line, written for people who use these apps rather than people who build them. We will use the architecture of a typical 2026-era AI nutrition app — the kind of pipeline you find in NutriShot AI, Cal AI, SnapCalorie, and others — as the running example.

Stage 1: Detection — what is on the plate?

The first model takes the raw image and proposes bounding boxes around things it thinks are food. Modern apps use a transformer-based vision backbone (often a derivative of ViT or DINOv2) fine-tuned on tens of millions of food photos. The output is a list: 'rice (96%), chicken thigh (89%), broccoli (94%), unknown sauce (61%).'

Detection is the most accurate stage of the pipeline. For common dishes the top-1 accuracy is typically above 95%. The errors that survive into your diary almost never come from this step — they come from the steps after it.

Stage 2: Segmentation — drawing around each item

A bounding box is a rectangle. To estimate how much rice is on the plate you actually need the outline of the rice — the polygon. Segmentation models produce a per-pixel mask: 'these pixels are rice, those pixels are chicken, these pixels are plate.' Most apps now use the Segment Anything family of models, fine-tuned on food.

Segmentation is where mixed dishes get tricky. A grain bowl with quinoa underneath roasted vegetables and a tahini drizzle is genuinely hard to disentangle, because the camera only sees the top layer. Apps handle this either by guessing the hidden volume from typical recipes or by asking you to confirm.

Stage 3: Portion estimation — the hardest step

Now the system has labelled regions. It needs to turn pixels into grams. There are three common strategies, and most apps use a blend.

• Plate geometry: assume a standard plate diameter and compute the food's projected area relative to it.
• Reference objects: use a known item in the frame (a phone, a credit card, a fork) to establish scale.
• Monocular depth: a separate neural network estimates the 3D shape of the food from a single 2D image, then integrates volume across the mask.

Monocular depth is the technology that has done the most to push portion estimation forward in the last 18 months. Models like Apple's depth API on newer iPhones, plus open-source Marigold and Depth Anything variants, give a usable depth map from a single photo. Combined with segmentation, you can estimate volume; combined with a per-food density table, you can estimate mass; combined with the database lookup, you can estimate calories.

Stage 4: Nutrient mapping — the database lookup

Once each item has a label and a mass, the app looks up nutrients in a database. The most common backbones are USDA's FoodData Central (FDC) and the Food and Nutrient Database for Dietary Studies (FNDDS) in the US, with regional supplements like CIQUAL in France or NEVO in the Netherlands. A few apps build proprietary databases on top, particularly for restaurant items.

The database choice matters more than people assume. FNDDS contains thousands of mixed-dish entries (e.g. 'beef stir-fry, with vegetables, with rice') that are statistically representative of how Americans actually eat, which produces realistic averages but can mask large variation. Raw-ingredient databases are more transparent but force the app to reconstruct recipes, which introduces its own error.

For micronutrients, database quality determines everything. If the database does not have a magnesium value for tempeh, no amount of vision-model improvement will fix it. Apps that surface 20+ micronutrients (Cronometer is the long-standing example, with several newer photo-based apps now joining it) have invested heavily on this side; apps that show only macros have, in effect, opted out of the problem.

Stage 5: Personalisation

The newest layer is personalisation. If you have logged your morning oatmeal 200 times, the app probably should not re-derive its calorie count from scratch each time. If your Tuesday dinners are usually pasta and your phone is in central Rome, the prior on 'is this carbonara' is higher. A handful of apps now adjust their estimates with these signals; most do not yet, and the gain in accuracy is real but modest.

What this means for you, the user

1. Trust detection. If the app says 'salmon,' it is salmon.
2. Be sceptical of portions. If the bowl is deeper than it looks from above, the app cannot see that without depth — adjust manually.
3. Photograph from above, with the full plate in frame, and good light. You are helping every stage at once.
4. Pick an app whose database you trust for what you care about. If micronutrients matter, check that the app reports them.
5. Treat single meals as noisy and weekly averages as the signal. The error averages out, and your behaviour does not.