Benchmarking calories evaluation with LLMs
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions.
I needed to pick a model so I made a quick benchmark.
The setup was:
- Nutrition5k photos for photo + calories: https://github.com/google-research-datasets/Nutrition5k
- A tool with access to calories information from USDA FoodData Central + MEXT
- I evaluated models based on how many of the meals they managed to have under 20% of error
- All on the same randomly picked 25 meals.
Models too big for my machine were run through OpenCode Go/OpenRouter. I've also included Spark 1.3 since it'll supposedly be open weights.
Results
| Model | % within 20% | Mean bias | Median Error |
|---|---|---|---|
| Qwen 3.8 27b | 16% | +64 kcal | 148 kcal |
| GLM 5.3 Flash | 28% | +18 kcal | 65 kcal |
| Qwen 3.8 Max | 32% | -11 kcal | 48 kcal |
| Muse Glimmer 30b | 32% | +25 kcal | 92 kcal |
| Qwen 3.8 Flash | 36% | +2 kcal | 91 kcal |
| DeepSeek v4 Flash Vision | 40% | +52 kcal | 65 kcal |
| Muse Spark 1.3 | 48% | -24 kcal | 45kcal |
I know it's not the most scientific benchmark, but it's interesting to see that the order is not really linked to model size.
The most interesting for me is how Muse Glimmer 30b trounces Qwen 3.8 27b here. I think it highlights how "the best" model on consumer hardware (~32Gb VRAM) really depends on the task.
[link] [comments]
More from r/LocalLLaMA
-
The AI race: China's open-weight models trail US closed models by just 7 months
Oct 14
-
Anthropic Executive Says Open-Source AI Models Are Higher Risk
Oct 14
-
ComfyUI v0.40.0 released
Oct 14
-
So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.