This finance-model benchmark card is more useful for what it discloses than for who "wins"
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| The official benchmark card for Ling-3.0-flash-Fin is a useful reminder that the unit being tested is rarely just “the model.” The release says most runs used temperature 1, top_p 0.95 and the highest available reasoning effort. FinFIRST and FinSearchComp Verified used a common ReAct scaffold with Web Search, Visit and Python. SpreadsheetBench used Claude Code 2.1.173 with LibreOffice 25.8.7, Search disabled, 120 or 300 maximum interaction turns and a three-hour task timeout; Ling used temperature 0.6 there. The chart also mixes evidence types. Some results come from official or externally published scores, while others are internal runs. FinSearchComp Verified is an internal 145-question set with expert-revised answers and a GPT-5 judge. FinCRAFT is internal. FinFIRST is announced as “coming soon,” not public today. None of that makes the chart useless. It makes the claim narrower: these are reported results under several specific agent systems, tool budgets and evaluation pipelines—not a clean intrinsic ranking of raw checkpoints. The finance weights are also not public yet; the team says they are due next week. Once they land, the most valuable follow-up would be the exact harnesses, prompts, tool adapters, per-run variance and failure traces. Until then, the bars are a test plan, not an independent reproduction. [link] [comments] |
More from r/LocalLLaMA
-
an unscientific qwen 3.8 flash next and glm 5.3 flash comparison
Aug 30
-
Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph
Aug 30
-
Nemotron-3.5-Lightning at 11.77 GiB, a 16 GB option for a model that didn't have one
Aug 29
-
Humaneval benchmark for Deepseek V4 Flash 0731 vs GLM5.3 Flash on 2x DGX Spark setup
Aug 29
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.