Generating exam questions from a school’s own past papers: looking for a second opinion on the pipeline [D]

Hi all,

I’m building a question bank feature for a school platform (high school, mixed Cambridge + in-house curriculum). The idea: new questions are generated from the school’s own past papers. The curriculum’s learning objectives, which are already in our database, act as a guard rail so questions stay in scope. A curriculum admin approves everything; teachers only use the approved bank.

I have a working prototype:

• Question papers + mark schemes (PDFs) go to an LLM, which splits them into items at sub-question level and tags each to objectives
• A small model writes new questions; a different model answers them blind as a check
• Multiple-choice answer keys are checked in code against the mark scheme, not trusted to the model
• So far: ~80% of MCQs pass all checks. Structured/practical questions are weaker; the small model tends to invent wrong chemistry in lab set-ups, and a mid-size model fixed most of it

What I’d love input on:

1. PDF extraction. The PDFs’ text layer mangles science notation (Δ dropped, subscripts split, table columns interleaved), so right now I send the whole PDF to the LLM. Has anyone used docling, marker or similar on exam papers and got genuinely more precise results, especially for equations, tables and diagrams?
2. Retrieval. Objective tags give exact retrieval for “past questions on topic X”. Is pgvector worth adding from day one, or only once the bank is large enough to need semantic de-duplication?
3. Cost at scale. Experience with batch APIs or small models at thousands of questions? Where did quality fall off?
4. Quality measurement. Beyond human review, how do you judge whether AI-generated assessment items are actually good? Any eval approaches that held up?

Stack: NestJS + Postgres (schema per tenant) on a self-hosted VPS. Exam content stays on our server; only curriculum text and question drafts go to the model.

If you’ve worked on something similar (edtech, document AI, assessment generation), I’d really appreciate comments, or a DM if you’re up for a longer chat. Happy to share more of the architecture there. Also open to paid consulting if it’s a good fit.

Thanks!