Replacing vector databases with SQLite and SIMD hypervectors in under 1.2GB VRAM (Hillock)

Disclosure: I am the creator of this project. After days of lurking and building up enough karma, I can finally post here.

Every time I tried running local RAG on my own machine, I hit the exact same bottlenecks. First, spinning up Chroma or another vector database alongside an 8B model just to chunk and parse documents takes up precious VRAM that you need for your main model. Second, cosine similarity over text chunks often fails at hard negative rejection, so the model tries to answer questions that are not even in your files and hallucinates with complete confidence.

I spent the last several months building an open source project called Hillock to see if I could solve this without vector databases. It extracts clean relational facts into SQLite using lightweight bi encoders in about five seconds, completely bypassing the generative LLM during ingestion. To stop hallucinations, queries pass through a 10,000 dimensional hypervector gate using late interaction scoring. If the factual graph does not mathematically overlap with the question, it blocks the LLM call before token generation can even start.

I just pushed version 0.8 which bit packs the hypervectors into 157 uint64 integers, allowing the CPU to run gating checks in under 0.01 milliseconds using hardware popcount instructions. It also includes an OpenAI compatible API server so you can drop it straight into Open WebUI, AnythingLLM, or Obsidian. It just landed on PyPI as well via pip install hillock.

The honest trade off is that this pipeline is built for structured, relational facts like technical specs, people, and dates. It is heavily biased toward precision over recall, so it will not do broad poetic or narrative summaries like a 70B model would.

Code is on GitHub at https://github.com/roandejager/Hillock
We also set up documentation at https://hillock.mintlify.site and a developer Discord at https://discord.gg/BGUPNBcVdp