A chunking lib in Rust that is ~20x faster [P]
Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.
Hey,
I wanted a faster chunking library for my system without affecting the overall accuracy. Did not find many options. So I've build https://github.com/d1pankarmedhi/chunkr
It has most of the chunking strategies like Character, Recursive, Markdown header, Late chunking, Hierarchical chunking, etc. It also supports native PDF loader, and other additional file types.
Some stats (MBA M4 16GB):
| Test Case (matched parameters) | Chunkr | LangChain | LlamaIndex | Chonkie | semchunk | text-splitter |
|---|---|---|---|---|---|---|
| Recursive (1 MB, 1000/200) | 2,264 MB/s | 769 MB/s | 10 MB/s | 225 MB/s | 42 MB/s | 175 MB/s |
| Recursive (5 MB, 1000/200) | 2,039 MB/s | 696 MB/s | — | 201 MB/s | 40 MB/s | 46 MB/s |
| Fixed Char (1 MB, 1000/200) | 750 MB/s | 1.7 MB/s | — | 22 MB/s | — | — |
| Markdown (500 KB, 1000/150) | 819 MB/s | 67 MB/s | 19 MB/s | — | — | 40 MB/s |
| Python Code (200 KB, 1500/200) | 3,232 MB/s | 622 MB/s | — | — | — | 5.7 MB/s |
| Sentence (500 KB) | 622 MB/s | — | 10 MB/s | 20 MB/s | — | — |
| BPE Tokens (200 KB, cl100k_base, 512/50) | 38 MB/s | 43 MB/s | 2.0 MB/s | 151 MB/s | — | 7.2 MB/s |
| 100 docs x 50 KB (parallel batch) | 3,224 MB/s | 679 MB/s | — | 213 MB/s | — | — |
| Extractor / Pipeline | Latency | Throughput | Speedup vs PyPDF |
|---|---|---|---|
| Chunkr PDFLoader (Full Text) | 747.9 ms | 2,762 pgs/s | 15.9x Faster |
| Chunkr PDFLoader (Page Documents) | 721.0 ms | 2,865 pgs/s | 16.5x Faster |
PyMuPDF (fitz) | 2,616.8 ms | 789.5 pgs/s | 4.5x Faster |
| pypdf (pure Python) | 11,900.5 ms | 173.6 pgs/s | 1.0x (baseline) |
| Chunkr End-to-End (PDF + Recursive) | 798.1 ms | 2,589 pgs/s | 14.9x Faster |
| PyMuPDF + LangChain RecursiveTextSplitter | 2,659.3 ms | 776.9 pgs/s | 4.5x Faster |
| pypdf + LangChain RecursiveTextSplitter | 12,054.5 ms | 171.4 pgs/s | 1.0x (baseline) |
Do check it out and share your feedback. Thanks!
[link] [comments]
More from r/MachineLearning
-
I turned my dog Jackies Brain into a working AI [P]
Oct 14
-
Event2Vec: additive event embeddings where a linear score on a history decomposes exactly into per-event terms [P]
Oct 14
-
Binary noise classification using CRNN mean teacher algo [R]
Oct 14
-
Localized Neural learning vs backpropagation [D]
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.