Pandas API for DuckDB, PostgreSQL & ClickHouse — keeping computation inside the database[P]
Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.
I've been building memFrame — an open-source dataframe API that compiles operations to SQL.
The idea:
**Python/DataFrame API → SQL → DuckDB / PostgreSQL / ClickHouse**
Instead of pulling data into Python and doing everything in pandas, memFrame tries to keep computation inside the database.
Currently it has operations for inspection, selection, cleaning, statistics, arithmetic, visualization, etc.
I'm intentionally releasing analytics operations incrementally , because first I want the currently released features to be well tested in public and once all the issues on the current features are solved then — groupby, window functions, sorting, filtering are on the way.
It comes with built-in multiagent architecture for chatting with your data in natural languages.
Would love some technical feedback & suggestions to improve this from people working with pandas/SQL/DuckDB/Postgres.
If the idea sounds useful, a ⭐ helps a lot.
[link] [comments]
More from r/MachineLearning
-
I turned my dog Jackies Brain into a working AI [P]
Oct 14
-
Event2Vec: additive event embeddings where a linear score on a history decomposes exactly into per-event terms [P]
Oct 14
-
Binary noise classification using CRNN mean teacher algo [R]
Oct 14
-
Localized Neural learning vs backpropagation [D]
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.