r/MachineLearning · · 1 min read

Pandas API for DuckDB, PostgreSQL & ClickHouse — keeping computation inside the database[P]

Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.

I've been building memFrame — an open-source dataframe API that compiles operations to SQL.

The idea:

**Python/DataFrame API → SQL → DuckDB / PostgreSQL / ClickHouse**

Instead of pulling data into Python and doing everything in pandas, memFrame tries to keep computation inside the database.

Currently it has operations for inspection, selection, cleaning, statistics, arithmetic, visualization, etc.

I'm intentionally releasing analytics operations incrementally , because first I want the currently released features to be well tested in public and once all the issues on the current features are solved then — groupby, window functions, sorting, filtering are on the way.

It comes with built-in multiagent architecture for chatting with your data in natural languages.

Would love some technical feedback & suggestions to improve this from people working with pandas/SQL/DuckDB/Postgres.

github || docs

If the idea sounds useful, a ⭐ helps a lot.

submitted by /u/Lumpy_Vermicelli8869
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/MachineLearning