Were designing a tiny autonomous research agent

This base model is only 43m parameters trained on 3m arXiv abstracts. We plan to continue pre-training and post training. If you create fine-tuning datasets or if you know of any datasets that can help shape the behavior for our goal we appreciate all contributors. The goal is to make a local agent that can autonomously do research. Its just a simple loop to search the web & document its findings as an experiment to see what is possible. If we train a language model on nothing but science, physics and technology can it make new discoveries?

We are testing this by creating fine-tuning examples that contain a pattern of asking questions and answering them until coming to a conclusion from first principles. If you have any suggestions to achieve the goal we are all ears. Please leave a comment.