This is a project to (1) collect twitter archives from volunteers to study ourselves (2) build open source tools for collection & analysis so that other communities can reproduce this work on their own data.
https://github.com/DefenderOfBasic/PhD/blob/main/twitter-community-archive.md
This story is an example of Marketing companies into existence.
On Dec 2023 I wrote:
I want this to exist so badly and I want to see how far I can take it with just Twitter search and lots of manual curation
Maybe even a little initiative to export a bunch of our Twitter archives and apply some interesting semantic clustering and visualization https://x.com/DefenderOfBasic/status/1734912039598928379
On July 2024 I made a prototype where anyone could upload their twitter archive
here’s all my tweets, grouped by threads, and I can client-side search it https://x.com/DefenderOfBasic/status/1816225499888378197
Xiq (https://x.com/exgenesis) saw this and built a non-prototype version of it. I wrote about it and it got on the front page of HN:
Twitter shut off API access; users are now volunteering their data for an open API
A lot of community applications died when API access to twitter, reddit & others was severely restricted, but they can’t stop us from sharing our own data with each other! https://omarshehata.substack.com/p/twitter-shut-off-api-access-users
The more users we got, the more interesting insights & analysis we could do, which attracted more users. Eventually in April 2025 I made a tweet about “the first glimpse of the nooscope” which attracted investors/funding.
we’re already getting the first views of what our collective mind looks like under a nooscope! Each “clump” here is an attractor in idea space, naturally forming structures in the space between our minds https://x.com/DefenderOfBasic/status/1908916969467687181
The demo itself was made by Brandon @ nomic, in our discord. I just made a video of it and tweeted it / explained why this was cool.