Crawling and indexing lemmy inter-instance would be an incredible boon to discoverability on the platform.

It does. And Firefox is my default browser app.

I hear you, but genetic change at the level of these diseases and traits can take on the order of hundreds of thousands of years or more to accumulate into meaningful trends. Social society is a part of that process, in the way it might be for other social animals. If social dynamics tend to result in communities harboring vulnerable individuals, then there is probably some selective advantage to that behavior, not the other way around.

This is a common misconception. These traits are not likely due to modern medicine (which is very, very new compared to the scale of human evolution). The environment plays a big role, but there is always a distribution of traits in a normal population, some good, some bad. Not to mention that what we might be self-selecting for must change very rapidly as civilizations rise and fall, preferences shift like the winds, and ethics rapidly evolve. I think this misconception can be dangerous, because of what you mentioned. Eugenics.

looks like https://lemmyverse.net/ is doing a decent job with indexing at the moment. I do honestly feel that indexing should take place on every instance, since each instance has a unique position in the network, and the indexing parameters/ranking algorithms could be under per-instance control rather than an outside third party.

I really like the idea. There one major issue that I see currently, and that is discoverability. It takes some real effort and time to explore things outside of your own instance. I think the federation of pre-federation content will be important for discoverability, since the foundation of a community is in it’s ranking of posts, which takes time and interaction. Right now, votes, comments, and most posts pre-federation on another instance are just not reachable.

I believe this problem can be solved, and there are a lot of motivated developers here, so I’m all in on lemmy.

Use tor.

Maybe we need a discover tab. Something different from ‘All’. It would request top posts, or highly active posts, or whatever, from as many instances as it can find by crawling instance-to-instance trying to explore the entire network periodically. Does federation take place after the first post request from a new instance, or from the first subscribe request?

Follow up question: After the first contact, will all new posts, comments, votes always be federated between the two instances?

Thanks for the insight.

This is a major hurdle for discovery. My workflow to find new communities is basically to search for them on several different instances, visit their instance domain directly, search for their url through my home instance, and even then it’s occasionally useless because the only posts available to view in a feed from my home instance have no votes, no comments, and are pretty random. All the interesting posts that have had a moment to be voted on and commented on are a couple days old, and thus the only way to find things to interact with is to use the instance through it’s native domain. I suppose I could manually search for each URL of posts and comments as I browse in the native domain back into the search engine of my home domain, but this is insanity. Is this really the way we want to do things?

What if there was a way for one instance to request not the entire backlog of posts at once in these situations, but a series of posts long enough to fill one page of a feed at a time that match some search criterion when they attempt to directly explore a new community/instance, even if they are posts created pre-federation. Then the ‘most commented’ or ‘top of the week’ or especially ‘Top of All Time’ and old pinned posts, basically the ones people would want to see, that define a community, would over time be federated as users browsed, accumulating a select subset of pre-federated posts on-demand. Also, it seems like having all the votes and comments on a posts that you just requested would be important. Right now if I search for a specific pre-federated post, I get a bare bones version of it with a tiny fraction of the votes, and usually 0 comments. Something like that seems like it would be a huge step in usability and discoverability, especially for users on newer, smaller instances. I don’t know much about how the system works though, so I don’t know it this is the right idea.