# Crawl lists of active Session Communities

## What does this site do?

This script crawls known sources of published Session Communities, 
queries their servers for available information and 
displays this information as a static HTML page.  
The results of this can be viewed on https://sessioncommunities.online/.


## What is Session?

Session is a private messaging app that protects your meta-data, 
encrypts your communications, and makes sure your messaging activities 
leave no digital trail behind.  
https://getsession.org/


## Details

### Which sources are crawled?

Currently this script crawls the following sites:

- https://github.com/GNU-Linux-libre/Awesome-Session-Group-List
- https://lokilocker.com/Mods/Session-Groups/wiki/Session-Open-Groups
- https://session.directory/

Additionally, the following open community servers are polled:

- https://open.getsession.org
- http://13.233.251.36:8081

### Steps

1. Fetching source HTML: `get_html_from_known_sources()`
1. Extracting Session invites from the HTML:
`extract_join_links_from_html()` and `get_servers_from_join_links()`
1. Making sure servers are online: `reduce_servers()`
1. Querying the servers for all available rooms 
and normalize active user numbers: `query_servers_for_rooms()`
1. De-duplicating servers based on public keys: 
`get_pubkeys_of_servers()` and `reduce_addresses_of_pubkeys()`
1. Aggregating all server info & adding language data: `generate_info_arrays()`
1. Generating static HTML content: `generateHTML()`

### Legacy support

Right now we fully support legacy SOGS servers, 
although this support is likely going to be dropped soon, 
since those servers can not even be joined anymore with current Session clients.  
Dropping legacy support will also increase maintainability.

### Work around bad routing to Chinese servers

Depending on your location, it is possible for you to get really bad routing to 
SOGS servers behind the GFW. In this case, 
the initial connection is still successful, but you'll never receive 
any actual content and the retrieval attempt will simply time out. 
This happens randomly. To make sure this won't affect the results, we simply
check whether the server is online (the initial connection being successful), 
and then retry a lot of times with a short timeout 
until we eventually get the content.  
The details can be seen in `curl_get_contents()`.

### Official repositories
- https://github.com/mdPlusPlus/sessioncommunities.online
- https://lokilocker.com/SomeGuy/sessioncommunities.online

If your favourite Session community is missing a language flag, 
you can issue a pull request here:
- https://github.com/mdPlusPlus/sessioncommunities.online-languages/


## Contact

If you want to contact me, you can add me on Session via my 
[ONS](https://docs.oxen.io/using-the-oxen-blockchain/using-oxen-name-system): 
"someguy" (without the quotes)