JSON API documentation static site https://docs.biotorrents.de

Install

Thanks for your interest in BioTorrents.de development! Gazelle is notoriously difficult to install and good docs are lacking. This page, based on the original announcement protocol, is an attempt to make an evergreen install guide. It’s fast paced and only covers the essentials.

Debian system profile

$ uname -a
Linux ohm 5.10.0-6-amd64 #1 SMP Debian 5.10.28-1 (2021-04-09) x86_64 GNU/Linux

$ nginx -v
nginx version: nginx/1.18.0

$ mysql -V
mysql  Ver 15.1 Distrib 10.5.10-MariaDB, for debian-linux-gnu (x86_64) using  EditLine wrapper

$ php -v
PHP 7.4.21 (cli) (built: Jul  2 2021 03:59:48) ( NTS )

$ memcached -h
memcached 1.6.9

$ indexer
Sphinx 2.2.11-id64-release (95ae9a6)

Server setup

This section is done as root. If you’re in a user shell, preface commands with sudo.

Upgrading to Debian unstable

Install the most recent Debian netinst image and modify /etc/apt/sources.list:

deb http://deb.debian.org/debian unstable main contrib non-free
deb-src http://deb.debian.org/debian unstable main contrib non-free

Optionally, create /etc/apt/apt.conf with this content:

APT::Install-Recommends "1";
APT::Install-Suggests "0";

Then upgrade the system:

# apt update
# apt dist-upgrade
# reboot

Further server setup, including TLD considerations, DNS, email, etc., are beyond this guide’s scope. For more info about SSH, Unbound, NSD, OpenSMTPd, Dovecot, Unix users, etc., please see the original launch announcement.

Nginx and Certbot

Install Nginx and Certbot with apt install nginx certbot python3-certbot-nginx.

The basic Gazelle Nginx config should look similar to this. Note that you’ll likely have to change the file paths based on your setup. Also, PHP-FPM may need larger-than-default buffers to serve without 502 errors:

server {
        listen 443 ssl http2;
        root /var/www/html/dev.biotorrents.de;

        client_max_body_size 4M;
        server_name dev.biotorrents.de;

        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

        ssl_certificate /etc/letsencrypt/live/biotorrents.de-0001/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/biotorrents.de-0001/privkey.pem; # managed by Certbot

        access_log off;
        error_log  /var/log/nginx/dev.biotorrents.de-error.log;

        location / {
                index index.php;
        }

        # https://www.nginx.com/resources/wiki/start/topics/examples/phpfcgi/
        location ~ [^/]\.php(/|$) {
                fastcgi_split_path_info ^(.+?\.php)(/.*)$;
                if (!-f $document_root$fastcgi_script_name) {
                        return 404;
                }

                # Mitigate https://httpoxy.org/ vulnerabilities
                fastcgi_param HTTP_PROXY "";

                fastcgi_pass  unix:/var/run/php/php7.4-fpm.sock;
                fastcgi_index index.php;

                # include the fastcgi_param setting
                include /etc/nginx/params/fastcgi_params;

                # https://stackoverflow.com/a/23845727
                fastcgi_buffers 16 16k;
                fastcgi_buffer_size 32k;
        }

        # https://nginx.org/en/docs/http/ngx_http_access_module.html
        location ^~ /.git/ {
                deny all;
        }

        location ^~ /classes/config.php {
                deny all;
        }
}

The Nginx config for the Ocelot tracker should look like this. Nginx acts as a TLS reverse proxy so that Ocelot isn’t directly exposed. Note the additional caveats of Ocelot’s listening port (34000 is default), and setting the correct Host header (so tracker connections don’t show up as localhost):

server {
        listen 443 ssl http2;
        server_name tr0.biotorrents.de track.biotorrents.de;

        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

        ssl_certificate /etc/letsencrypt/live/biotorrents.de-0001/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/biotorrents.de-0001/privkey.pem; # managed by Certbot

        access_log off;
        error_log  /var/log/nginx/tr0.biotorrents.de-error.log;

        # https://nginx.org/en/docs/http/ngx_http_proxy_module.html
        location / {
                proxy_pass http://127.0.0.1:34000;
                proxy_set_header Host $remote_addr:$proxy_port;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Real-IP $remote_addr;
        }
}

Finally, set up TLS certificates with certbot --nginx. Add this entry to the root crontab to renew the certs daily:

20 1 * * * certbot renew >/dev/null 2>&1 && systemctl restart nginx >/dev/null 2>&1

Please see the Certbot docs for more info. Also see the OWASP Secure Headers Project for info on proper Nginx security.

MariaDB

Install MariaDB with apt install mariadb-server and initialize it with mysql_secure_installation.

Note that BioTorrents.de uses TLS connections to a dedicated database server. Unix sockets are preferred for the database running on localhost. The config should look similar to this, paying attention to sql-mode:

[client-server]
# Port or socket location where to connect
port = 3306
#socket = /var/run/mysqld/mysqld.sock

# Import all .cnf files from configuration directory
#!includedir /etc/mysql/conf.d/
#!includedir /etc/mysql/mariadb.conf.d/

[mariadb]
# https://mariadb.com/kb/en/secure-connections-overview/
tls-version = TLSv1.3
#require-secure-transport = on

# https://mariadb.com/kb/en/securing-connections-for-client-and-server/
ssl-cert=/etc/mysql/ssl/server-cert.pem
ssl-key=/etc/mysql/ssl/server-key.pem
ssl-ca=/etc/mysql/ssl/ca.pem

# https://mariadb.com/kb/en/configuring-mariadb-for-remote-client-access/
skip-networking = 0
bind-address = 10.0.0.3

# https://johnemb.blogspot.com/2014/09/adding-or-removing-individual-sql-modes.html
sql-mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

If you’d like to use TLS crypto in your database connections, please see the MariaDB docs. They contain the necessary info to generate self-signed certs.

Finally, load the Gazelle database schema in an SQL shell. I prefer to generate secure passphrases with pwgen -s | encrypt.

CREATE DATABASE gazelle_development;
USE gazelle_development;
SOURCE /var/www/html/dev.biotorrents.de/gazelle.sql;
CREATE USER 'gazelle_development'@'localhost' IDENTIFIED BY '';
GRANT ALL PRIVILEGES ON `gazelle_development`.* TO 'gazelle_development'@'localhost';

PHP

Install PHP and the necessary extensions. The basic PHP package: apt install php php-dev php-fpm.

The PHP extensions. Note that there are two PHP memcached extensions. The correct one is just php-memcache without the “d.” Your distro may require other extensions: php-apcu php-mbstring php-memcache php-mysql.

BioTorrents.de also supports the Seqhash algorithm and requires Blake3 hash support for this feature. Optionally, please install php-blake3.

Gazelle isn’t picky about its PHP config needs, except that short_open_tag = On is still required. I strongly recommend crafting a secure PHP config.

memcached

Install memcached with apt install memcached.

BioTorrents.de supports separate production and development instances. If you with to run two instances, it’s necessary to run two memcached sockets. Otherwise the sites will experience significant data cross-contamination. The required /etc/memcached.conf content:

-d
-m 5120
-s /var/run/memcached/memcached.sock
-a 0777
-t16
-C
-u memcache

A helper script to quickly bring up a second memcached as root:

#!/bin/bash
memcached -d -m 5120 -s /var/run/memcached/memcached-dev.sock -a 0777 -t16 -C -u memcache

Sphinx

Install Sphinx with apt install sphinxsearch.

The /etc/sphinxsearch/sphinx.conf content is too large to reasonably inline. Please find a copy at biotorrents/documentation. The only other Sphinx-related configs are root crontab entries:

@reboot /usr/bin/searchd`.
* * * * * indexer -c /etc/sphinxsearch/sphinx.conf --rotate delta requests_delta >/dev/null 2>&1
7 * * * * indexer -c /etc/sphinxsearch/sphinx.conf --rotate torrents >/dev/null 2>&1
7 0,12 * * * indexer -c /etc/sphinxsearch/sphinx.conf --rotate --all >/dev/null 2>&1

Email

Email setup is beyond this guide’s scope. Please see Gilles’s OpenSMTPd and Dovecot guide.

Application setup

This section should use a separate Unix user for each component. Gazelle, Ocelot, IRC, and sitebot should each have a distinct home folder and full shell. The applications would otherwise be an insecure jumble and hard to maintain.

Gazelle

First create the necessary files and folders. Note that locations are arbitrary and may be one of: home folder, subfolder of /var/www, etc.

# nginx(8) log location
mkdir -m 700 -p /var/www/log/{development,production}
touch /var/www/log/production/{peerupdate.log,schedule.log}
touch /var/www/log/development/{peerupdate.log,schedule.log}
chown -R biotorrents:biotorrents /var/www/log

# files outside the web root
mkdir -m 700 -p /var/www/pictures /var/www/torrents
chown -R www-data:www-data /var/www/pictures /var/www/torrents /var/www/torrents-dev

The Gazelle app user also needs this crontab. Expect to have backups in place by this stage, and note that server backups are beyond this guide’s scope:

0,15,30,45 * * * * curl -s "https://dev.biotorrents.de/schedule.php?key=00000000000000000000000000000000" >> /var/www/log/development/schedule.log 2>&1
2-59/5 * * * * php /var/www/html/dev.biotorrents.de/peerupdate.php "00000000000000000000000000000000" >> /var/www/log/development/peerupdate.log

A useful script for resetting file permissions:

#!/bin/sh
find . -type f -print0 | xargs -0 chmod 0644
find . -type d -print0 | xargs -0 chmod 0755

Gazelle app config

classes/config.php warrants its own section. When setting up Gazelle for the first time, these options must be enabled.

'DEBUG_MODE' = false
'OPEN_REGISTRATION' = true
'FEATURE_SET_ENC_KEY_PUBLIC' = true

Note the separate production and development values and the possibility of duplicate definitions. BioTorrents.de uses a singleton class with extended recursive ArrayObject support invoked by $ENV = ENV::go().

There are some other values to set up. Please pay attention to these values that Gazelle needs for proper function:

SITE_DOMAIN and IMAGE_DOMAIN
WEB_ROOT and SERVER_ROOT
TORRENT_STORE and STATIC_SERVER
The “App keys” and “Database” sections
The “Tracker” and “Tracker URLs” sections
MEMCACHED_SERVERS

Composer

Assuming that php.ini is correctly set up, you may need these modified setup instructions from the official Composer docs:

mkdir -p /var/www/bin
php -d allow_url_fopen=1 -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
# Hash checking, running the install script, and unlinking
# See the official Composer docs for details
mv composer.phar /var/www/bin/composer

Then add /var/www/bin to the Gazelle user’s $PATH and run: php composer.phar update.

SCSS and fonts

Install SassC with apt install sassc.

Download the font pack and extract it with: tar zxvf fonts.tgz -C /var/www/html/dev.biotorrents.de/static/styles/assets/fonts.

Then install SassC with apt install sassc. This should be a for loop, to compile the CSS:

#!/bin/bash
styles="/var/www/html/dev.biotorrents.de/static/styles"
sassc "$styles/beluga/beluga.scss" > "$styles/beluga.css"
sassc "$styles/bookish/bookish.scss" > "$styles/bookish.css"
sassc "$styles/development/development.scss" > "$styles/development.css"
sassc "$styles/genaviv/genaviv.scss" > "$styles/genaviv.css"
sassc "$styles/global/global.scss" > "$styles/global.css"
sassc "$styles/matcha/matcha.scss" > "$styles/matcha.css"
sassc "$styles/oppai/oppai.scss" > "$styles/oppai.css"
sassc "$styles/postmod/postmod.scss" > "$styles/postmod.css"
sassc "$styles/public/public.scss" > "$styles/public.css"

Ocelot

BioTorrents.de uses What.CD’s Ocelot with the 10th anniversary mixtape patches and developing TLS support. The patched version is available at biotorrents/ocelot.

Then installed the dependencies like below. Note that specific dependencies may differ on your system.

apt install \
	automake \
	g++ \
	gcc \
	libboost-dev \
	libboost-iostreams-dev \
	libboost-system-dev \
	libev-dev \
	libmysql++-dev \
	libtcmalloc-minimal4 \
	make

The compilation ritual is likewise fraught with pitfalls. I found autoreconf with manual library locations to work best:

autoreconf
./configure \
	--with-boost-libdir=/usr/lib/x86_64-linux-gnu \
	--with-ev-lib=/usr/lib/x86_64-linux-gnu \
	--with-mysql-lib=/usr/lib/x86_64-linux-gnu \
	--with-mysqlpp-lib=/usr/lib/x86_64-linux-gnu
make
make install

Copy ocelot/ocelot.conf.dist. The daemon runs on localhost:34000 and Nginx TLS reverse proxies it to localhost:443. ocelot.conf lives in the Ocelot user’s home folder and the daemon runs in a tmux window there.

IRC and kana (sitebot)

Docs pending the completion of sitebot API integration.

Inside the Gazelle admin panel

At this point it should be possible to register for the site. The first account is the sysop so please act quickly here. Disable DEBUG_MODE and FEATURE_SET_ENC_KEY_PUBLIC as soon as you register! Do apt install qrencode for 2FA support and enable it with a GPG key on the sysop account.

Configure a client whitelist on the Toolbox page by the BitTorrent spec’s peer ID list. Please find a list of quality client peer IDs below. Note that LibTorrent 0.1x.y also covers rTorrent/ruTorrent and other clients that use rakshasa’s library:

Client Name	Peer ID
Deluge 1.x.y	`-DE1`
Deluge 2.x.y	`-DE2`
KTorrent 3.x.y	`-KT3`
KTorrent 4.x.y	`-KT4`
KTorrent 5.x.y	`-KT5`
LibTorrent 0.x.y	`-lt0`
qBittorrent 2.x.y	`-qB2`
qBittorrent 3.x.y	`-qB3`
qBittorrent 4.x.y	`-qB4`
Transmission 2.xy	`-TR2`
Transmission 3.xy	`-TR3`

Most of the Toolbox pages don’t write to the database and all of them should work. For more BitTorrents info see Calomel’s rTorrent hacking guide.

Building this documentation

biotorrents/documentation has the Mkdocs source code. Install MkDocs with apt install mkdocs. Then do mkdocs build.

install.md 15KB History Raw