Chère de Prince

Table of Contents

1 General

1.1 Domain Names

1.1.1 buron.coffee

  • book.buron.coffee
  • buron.coffee
  • emby.buron.coffee
  • headphones.buron.coffee
  • mathilde.buron.coffee
  • medias-dl.buron.coffee
  • mpd.buron.coffee
  • music.buron.coffee
  • piwigo.buron.coffee
  • radio.buron.coffee
  • shaarli.buron.coffee
  • transmission.buron.coffee

1.1.2 cheredeprince.net

  • cheredeprince.net
  • conway.cheredeprince.net
  • herculevshydre.cheredeprince.net
  • life.cheredeprince.net
  • math.cheredeprince.net
  • owncloud.cheredeprince.net
  • rss-bridge.cheredeprince.net
  • rss.cheredeprince.net
  • static.cheredeprince.net
  • syracuse.cheredeprince.net
  • votenroll.cheredeprince.net
  • wallabag.cheredeprince.net

1.3 IP V6

It seems mandatory in order that synapse federation works properly.

The livebox may block some ports with its firewall.

2 Administration

Some tools, I use to check that everything is all right.

2.1 Useful Commands

  • Find the largest files in a folder
find /mnt/pellicule/ -type f -printf "%s\t%p\n" | sort -n | tail
  • List the extensions of files in a directory
find /mnt/super8/films/ -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u
  • find the total size of the files created during the last month :
find /data/youtube-dl-archives/ -type f -ctime -30 -print0 | du -shc --files0-from - | tail -n1
  • find the total size of the files accessed during the last month :
find /data/youtube-dl-archives/ -type f -atime -30 -print0 | du -shc --files0-from - | tail -n1
  • find the most recent file in a directory:

    find /mnt/super8/ -type f -printf "%T@ %p\n" | sort -n | cut -d' ' -f 2- | tail -n 1
    

2.2 Network

# network usage during the current day
vnstat 
vnstati -vs -o /tmp/vnstat.png -i eno1

# live network usage
ifstat

# network usage per process
nethogs

2.3 Fail2ban

I follow this tutorial.

2.4 Security

chmod 740 /data/owncloud /data/piwigo /data/backup-db /var/lib/mysql* /var/lib/postgresql /var/lib/mongodb /var/backups /var/log/nginx/*

3 Backup

3.1 General

There are two main backup storages:

  1. The backup cheredeprince server is a daily backup of:
    • /mnt/pellicule movies hard drive (3To full)
    • /mnt/super8 video hard drive (3To)
    • important directories of the server
/etc 
/home 
/root 
/opt 
/var 
/data/music 
/data/owncloud 
/data/piwigo 
/data/youtube-dl-archives 
/data/backup-db # dumps of databases 
  1. An external encrypted hard drive called coffre (3To) saving:
    • version of images of the root hard drive of the server
    • an archive of the hard drive /data

The *machine uses the following hardware:

3.2 Borg Configuration (important directories backup)

I use this repository for installation on client and server side: https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup

3.2.1 Basic commands

To get the list of all archives. The first column contains archive id.

export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
borg list ssh://backup.cheredeprince.net:1516/mnt/backup/backup/

blackcube-31.07.2018 Tue, 2018-07-31 03:00:09 blackcube-31.08.2018 Fri, 2018-08-31 03:00:08 blackcube-30.09.2018 Sun, 2018-09-30 03:00:08 blackcube-31.10.2018 Wed, 2018-10-31 03:00:09

To information about one archives particular archive (using its id, see above)

export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
borg info ssh://backup.cheredeprince.net:1516/mnt/backup/backup/::blackcube-31.07.2018

To get the list of a path in a specific archive

export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
borg list ssh://backup.cheredeprince.net:1516/mnt/backup/backup/::blackcube-31.07.2018 data/backup-db/

3.3 Code on Paper

In order to be sure that passwords and others vital information are safely saved, we have to print it.

I use qrencode to encode text in a qrcode, install it with:

sudo apt-get install libqrencode3 

and SecScanQR to decode using an android smartphone: https://f-droid.org/en/packages/de.t_dankworth.secscanqr/

To encode some text:

qrencode "text" -o qrcode.png

print qrcode.png and delete it.

3.4 Configuration du serveur de backup

3.4.1 SSH config

Host backup
     HostName backup.cheredeprince.net
     Port 1516
     User root
     IdentityFile ~/.ssh/id_rsa

Le port ssh sur la machine backup.cheredeprince.net est 22 !

3.5 Accès de local vers backup

L'adresse ip local du serveur est 192.168.1.21. L'accès local des partitions est fait par NFS avec la configuration :

# /etc/exports
/mnt/pellicule 192.168.1.0/24(ro,sync,all_squash,anonuid=1000,anongid=1000,subtree_check)
/mnt/super8    192.168.1.0/24(ro,sync,all_squash,anonuid=1000,anongid=1000,subtree_check)

3.5.1 Samba

https://www.howtoforge.com/tutorial/samba-server-ubuntu-16-04/

installation

apt-get install -y samba samba-common python-glade2 system-config-samba
# /etc/smb.conf
[global]
workgroup = WORKGROUP
server string = Samba Server %v
netbios name = ubuntu
security = user
map to guest = bad user
dns proxy = no

#============================ Share Definitions ============================== 

[Anonymous]
path = /samba/anonymous
browsable =yes
writable = yes
guest ok = yes
read only = no
force user = nobody
chmod -R 0775 /mnt/super8/films-martine
chown -R nobody:nogroup /mnt/super8/films-martine

service smbd restart

Il y a des problèmes quand on déplace des dossiers, on a pas les droits d'écrire les fichiers à l'intérieur …

Il y a une source dans Emby de /mnt/super8/films-martine par Samba.

4 Let encrypt

4.0.1 Get a new certificat

certbot certonly --webroot -w /var/www/letsencrypt -d {domain}

4.0.2 Renew existing certificats

certbot renew

5 Iptable

Iptable rules definitions can be found in /etc/iptables.sh

6 Matrix

6.1 Riot

6.1.1 Login

Il semble qu'au login, il faut

  1. spécifier le domaine du serveur
  2. entrer le nom d'utilisateur local (emixam150 et pas @emixam150:cheredeprince.net) et le mot de passe.

Pour que la version "simple" (sans spécifier le nom de domaine et utiliser @emixam150:cheredeprince.net) fonctionne au login, il faut définir un accès vers .well-known/matrix/client (https://github.com/vector-im/riot-web/issues/9224).

6.2 Synapse

D'après cette documentation, il faut utiliser un certificat signé (comme ceux fournis par letsencrypt) pour les connections de fédération. Vu que le nom de domaine du serveur matrix est "cheredeprince.net", je peux utiliser le certificat que j'utilise pour ce domaine. Et ça marche !

Testeur pour la fédération

6.3 Mautrix Telegram

6.3.1 Dependencies

installation of libolm3.xx (optional and still not working)

 cd /tmp
 git clone https://gitlab.matrix.org/matrix-org/olm.git
 cd olm
 cmake . -Bbuild
 cmake --build build
cp build/libolm.so /usr/lib/x86_64-linux-gnu/libolm.so

6.4 Mautrix Facebook

6.4.1 Bridge setup

https://github.com/tulir/mautrix-facebook/wiki/Bridge-setup

#create user
useradd -M --shell /bin/false mautrix-facebook
usermod -L mautrix-facebook

#create directory
mkdir /usr/share/mautrix-facebook
cd /usr/share/mautrix-facebook


# install in root
virtualenv -p /usr/bin/python3 .
source ./bin/activate
pip install --upgrade mautrix-facebook

# config and change the port to 8006 (for telegram it is 8005)
cp lib/python3.6/site-packages/mautrix_facebook/example-config.yaml config.yaml
chown -R mautrix-facebook:mautrix-facebook . 

# Generate the appservice registration
sudo -u mautrix-facebook bin/python3.6 -m mautrix_facebook -g

# Please ensure that the file /etc/matrix-synapse/homeserver.yaml contains:
# app_service_config_files: ['/usr/share/mautrix-telegram/registration.yaml', '/usr/share/mautrix-facebook/registration.yaml']

# update db
sudo -u mautrix-facebook bin/alembic upgrade head

# start with
sudo -u mautrix-facebook bin/python3.6 -m mautrix_facebook

6.4.2 Authentication

6.4.3 Problème

Pas de sync Facebook -> matrix

revenir dans la conversation avec le bridge et faire sync.

6.5 Matrix Registration

Matrix registration is a project aiming at allow people to register to a matrix server using web page link.

The installation command is:

pip3 install matrix-registration

Then we create an user for the daemon

useradd -M --shell /bin/false matrix-registration
usermod -L matrix-registration

then we create working directory:

mkdir /etc/matrix-registration
chown matrix-registration /etc/matrix-registration
chmod 770 /etc/matrix-registration

copy the following file to /etc/matrix-registration/config.yaml

server_location: 'https://cheredeprince.net
server_name: 'cheredeprince.net
shared_secret: 'copy here the registration_shared_secret value from homeserver.yaml of matrix-synapse'
admin_secret: ''
base_url: ''
riot_instance: 'https://riot.im/app/'
db: 'db.sqlite3'
host: 'localhost'
port: 5000
rate_limit: ["100 per day", "10 per minute"]
allow_cors: false
logging:
  disable_existing_loggers: False
  version: 1
  root:
    level: DEBUG
    handlers: [console, file]
  formatters:
    brief:
      format: '%(name)s - %(levelname)s - %(message)s'
    precise:
      format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
  handlers:
    console:
      class: logging.StreamHandler
      level: INFO
      formatter: brief
      stream: ext://sys.stdout
    file:
      class: logging.handlers.RotatingFileHandler
      formatter: precise
      level: INFO
      filename: m_reg.log
      maxBytes: 10485760 # 10MB
      backupCount: 3
      encoding: utf8
# password requirements
password:
  min_length: 8

and create a service file /etc/systemd/system/matrix-registration.service:

[Unit]
Description=Matrix Registration

[Service]
WorkingDirectory=/etc/matrix-registration/
Type=simple
User=matrix-registration
ExecStart=/usr/bin/python3 /usr/local/bin/matrix-registration serve
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

enable and start the service

systemctl enable matrix-registration
systemctl start matrix-registration

list the tokens with:

curl -X POST -H "Authorization: SharedSecret shared_secret" http://localhost:5000/token

6.6 Guest room

25/11/2020 j'active l'accès guest sur synapse : https://github.com/vector-im/element-web/issues/9264#issuecomment-699536161

7 Bobine

The root is /var/www/buron.coffee/bobine and the content is generated using /home/becasse/tinymoviemanager

We create an symbolic link for create path to the movies

ln -s /mnt/ /var/www/buron.coffee/bobine/mnt

To update the website:

cd build
# update the db, scrape the new files metadata, rename the new scraped files
./tinyMediaManagerCMD.sh -updateMovies1 -scrapeUnscraped -rename
./tinyMediaManagerCMD.sh -export DarkTemplate /var/www/buron.coffee/bobine/
./tinyMediaManagerCMD.sh -export DarkTemplateTV /var/www/buron.coffee/bobine/

To update TinyMediaManager

cd ~/tinyMediaManager
git pull 
git checkout origin/disableimgreexport
#save data directory
cp -r build/data .
# buiding 
mvn package
# restore data
cp -r data build/

# basic setup
cd build
chmod +x *.sh
# do not ask me why
mv getdown-new.jar getdown.jar
# at least to generate templates
./tinyMediaManager.sh

Here a little script to insert a movie from the torrent directory

# copy the movie film
cp /data/torrents/$1 /mnt/super8/films/ 
# update the bobine database and website
cd /home/becasse/tinyMediaManager/build
./tinyMediaManagerCMD.sh -updateMovies1 -scrapeUnscraped -rename
./tinyMediaManagerCMD.sh -export DarkTemplate /var/www/buron.coffee/bobine/
./tinyMediaManagerCMD.sh -export DarkTemplateTV /var/www/buron.coffee/bobine/

7.1 Mount Movie Folders

sshfs -o default_permissions,IdentityFile=~/.ssh/id_rsa bcdp:/mnt/super8 /mnt/super8 -o idmap=user
sshfs -o default_permissions,IdentityFile=~/.ssh/id_rsa bcdp:/mnt/pellicule /mnt/pellicule -o idmap=user

7.2 pipeline for json database

7.2.1 First idea: export the database in XML and translate it into JSON

./tinyMediaManagerCMD.sh -export ListExampleXml /tmp/
xml2json -t xml2json -o /tmp/movielist.json /tmp/movielist.xml --pretty --strip_text

Why not using xml2jsonmappings ? It is a bug with the export of Person ids with new TMM version …

7.2.2 Second idea: use the .nfo files to create xml database

DIRS="/mnt/super8/films"
OUTPUT="/tmp/movies.json"

> $OUTPUT
for dir in $DIRS;
do
    for f in $dir/*/*.nfo;
    do
        echo $f;
        xml2json -t xml2json "$f" --pretty --strip_text >> $OUTPUT;
    done;
done;

reshaping the database file by tags:

jq -s '[.[].movie| select(has("tag") and has("tmdbid") and .tmdbid != null) | select(.tag | type =="array") | {id: .tmdbid, tag: .tag[]}] | group_by(.tag)| [.[]|{tag:.[0].tag, ids:[.[].id]}]| sort_by(.ids | length)| reverse | [limit(10; .[])]' /tmp/movies.json

We can split the previous command in the following steps as a map and reduce program:

  1. [.[].movie| select(has("tag") and has("id") and .id != null) | select(.tag | type =="array") | {id: .id, tag: .tag[]}] builds an array of tag by movie id,
  2. ..| group_by(.tag)| [.[]|{tag:.[0].tag, ids:[.[].id]}] reduces the array by grouping the movie ids by tags,
  3. ..| sort_by(.ids | length) sorts the tags by number of movies.

8 MySQL

Connection

mysql -u root -p<pass>

dump all the database

mysqldump -u root -p<pass> --all-databases > backup.db

Restore a database from a all database dump

mysql -u root -p<pass> --one-database database_name < backup.db

List the databases

SHOW DATABASES;

Delete a database

DROP DATABASE name;

9 WriteFreely

depot: https://github.com/writeas/writefreely

The install path is: /var/www/buron.coffee/writefreely

9.1 Update

Most of the time, the update are resumed by the following lines:

# as root 
systemctl stop writefreelyburon.service
# backup the databases
mysqldump -u root -p<pass> --all-databases > backup.db
# as becasse
cd /var/www/buron.coffee/
# backup the folder 
cp -r writefreely ~/
# download and install the update
wget path to tar.gz build
tar -xvzf downloaded file
# migrate the database
cd writefreely
./writefreely -migrate
# as root 
systemctl start writefreelyburon.service

10 FreshRSS

La version actuelle est 1.17.1-dev Le répertoire d'installation est /usr/share/FreshRSS

Pour lister les utilisateurs par date de dernières connections:

./cli/user-info.php -h | sort -k3 -r

11 RSS Bridge

11.1 Instagram Bridge

Le 14/12/2020, je constate que rss bridge est utilisé à plus de 50% pour Instagram:

tail /var/log/nginx/cheredeprince.net/rss-bridge.access.log  -n 100000 | grep "Instagram" | wc -l
# 53713

Le problème est multiple:

  1. depuis plusieurs mois, il y a une erreur sur le bridge, qui n'est plus maintenu : https://github.com/RSS-Bridge/rss-bridge/issues/1891
  2. le nombre de requête vers Instagram provenant de mon serveur est sûrement trop important et son IP a dû être blacklistée.

Le 14/12/2020, je décide de désactiver le brigde pour quelque jours histoire de voir, s'il marchera mieux plus tard.

J'ai essayé de ne permettre que les requêtes locales avec nginx, par example celles de FreshRSS, mais ça ne marche pas:

if ($args ~ "bridge=Instagram") {
  # disabled the bridge for external requests !!!!
  # allow anyone in 192.168.1.0/24
  allow   192.168.1.0/24;
  allow  86.247.20.120;
  allow  2a01:cb04:751:b600::/56;
  # drop rest of the world
  deny    all;
}

12 Wallabag

The actual version of wallabag is 2.4

12.1 Update

cd /var/www/cheredeprince.net/wallabag/
sudo -u www-data make update

13 Owncloud

La version actuelle est 10.5.0

13.1 Hack

For forcing shared videos player to be not larger than the window, I have to add into /var/www/cheredeprince.net/owncloud/core/css/styles.css:

video {
max-width: 100% !important;
}

14 Php

14.1 Errors and slow executions

J'ai découvert qu'il y avait cette erreur:

tail /var/log/php7.2-fpm.log
[14-Jul-2020 13:34:02] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

J'ai passé max_children de 5 à 10, suivant ces conseils.

15 Emission CO2

En France, les emissions sont en moyenne de 72g CO2eq/kWh en novembre 2020.

La puissance du serveur HP ProLiant MicroServer Gen8 G1610T peut être majorée par 50W:

Donc, le serveur consomme 50 x 24 x 365 = 438 000Wh = 438kWh, ce qui émet 438 x 72 = 31 536g CO2eq c'est à dire presque 32kg CO2eq/an.

Avec un rasberry pi, il semble que les résultats de consommation énergétique soient d'un ordre de grandeur en dessous !

Author: Maxime Buron

Created: 2021-12-09 jeu. 12:07

Validate