Chère de Prince
Table of Contents
1 General
1.1 Domain Names
1.1.1 buron.coffee
- book.buron.coffee
- buron.coffee
- emby.buron.coffee
- headphones.buron.coffee
- mathilde.buron.coffee
- medias-dl.buron.coffee
- mpd.buron.coffee
- music.buron.coffee
- piwigo.buron.coffee
- radio.buron.coffee
- shaarli.buron.coffee
- transmission.buron.coffee
1.1.2 cheredeprince.net
- cheredeprince.net
- conway.cheredeprince.net
- herculevshydre.cheredeprince.net
- life.cheredeprince.net
- math.cheredeprince.net
- owncloud.cheredeprince.net
- rss-bridge.cheredeprince.net
- rss.cheredeprince.net
- static.cheredeprince.net
- syracuse.cheredeprince.net
- votenroll.cheredeprince.net
- wallabag.cheredeprince.net
1.2 Activate Security Updates
1.3 IP V6
It seems mandatory in order that synapse federation works properly.
The livebox may block some ports with its firewall.
2 Administration
Some tools, I use to check that everything is all right.
2.1 Useful Commands
- Find the largest files in a folder
find /mnt/pellicule/ -type f -printf "%s\t%p\n" | sort -n | tail
- List the extensions of files in a directory
find /mnt/super8/films/ -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u
- find the total size of the files created during the last month :
find /data/youtube-dl-archives/ -type f -ctime -30 -print0 | du -shc --files0-from - | tail -n1
- find the total size of the files accessed during the last month :
find /data/youtube-dl-archives/ -type f -atime -30 -print0 | du -shc --files0-from - | tail -n1
find the most recent file in a directory:
find /mnt/super8/ -type f -printf "%T@ %p\n" | sort -n | cut -d' ' -f 2- | tail -n 1
2.2 Network
# network usage during the current day vnstat vnstati -vs -o /tmp/vnstat.png -i eno1 # live network usage ifstat # network usage per process nethogs
2.3 Fail2ban
I follow this tutorial.
2.4 Security
chmod 740 /data/owncloud /data/piwigo /data/backup-db /var/lib/mysql* /var/lib/postgresql /var/lib/mongodb /var/backups /var/log/nginx/*
3 Backup
3.1 General
There are two main backup storages:
- The backup cheredeprince server is a daily backup of:
/mnt/pellicule
movies hard drive (3To full)/mnt/super8
video hard drive (3To)- important directories of the server
/etc /home /root /opt /var /data/music /data/owncloud /data/piwigo /data/youtube-dl-archives /data/backup-db # dumps of databases
- An external encrypted hard drive called coffre (3To) saving:
- version of images of the root hard drive of the server
- an archive of the hard drive
/data
The *machine uses the following hardware:
3.2 Borg Configuration (important directories backup)
I use this repository for installation on client and server side: https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup
3.2.1 Basic commands
To get the list of all archives. The first column contains archive id.
export BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg list ssh://backup.cheredeprince.net:1516/mnt/backup/backup/
blackcube-31.07.2018 Tue, 2018-07-31 03:00:09 blackcube-31.08.2018 Fri, 2018-08-31 03:00:08 blackcube-30.09.2018 Sun, 2018-09-30 03:00:08 blackcube-31.10.2018 Wed, 2018-10-31 03:00:09
To information about one archives particular archive (using its id, see above)
export BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info ssh://backup.cheredeprince.net:1516/mnt/backup/backup/::blackcube-31.07.2018
To get the list of a path in a specific archive
export BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg list ssh://backup.cheredeprince.net:1516/mnt/backup/backup/::blackcube-31.07.2018 data/backup-db/
3.3 Code on Paper
In order to be sure that passwords and others vital information are safely saved, we have to print it.
I use qrencode
to encode text in a qrcode, install it with:
sudo apt-get install libqrencode3
and SecScanQR to decode using an android smartphone: https://f-droid.org/en/packages/de.t_dankworth.secscanqr/
To encode some text:
qrencode "text" -o qrcode.png
print qrcode.png
and delete it.
3.4 Configuration du serveur de backup
3.4.1 SSH config
Host backup HostName backup.cheredeprince.net Port 1516 User root IdentityFile ~/.ssh/id_rsa
Le port ssh sur la machine backup.cheredeprince.net est 22 !
3.5 Accès de local vers backup
L'adresse ip local du serveur est 192.168.1.21. L'accès local des partitions est fait par NFS avec la configuration :
# /etc/exports /mnt/pellicule 192.168.1.0/24(ro,sync,all_squash,anonuid=1000,anongid=1000,subtree_check) /mnt/super8 192.168.1.0/24(ro,sync,all_squash,anonuid=1000,anongid=1000,subtree_check)
3.5.1 Samba
https://www.howtoforge.com/tutorial/samba-server-ubuntu-16-04/
installation
apt-get install -y samba samba-common python-glade2 system-config-samba
# /etc/smb.conf [global] workgroup = WORKGROUP server string = Samba Server %v netbios name = ubuntu security = user map to guest = bad user dns proxy = no #============================ Share Definitions ============================== [Anonymous] path = /samba/anonymous browsable =yes writable = yes guest ok = yes read only = no force user = nobody
chmod -R 0775 /mnt/super8/films-martine chown -R nobody:nogroup /mnt/super8/films-martine service smbd restart
Il y a des problèmes quand on déplace des dossiers, on a pas les droits d'écrire les fichiers à l'intérieur …
Il y a une source dans Emby de /mnt/super8/films-martine
par Samba.
4 Let encrypt
4.0.1 Get a new certificat
certbot certonly --webroot -w /var/www/letsencrypt -d {domain}
4.0.2 Renew existing certificats
certbot renew
5 Iptable
Iptable rules definitions can be found in /etc/iptables.sh
6 Matrix
6.1 Riot
6.1.1 Login
Il semble qu'au login, il faut
- spécifier le domaine du serveur
- entrer le nom d'utilisateur local (emixam150 et pas @emixam150:cheredeprince.net) et le mot de passe.
Pour que la version "simple" (sans spécifier le nom de domaine et utiliser @emixam150:cheredeprince.net) fonctionne au login, il faut définir un accès vers .well-known/matrix/client (https://github.com/vector-im/riot-web/issues/9224).
6.2 Synapse
D'après cette documentation, il faut utiliser un certificat signé (comme ceux fournis par letsencrypt) pour les connections de fédération. Vu que le nom de domaine du serveur matrix est "cheredeprince.net", je peux utiliser le certificat que j'utilise pour ce domaine. Et ça marche !
6.3 Mautrix Telegram
6.3.1 Dependencies
installation of libolm3.xx (optional and still not working)
cd /tmp git clone https://gitlab.matrix.org/matrix-org/olm.git cd olm cmake . -Bbuild cmake --build build cp build/libolm.so /usr/lib/x86_64-linux-gnu/libolm.so
6.4 Mautrix Facebook
6.4.1 Bridge setup
https://github.com/tulir/mautrix-facebook/wiki/Bridge-setup
#create user useradd -M --shell /bin/false mautrix-facebook usermod -L mautrix-facebook #create directory mkdir /usr/share/mautrix-facebook cd /usr/share/mautrix-facebook # install in root virtualenv -p /usr/bin/python3 . source ./bin/activate pip install --upgrade mautrix-facebook # config and change the port to 8006 (for telegram it is 8005) cp lib/python3.6/site-packages/mautrix_facebook/example-config.yaml config.yaml chown -R mautrix-facebook:mautrix-facebook . # Generate the appservice registration sudo -u mautrix-facebook bin/python3.6 -m mautrix_facebook -g # Please ensure that the file /etc/matrix-synapse/homeserver.yaml contains: # app_service_config_files: ['/usr/share/mautrix-telegram/registration.yaml', '/usr/share/mautrix-facebook/registration.yaml'] # update db sudo -u mautrix-facebook bin/alembic upgrade head # start with sudo -u mautrix-facebook bin/python3.6 -m mautrix_facebook
6.4.2 Authentication
https://github.com/tulir/mautrix-facebook/wiki/Authentication
Je suivis la méthode : login-cookie.
6.4.3 Problème
Pas de sync Facebook -> matrix
revenir dans la conversation avec le bridge et faire sync
.
6.5 Matrix Registration
Matrix registration is a project aiming at allow people to register to a matrix server using web page link.
The installation command is:
pip3 install matrix-registration
Then we create an user for the daemon
useradd -M --shell /bin/false matrix-registration usermod -L matrix-registration
then we create working directory:
mkdir /etc/matrix-registration chown matrix-registration /etc/matrix-registration chmod 770 /etc/matrix-registration
copy the following file to /etc/matrix-registration/config.yaml
server_location: 'https://cheredeprince.net server_name: 'cheredeprince.net shared_secret: 'copy here the registration_shared_secret value from homeserver.yaml of matrix-synapse' admin_secret: '' base_url: '' riot_instance: 'https://riot.im/app/' db: 'db.sqlite3' host: 'localhost' port: 5000 rate_limit: ["100 per day", "10 per minute"] allow_cors: false logging: disable_existing_loggers: False version: 1 root: level: DEBUG handlers: [console, file] formatters: brief: format: '%(name)s - %(levelname)s - %(message)s' precise: format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s' handlers: console: class: logging.StreamHandler level: INFO formatter: brief stream: ext://sys.stdout file: class: logging.handlers.RotatingFileHandler formatter: precise level: INFO filename: m_reg.log maxBytes: 10485760 # 10MB backupCount: 3 encoding: utf8 # password requirements password: min_length: 8
and create a service file /etc/systemd/system/matrix-registration.service
:
[Unit] Description=Matrix Registration [Service] WorkingDirectory=/etc/matrix-registration/ Type=simple User=matrix-registration ExecStart=/usr/bin/python3 /usr/local/bin/matrix-registration serve Restart=always RestartSec=10 [Install] WantedBy=default.target
enable and start the service
systemctl enable matrix-registration systemctl start matrix-registration
list the tokens with:
curl -X POST -H "Authorization: SharedSecret shared_secret" http://localhost:5000/token
6.6 Guest room
https://github.com/vector-im/element-web/issues/9264#issuecomment-699536161
j'active l'accès guest sur synapse :7 Bobine
The root is /var/www/buron.coffee/bobine
and the content is generated using /home/becasse/tinymoviemanager
We create an symbolic link for create path to the movies
ln -s /mnt/ /var/www/buron.coffee/bobine/mnt
To update the website:
cd build # update the db, scrape the new files metadata, rename the new scraped files ./tinyMediaManagerCMD.sh -updateMovies1 -scrapeUnscraped -rename ./tinyMediaManagerCMD.sh -export DarkTemplate /var/www/buron.coffee/bobine/ ./tinyMediaManagerCMD.sh -export DarkTemplateTV /var/www/buron.coffee/bobine/
To update TinyMediaManager
cd ~/tinyMediaManager git pull git checkout origin/disableimgreexport #save data directory cp -r build/data . # buiding mvn package # restore data cp -r data build/ # basic setup cd build chmod +x *.sh # do not ask me why mv getdown-new.jar getdown.jar # at least to generate templates ./tinyMediaManager.sh
Here a little script to insert a movie from the torrent directory
# copy the movie film cp /data/torrents/$1 /mnt/super8/films/ # update the bobine database and website cd /home/becasse/tinyMediaManager/build ./tinyMediaManagerCMD.sh -updateMovies1 -scrapeUnscraped -rename ./tinyMediaManagerCMD.sh -export DarkTemplate /var/www/buron.coffee/bobine/ ./tinyMediaManagerCMD.sh -export DarkTemplateTV /var/www/buron.coffee/bobine/
7.1 Mount Movie Folders
sshfs -o default_permissions,IdentityFile=~/.ssh/id_rsa bcdp:/mnt/super8 /mnt/super8 -o idmap=user sshfs -o default_permissions,IdentityFile=~/.ssh/id_rsa bcdp:/mnt/pellicule /mnt/pellicule -o idmap=user
7.2 pipeline for json database
7.2.1 First idea: export the database in XML and translate it into JSON
./tinyMediaManagerCMD.sh -export ListExampleXml /tmp/ xml2json -t xml2json -o /tmp/movielist.json /tmp/movielist.xml --pretty --strip_text
Why not using xml2jsonmappings ? It is a bug with the export of Person ids with new TMM version …
7.2.2 Second idea: use the .nfo files to create xml database
DIRS="/mnt/super8/films" OUTPUT="/tmp/movies.json" > $OUTPUT for dir in $DIRS; do for f in $dir/*/*.nfo; do echo $f; xml2json -t xml2json "$f" --pretty --strip_text >> $OUTPUT; done; done;
reshaping the database file by tags:
jq -s '[.[].movie| select(has("tag") and has("tmdbid") and .tmdbid != null) | select(.tag | type =="array") | {id: .tmdbid, tag: .tag[]}] | group_by(.tag)| [.[]|{tag:.[0].tag, ids:[.[].id]}]| sort_by(.ids | length)| reverse | [limit(10; .[])]' /tmp/movies.json
We can split the previous command in the following steps as a map and reduce program:
[.[].movie| select(has("tag") and has("id") and .id != null) | select(.tag | type =="array") | {id: .id, tag: .tag[]}]
builds an array of tag by movie id,..| group_by(.tag)| [.[]|{tag:.[0].tag, ids:[.[].id]}]
reduces the array by grouping the movie ids by tags,..| sort_by(.ids | length)
sorts the tags by number of movies.
8 MySQL
Connection
mysql -u root -p<pass>
dump all the database
mysqldump -u root -p<pass> --all-databases > backup.db
Restore a database from a all database dump
mysql -u root -p<pass> --one-database database_name < backup.db
List the databases
SHOW DATABASES;
Delete a database
DROP DATABASE name;
9 WriteFreely
depot: https://github.com/writeas/writefreely
The install path is: /var/www/buron.coffee/writefreely
9.1 Update
Most of the time, the update are resumed by the following lines:
# as root systemctl stop writefreelyburon.service # backup the databases mysqldump -u root -p<pass> --all-databases > backup.db # as becasse cd /var/www/buron.coffee/ # backup the folder cp -r writefreely ~/ # download and install the update wget path to tar.gz build tar -xvzf downloaded file # migrate the database cd writefreely ./writefreely -migrate # as root systemctl start writefreelyburon.service
10 FreshRSS
La version actuelle est 1.17.1-dev
Le répertoire d'installation est /usr/share/FreshRSS
Pour lister les utilisateurs par date de dernières connections:
./cli/user-info.php -h | sort -k3 -r
11 RSS Bridge
11.1 Instagram Bridge
Le 14/12/2020, je constate que rss bridge est utilisé à plus de 50% pour Instagram:
tail /var/log/nginx/cheredeprince.net/rss-bridge.access.log -n 100000 | grep "Instagram" | wc -l # 53713
Le problème est multiple:
- depuis plusieurs mois, il y a une erreur sur le bridge, qui n'est plus maintenu : https://github.com/RSS-Bridge/rss-bridge/issues/1891
- le nombre de requête vers Instagram provenant de mon serveur est sûrement trop important et son IP a dû être blacklistée.
Le 14/12/2020, je décide de désactiver le brigde pour quelque jours histoire de voir, s'il marchera mieux plus tard.
J'ai essayé de ne permettre que les requêtes locales avec nginx, par example celles de FreshRSS, mais ça ne marche pas:
if ($args ~ "bridge=Instagram") { # disabled the bridge for external requests !!!! # allow anyone in 192.168.1.0/24 allow 192.168.1.0/24; allow 86.247.20.120; allow 2a01:cb04:751:b600::/56; # drop rest of the world deny all; }
12 Wallabag
The actual version of wallabag is 2.4
12.1 Update
cd /var/www/cheredeprince.net/wallabag/
sudo -u www-data make update
13 Owncloud
La version actuelle est 10.5.0
13.1 Hack
For forcing shared videos player to be not larger than the window, I have to add into /var/www/cheredeprince.net/owncloud/core/css/styles.css
:
video { max-width: 100% !important; }
14 Php
14.1 Errors and slow executions
J'ai découvert qu'il y avait cette erreur:
tail /var/log/php7.2-fpm.log [14-Jul-2020 13:34:02] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
J'ai passé max_children
de 5 à 10, suivant ces conseils.
15 Emission CO2
En France, les emissions sont en moyenne de 72g CO2eq/kWh en novembre 2020.
La puissance du serveur HP ProLiant MicroServer Gen8 G1610T
peut être majorée par 50W:
- https://community.hpe.com/t5/proliant-servers-netservers/proliant-gen8-microserver-idle-power-consumption-50w/td-p/6937756
- https://www.reddit.com/r/homelab/comments/3l0t8z/power_consumption_of_hp_proliant_microserver_gen8/
Donc, le serveur consomme 50 x 24 x 365 = 438 000Wh = 438kWh
, ce qui émet 438 x 72 = 31 536g CO2eq
c'est à dire presque 32kg CO2eq/an.
Avec un rasberry pi, il semble que les résultats de consommation énergétique soient d'un ordre de grandeur en dessous !