How to merge Apache access logs and sort them by date

Just dropping this here:

https://gist.github.com/Cybso/77b8f7bdec78e0a8df8f8435e3ccb8fc

# Sorts the rows of one or multiple apache access_log files by date (ascending) 
# and converts them from 'common' log format
# into 'combined' (if you switched the log file format meanwhile).
#
# This is useful when you want to merge multiple log files, e.g. when
# you have different files for HTTP and HTTPS.
#
# Usage:
#     sort-access-log [file [file...]]
#
# If the argument ends on .gz it will be extracted using 'gunzip'.
# If no argument has been given the script expects the input on STDIN.
#
# The date is expected to have the following format, which must occur
# *anywhere* within the line:
#
#     [20/Mar/2017:12:00:37 +0100]
#
# The timezone argument is ignored, so this might be inaccurate when
# changing from summer time to winter time - for me this doesn't matter.
#
# The output of the program can be used as input for awffull or
# webalizer.

• #apache
• #github

Put a stamp on the last page of a PDF document

PDFTK, the swiss-knife pdf toolkit, can be used to put a stamp an all pages of a pdf file; but it is not possible to put the stamp on the last page only. There is a commercial tool "stamptk" available for this. But you can also workaround this with a few lines of shell code:

input="INPUT.pdf"
stamp="STAMP.pdf"
output="OUTPUT.pdf"
tmp1="tmp1.pdf"
tmp2="tmp2.pdf"

pdftk "$input" cat end output "$tmp1"
pdftk "$tmp1" stamp "$stamp" output "$tmp2"
pdftk "$input" cat 1-r2 output "$tmp1"
pdftk "$tmp1" "$tmp2" cat output "$output"

I've put this code into a simple bash script and uploaded it to github: stamp-last-page.sh.

Program Arduino Pro Mini with CP2102 USB module

When I ordered a bunch of Arduino Pro Mini clones I didn't notice that these don't come with an integrated USB-to-Serial adapter (in fact I needed Arduino Nano - well, I was relative new into this).

So to get a change to use them I've also ordered a CP2102 board that converts between USB and TTL. Since it took a while for me to figure out how to use that I will document it here.

First, connect the CP2102 with the Arduino:

Next, select the correct configuration.

Arduino IDE

Board: "Arduino Pro or Pro Mini"
Processor: "ATmega328 (5V, 16MHz)" (or the one that is soldered on your board)
Port: /dev/ttyUSB0 (or whatever port the CP2101 is connected to)
Programmer: "ArduinoISP"

Eclipse

Board: "Arduino Pro or Pro Mini"
Processor: "ATmega328 (5V, 16MHz)" (or the one that is soldered on your board)
Port: /dev/ttyUSB0 (or whatever port the CP2101 is connected to)
Upload Protocol: "Default"

Command Line

$ avrdude -C/etc/avrdude.conf <strong>-patmega328p -carduino -P/dev/ttyUSB0 -b57600 </strong>-D -Uflash:w:PATH_TO_HEXFILE:i

• #arduino

KDE: Remove "Switch User" and "Start New Session" from Plasma's lockscreen

If you are the only user on your workstation you probably do not have any need for the "Start New Session" button on Plasma's lockscreen. Quite the contrary it's extremly anoying if you just wanted to unlock your session and accidentally created a new one instead.

These is no UI switch to disable this button but you can use the KDE Action Restrictions for this. Just append the following lines to your $HOME/.config/kdeglobals (or $HOME/.kde/share/config/kdeglobals) and login again:

[KDE Action Restrictions][$i]
action/switch_user=false
action/start_new_session=false

• #kde

How to wake on lan a remote host on demand using systemd's sockets

Some years ago I wrote an article on how to wake on lan a (SSHFS) fileserver on demand using autofs.

Today I want to describe a more generic way to do the same. This should not only work for SSH but for every services that communicates via TCP. Everything you need is systemd, netcat (nc) and a wake-on-lan tool like etherwake.

For the impatient

The first step is to create a systemd socket that listens on a local port for new connections. Since SSH normally uses port 22 I'm going to bind this on localhost:2222. Of course you can also listen on a public interface if you are building a public gateway to a server within the local net.

/etc/systemd/system/<b>forward-ssh.socket:

[Socket]
ListenStream=127.0.0.1:2222
Accept=true

[Install]
WantedBy=sockets.target

The second file we have to create is a service file that will be invoked by the socket. The basename must be identical to that of the socket, but followed by an @ sign and the .service suffix.

/etc/systemd/system/<b>forward-ssh@.service:

[Unit]
Description=Forwards request to another host. Sends WOL if not reachable.
Wants=network-online.target
After=network.target network-online.target

[Service]
ExecStartPre=/bin/sh -c 'for n in `seq 1 6`; do nc -z 192.168.2.2 22 && break || (wakeonlan D8:F4:62:67:86:3A >&2 && sleep 10); done'
ExecStart=/bin/nc -q0 192.168.2.2 22
StandardInput=socket
StandardOutput=socket

Replace the the IP "192.168.2.2", Port "22" and MAC address "D8:F4:62:67:86:3A" with those of the server that you want to connect to.

Next, reload and enable the daemon (Note: You have to enable the socket, not the service):

$ systemctl daemon-reload && systemctl start ssh-forward.socket && systemctl enable ssh-forward.socket

So, what does this do?

The socket file tells systemd to listen in Port 2222 for incoming connections. Whenever (this is what "Accept=true" means) a new connection to this port is established the service is invoked.

The service file first launches the command in ExecStartPre. You can ignore the "/bin/sh -c '...'" wrapper for the moment, it is only required because systemd does not allow shell operators within the command:

$ for n in `seq 1 6`; do nc -z 192.168.2.2 22 && break || (wakeonlan D8:F4:62:67:86:3A >&2 && sleep 10); done

nc -z 192.168.2.2 22 checks if the remote port is reachable. This is more reliable than "ping" (as I used in my old article) since ICMP might be blocked by a firewall. If the port is reachable ExecStartPre returns immediately. If not, it sends a Wake-on-Lan package to the server's MAC address and waits 10 seconds for it to wake up. After that the check is done again, up to 5 times (seq 1 6), so the server has to wake up and boot within 60 seconds.

When ExecStartPre returns than ExecStart is invoked:

ExecStart=/bin/nc -q0 192.168.2.2 22
StandardInput=socket
StandardOutput=socket

This opens a TCP connection to 192.168.2.2 on port 22, and pipes the incoming socket's connection to STDIN/STDOUT. When the connection to localhost:2222 finally is closed the nc process will end too, so the remote server can go to sleep again.

And what is this good for?

My fileserver goes to sleep after an idle time of 15 minutes. I have a Raspberry Pi behind my router that is running Nextcloud. Within Nextcloud I've defined a remote connection to the fileserver. This trick allows me to wake up the fileserver only if Nextcloud demands access to the fileserver.

Since Nextcloud only waits up to 10 seconds for an SFTP connection you might also want to apply this small patch to the the Nextcloud source that increases the timeout to 60 seconds:

$ patch -p1 <<EOF
--- a/apps/files_external/lib/Lib/Storage/SFTP.php	Tue Feb 07 21:48:03 2017 +0100
+++ b/apps/files_external/lib/Lib/Storage/SFTP.php	Tue Feb 07 21:49:28 2017 +0100
@@ -124,7 +124,7 @@
 		}
 
 		$hostKeys = $this->readHostKeys();
-		$this->client = new \phpseclib\Net\SFTP($this->host, $this->port);
+		$this->client = new \phpseclib\Net\SFTP($this->host, $this->port, 60);
 
 		// The SSH Host Key MUST be verified before login().
 		$currentHostKey = $this->client->getServerPublicHostKey();
EOF

But why not running the fileserver directly on the Pi?

• RAID 5
• Gigabit-Ethernet
• BananaPi sucks (sorry, but that's true. I've tried it. Terrible software support. Really.)

• #linux
• #systemd