Saturday, February 19, 2022

TLS on DietPi with LetsEncrypt/Certbot


In a previous post, I went over the importance of using TLS whenever possible and demonstrated some steps to secure local applications such as Node-RED and Domoticz with self-signed SSL certificates. Since I have a domain name (with Ionos) that I use for my home lab, I figured I would try out the LetsEncrypt/certbot functionality built into DietPi OS. Typically, an SSL certificate would have a Certificate Signing Request (CSR) that that is manually generated then submitted to the Certificate Authority (CA) in order to obtain the SSL certificate and private key that is needed to configure TLS/HTTPS. 

Certbot automates most of this process, and automatically configures (some) web applications to use the new SSL cert. When running the process on DietPi OS, the Pi-hole installation is automatically secured. We will go through the steps to make this work, as well as manually configuring Node-RED and Domoticz to use the SSL cert/key moving forward. LetsEncrypt certificates are good for only 3 months, so you would need to repeat this process about every 2.5 months (so you may want to stick with self-signed certs if this is too inconvenient). If you are serving a public facing web server, the renewal process is automatic. Additionally, if you wish to have a longer-term certificate (up to 1 year for most domain providers), you could look into the SSL certificate options provided by your domain provider (usually a wildcard cert is free) and learn how to manually configure SSL certs for your resources (which doesn’t rely on the DNS/port 80 validation that certbot does). 

  1. In your domain management DNS records, add “A” record(s) for the subdomains you wish to use (to your public IP address found by searching “my ip address” in a search engine). This is required for the LetsEncrypt certbot verification to work.
    Graphical user interface, text, application, chat or text message

Description automatically generated

  2. In your router, create a port forwarding rule to forward port 80 to the IP address of the device you are securing. This is only temporary to complete the certbot process. Some ISPs may block port 80; if this is the case you may have luck trying
    A screenshot of a computer

Description automatically generated

  3. Run the command sudo dietpi-letsencrypt. Configure the options and select Apply.
    Graphical user interface, text, application

Description automatically generated

  4. When the configuration completes, take note of the certificate and key paths. Repeat steps 2-3 for any other subdomains/devices you wish to secure. IMPORTANT: The port forwarding rule from step 2 should now be deleted/disabled if this is a service you only intend to access on your local area network.  Although the DietPi documentation advises to keep port 80 forwarded for automatic renewal, in the day and age of Zero Day exploits such as Log4Shell, and the fact that these are services I am only using in my internal network, I prefer to do the renewal manually. You may also wish to delete the DNS records as well, as they can be easily recreated at the time the renewal will need to be completed.
    Text

Description automatically generated

  5. To access the DNS name locally, a mapping would need to be created in Pi-hole Local DNS -> DNS Records
    For this to take effect right away, you may need to clear your local DNS cache by running a command on the machine your browser is running on, such as ipconfig /flushdns
    Graphical user interface, text

Description automatically generated

  6. Accessing Pi-hole with the DNS name with HTTPS, the web interface is now secured with a proper SSL cert that the browser does not warn about.
    Graphical user interface

Description automatically generated

  7. At this point, for other applications to use the cert/key, we need to create a certs user group, add the Node-RED user account to the group, and grant the group permissions to read the cert/key files. This ensures that when the certbot renewal is manually run, the applications will continue to use the renewed certificate with no further configuration required.

    sudo groupadd certs

sudo usermod -a -G certs nodered

sudo usermod -a -G certs domoticz

sudo chown -R root:certs /etc/letsencrypt/live

sudo chown -R root:certs /etc/letsencrypt/archive

sudo chmod -R 750 /etc/letsencrypt/live

sudo chmod -R 750 /etc/letsencrypt/archive

Text

Description automatically generated


  1. To add the certificate to Node-RED, edit the settings.js file
    sudo cp /mnt/dietpi_userdata/node-red/settings.js /mnt/dietpi_userdata/node-red/settings.js.bak
    sudo nano /mnt/dietpi_userdata/node-red/settings.js
    Edit the lines used in the previous guide to specify the self-cert/key to instead use the cert/key files provided by the LetsEncrypt wizard. Use ctrl-X to save.
    Text

Description automatically generated

  2. Restart the Node-RED service and access Node-RED via the HTTPS URL and port 1880
    sudo systemctl restart node-red
    A screenshot of a computer

Description automatically generated with medium confidence

  3. For Domoticz, edit the domoticz.conf configuration file. Update the 3 SSL configuration lines as follows (with the cert/key file paths provided by the LetsEncrypt wizard). Save with ctrl-X
    sudo nano /mnt/dietpi_userdata/domoticz/domoticz.conf
    A screenshot of a computer

Description automatically generated

  4. Restart the Domoticz service
    sudo systemctl restart domoticz
    Domoticz is now accessible via the HTTPS URL on port 8424
    Graphical user interface, text, application, website

Description automatically generated


When it’s time for certificate renewal, you should get an email from LetsEncrypt (about half month before renewal). At this point some steps will need to be repeated. If the IP address provided by your ISP has changed (or if you deleted the DNS record), step 1 will need to be performed again. Steps 2 and 3 will need to be performed again to complete the renewal (just be sure to disable/delete the port forwarding rule when completed). Keep in mind that services will need to be restarted (or reboot the OS) to pick up the renewed certificate.

The certbot renewal process will automatically attempt to run, which in the case you’re serving a public web server, should take care of itself so long as the DNS record and port forwarding exists. Additionally, it is possible to perform validation with a DNS challenge, but this would require your DNS provider has an API and you would need build this into your certbot setup. 









Sunday, February 6, 2022

RadSens - Smart home enabled Geiger Counter

Last year I was checking out some projects on Hackster.io and I stumbled upon this post about an Arduino compatible dosimeter/radiometer based on a Geiger-Müller Tube. Somebody actually made an Arduino compatible Geiger Counter (what I am going to call it anyway), so naturally I had to add it to my collection. I previously wrote about monitoring radon (which can be more harmful than other ionizing radiation being that alpha radiation is the most destructive form of ionizing radiation when inhaled/ingested). 

While it may not be as important to measure other forms of ionizing events such as cosmic radiation, it is still interesting in that it could be used to correlate soft errors in high performance computing environments (which is typically mitigated by ECC memory or radiation-hardened microcontrollers). Not to mention the fact that quantum computing could be even more affected by cosmic radiation.  If you are reading this, like me you are probably simply concerned about long term health risks of exposure to radiation/ionizing events, like the risk of radon exposure. If a gamma-ray burst  or a cloud of radiation were to come through my area, I would like to know about it no matter how brief. Since I have the radon sensor in the basement, I am running the Geiger Counter setup on the second floor to hopefully catch any readings from airborne or cosmic sources of ionizing events. I will go over how I set this up to report sensor values into Domoticz, but there is also a setup for Home Assistant as well.  


Graphical user interface

Description automatically generated


  1. I placed an ESP32 dev board (ESP-WROOM-32 with pre-soldered pin headers) on a breadboard, connected the Radsens unit to the 3.3 and ground pins (with Dupont jumper wires), and connected the I2C pins to the D21 and D21 pins for the ESP32 dev board (refer to the schematics and the pinout for your specific dev board.

    The pin numbering starts from the corner of the board.
    Table

Description automatically generated

Pinout for the ESP-WROOM 32 dev board
Graphical user interface, chart

Description automatically generated

  1. Supplying USB power to the dev board powers the RadSens unit. Out of the box, functionality is confirmed by the blue LED lighting up every time an ionizing event is detected. Ideally a protective case would be built for this setup; otherwise make sure it’s in a place that it won’t be disturbed or exposed to water or humans/animals (touching the Geiger–Müller tube can result in false positive detection for ionizing events).

    A picture containing electronics, circuit

Description automatically generated

  2. I modified the I2C_rad_test.ino Arduino sketch to add support for MQTT to transmit sensor data to Domoticz. This required installing the Adafruit MQTT library, and I also used the Simple Timer library (in case I needed to add other functionality like other sensors to read from). If you haven’t yet added ESP32 support to the Arduino IDE, navigate to File -> Preferences and paste https://dl.espressif.com/dl/package_esp32_index.json into Additional Boards Manager URLs. To install the libraries, navigate to Tools -> Manage Libraries and search/install the libraries.

    Graphical user interface, text, application, email

Description automatically generated

    Adafruit MQTT library
    Graphical user interface, text, application, email

Description automatically generated
    SimpleTimer library
    Graphical user interface, text, application, email

Description automatically generated

  3. After connecting the dev board, be sure to set the board type and set the com port to whatever com port is detected. If your operating system did not automatically install the drivers, they can be downloaded from here.

    Graphical user interface, text, application

Description automatically generated

  4. I published the sketch (and required files from the original repository) on GitHub. You’ll need to download the .zip file, extract it, and open the Radsens_to_Domoticz-main.ino file. In the #define section, you’ll need to configure wifi settings, MQTT settings, and the IDX for a “custom sensor” virtual sensor for both the dynamic and static sensor readings. The sensor readings are measured in CPM for “count per minute” (of ionizing events). I added some functions to the original sketch (as to putting the additional code in the main loop) so as to easily update if a new version of the original sketch is released for a bugfix or new features. I also commented out almost all unnecessary code which can be un-commented for troubleshooting/debugging.

    Domoticz “custom sensor” virtual sensor
    Graphical user interface

Description automatically generated

    Settings to configure
    Text

Description automatically generated

    Function that takes sensor values, builds the payloads, and sends to Domoticz via MQTT
    Text

Description automatically generated

  5. After uploading the sketch, if everything has been configured properly, sensor readings will be available under the Utility tab in Domoticz.

    A screenshot of a computer

Description automatically generated with medium confidence

    Chart, line chart

Description automatically generated

  6. Notifications can be set for various sensor readings. I found a chart listing some levels of radiation and set some notifications. While some of the charts I found have slightly conflicting guidance (“seek shelter immediately” for > 100 CPM vs “no need to panic”), I decided to play it safe being that I don’t live near an industrial area.

    Domoticz notifications
    A screenshot of a computer

Description automatically generated

    Advice to shelter immediately with CPM readings of 100 or more
    Daily Local Background Radiation Levels and Advisories :: Sunshine Coast  Computer Club

    Advice not to panic, but to take caution for CPM readings of 100 or moreTable

Description automatically generated

  7. The dynamic sensor readout is useful in detecting short term conditions, such as for a local pollution search (or perhaps a gamma-ray burst).

    Text, letter

Description automatically generated

  8. The static sensor readout is more appropriate for measurement of constant background radiation.

    Text, letter

Description automatically generated

  9. The impulse counter could be used to augment the built in LED that flashes every time an ionizing event occurs. Perhaps some code with a short delay in the main loop that sends a UDP real-time packet to a WLED device to trigger some fancy radiation themed lighting. Maybe I will write about this in a future post!

    Text

Description automatically generated

  10. Also, for additional insight into the data, I have my MQTT broker bridged with AWS IoT Core with a rule to send sensor data to AWS Timestream (which can be visualized with Grafana, which I have running in docker/ECS). I also have a rule that sends the data to a lambda function that formats the data as StatsD format and sends it over to a Graphite-Statsd docker container I have running on an ECS instance. 

    Timestream + Grafana
    Graphical user interface, chart

Description automatically generated

    Lambda function + Graphite-Statsd
    Chart, histogram

Description automatically generated

Sunday, January 30, 2022

Securing Node-RED, Domoticz, Pi-hole and MQTT with TLS

In my last post I emphasized some important aspects of home network security, but I skipped over the details of configuring web interfaces and SSH sessions with TLS (Transport Layer Security) support. The importance of using TLS is that this prevents passwords and other important information from traversing the network in plaintext (not encrypted). Although in an ideal world, a home network wouldn’t have any malicious devices sniffing the network for plaintext data, the reality is that supply chain attacks are occurring more frequently, and malware infections are more likely to happen now than ever. 

The difference that TLS could make in such a situation, is that a data breach on your home network is less likely to occur, and a malware infection is detected and removed (either by manual means or automatically, such as malware detection updates, app stores removing and automatically uninstalling malicious apps) before any damage can be incurred. Additionally, aside from reducing the risk of a data breach, using TLS and other security practice whenever possible in your network (home network or not) can stop lateral moving threats from gaining a foothold and becoming an APT (advanced persistent threat). In this guide I will go over some security measures that can be taken for DietPi OS, Node-RED, Domoticz, Pi-hole, and security implications to consider when using MQTT.


DietPi SSH security

  1. Open PuttyGen and click the Generate button. Move the mouse around the screen to generate some randomness until the progress bar is complete.
    Graphical user interface, application

Description automatically generated

  2. Enter a passphrase on the key and save the private key somewhere (and to a backup location for good measure). There is no expiration on this keypair to worry about.
    Graphical user interface, text, application, email

Description automatically generated

  3. Copy the entire public key string (beginning with ssh-rsa) at the top. SSH into the DietPi machine and create/edit the authorized keys file
    mkdir ~/.ssh
    nano ~/.ssh/authorized_keys
    Paste the public key into the file and save (Ctrl-X)
    Text

Description automatically generated

  4. Edit the DropBear SSH configuration
    sudo cp /etc/default/dropbear /etc/default/dropbear.bak
    sudo nano /etc/default/dropbear
    Edit the config file to disable root and password logins
    DROPBEAR_EXTRA_ARGS="-w -s"
    Save the file (Ctrl-X) and confirm.
    Text

Description automatically generated

  5. Restart the DropBear SSH service
    sudo systemctl restart dropbear
    Graphical user interface, text

Description automatically generated

  6. If using the OpenSSH service included with DietPi (non-default), run the following and make the following changes
    sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
    sudo nano /etc/ssh/sshd_config
    Comment out the PermitRootLogin line
    Uncomment the PubkeyAuthentication line
    Uncomment the AuthorizedKeysFile line
    Uncomment the PasswordAuthentication line and change to no
    Save with Ctrl-X
    sudo systemctl restart sshd
    Text

Description automatically generated

  7. Open PuTTY, select the session, and click the Load button. Navigate to Connection -> SSH -> Auth
    Graphical user interface, application

Description automatically generated

  8. Add the password protected private key that was created earlier. Be sure to go back to Session and save the sessionGraphical user interface, text, application

Description automatically generated

  9. Open the session and you are now prompted to enter the password to decrypt the private key and you should be more securely connected via SSH now.
    Text

Description automatically generated

Local DNS names (for web SSL certs)

Log into both Pi-hole web interfaces (if you have two) and set the local DNS names you will be using to access your devices (Settings -> Local DNS -> DNS records). Usually this would be the hostname of the device followed by .local. This is required for web site SSL certificates to work as intended on the local network.
A screenshot of a computer

Description automatically generated

Node-RED

  1. Run the following command to generate the required key and certificate. This creates a certificate that is valid for 10 years (and the -nodes flag disables the requirement to set a password on the certificate)
    mkdir ~/nodecerts
    openssl req -x509 -newkey rsa:2048 -keyout nodecerts/node-key.pem -out nodecerts/node-cert.pem -sha256 -days 3650 -nodes
    You don’t need to enter any information until you get to the common name (hit enter until you get to this prompt). For the common name enter the DNS name of the device you are securing (that you configured on the Pi-hole web interfaces).
    Text

Description automatically generated

  1. Copy the nodecerts directory to the Node-RED installation directory (this command will work for Node-RED installed on DietPi
    cp -R nodecerts/ /mnt/dietpi_userdata/node-red/
    Then change directory to the Node-RED install location
    cd /mnt/dietpi_userdata/node-red/
    Change ownership of the nodecerts directory to the Node-RED user
    sudo chown -R nodered:nodered nodecerts
    A screenshot of a computer

Description automatically generated

  2. To generate a password hash that will be used to secure Node-RED, enter the command
    node-red-admin hash-pw
    This hash translates to “Examplepassword” which I’m not actually using. Copy the hash (selecting the string will automatically copy it into the clipboard) and save it to enter into the settings.js file a few steps later.
    A screenshot of a computer

Description automatically generated

  3. Make a backup of the settings.js file
    sudo cp settings.js settings.js.bak
    Edit the settings.js file
    sudo nano settings.js
    Text

Description automatically generated



  1. Hit Ctrl-W, type cert: and hit enter to find the required configuration stanza
    Uncomment out the lines like in the screenshot (delete the slashes at the beginning of the lines)
    In the key path paste: /mnt/dietpi_userdata/node-red/nodecerts/node-key.pem
    In the cert path paste : /mnt/dietpi_userdata/node-red/nodecerts/node-cert.pem
    A screenshot of a computer

Description automatically generated

  2. Hit Ctrl-W and search for adminAuth. Uncomment out the lines like below and replace the password hash with the one you created.
    A screenshot of a computer

Description automatically generated

  3. Hit Ctrl-W again and search for requireHttps. Uncomment out this line.
    A screenshot of a computer

Description automatically generated

  4. Hit Ctrl-X and confirm to save the file. Run the following command to restart Node-RED
    sudo systemctl restart node-red
    Graphical user interface, text

Description automatically generated

  1. Navigate to Node-RED with the HTTPS URL, the DNS name, and don’t forget port 1880. This warning is displayed because the certificate is self-signed and not issued by a public certificate authority. It is possible to add the certificate as a trusted certificate but that differs per browser/operating system. To bypass the warning, click Advanced, and Proceed to [DNS name]
    Graphical user interface, text, application, email

Description automatically generated

  2. If running into issues, the command sudo journalctl -b -u node-red can be run to look at the service logs.

  3. The certificate and expiration date are visible from the browser
    Graphical user interface, text, application, email

Description automatically generated

Pi-hole

Unfortunately, due to the way that Pi-hole is installed with DietPi, it doesn’t seem so straightforward to use a self-signed certificate to configure on the Pi-hole web interface. If you use dietpi-letsencrypt (which is only for public facing hostnames), DietPi should automatically configure SSL for your Pi-hole web interface, but we aren’t doing that (can’t think of any good reasons to have it public facing). Fortunately, we can still use SSH tunneling to prevent the password from being transmitted in plaintext.


  1.  Edit the PuTTY session you use to SSH to your Pi-hole host. Navigate to Connection -> SSH -> Tunnels. Enter the desired source port (to connect via your local machine) and set the destination IP and port to match that of the Pi-hole instance. For a second Pi-hole you could source port 81 for the local connection. Save and open the session.
    Graphical user interface, application

Description automatically generated

  2. After authenticating via SSH, the tunnel is now connected, and you can log into the Pi-hole locally via the tunnel without transmitting the password over the network via plaintext
    http://127.0.0.1/admin/
    Graphical user interface, application

Description automatically generated

  3. SSH tunneling can also be a useful tool in bypassing a network firewall in a corporate network, for example to test the web interface on a development machine where development is being done locally, yet a firewall entry hasn’t yet been created to allow traffic through. 


Domoticz

  1. The Domoticz install that is optimized for DietPi fortunately does have a built-in self-signed SSL certificate. The HTTPS web interface is available on port 8424:
    Graphical user interface, text, application, chat or text message

Description automatically generated

  2. This certificate is only valid for a few years but will likely be updated when Domoticz is updated.
    Graphical user interface, text, application, email

Description automatically generated

MQTT


Unfortunately, although both the Mosquitto MQTT broker and Domoticz support TLS, ESP8266 devices are not built with MQTT/TLS support due to the lack of device memory. ESP32 devices can however be built to support MQTT/TLS. The M5stack ESP32 based devices support MQTT/TLS configuration via UiFlow (blockly based design system) or with Arduino libraries. Tasmota for ESP32 appears to support MQTT/TLS which I will try to explore soon. The ESPurna firmware also appears to support MQTT/TLS however this would require building the firmware yourself. The ESPeasy firmware doesn’t even appear to support it. So, at this point I have the following thoughts regarding securing MQTT:

  • The MQTT broker password could easily be sniffed on the local area network and makes the broker susceptible to Denial of Service attacks (including “Denial of Wallet” attacks if you have an MQTT bridge to AWS IoT core like I do), data corruption attacks, and not to mention the loss of privacy regarding IoT data sent via MQTT (although sensor data doesn’t seem very revealing).

  • The MQTT broker password should *definitely not* be re-used anywhere (not that you should be re-using passwords at all), even on the local network, being that if a data breach occurs that would likely be one of the first credentials to be exfiltrated. This would facilitate the ability of a lateral-moving threat to move across your network and become an APT (advanced persistent threat). 

  • MQTT with TLS support should be explored with priority. This means either compiling firmware that supports MQTT or purpose-built Arduino-based code (or potentially FreeRTOS) that supports only the functionality required for a given implementation (including of course, storing the required key/certificate file required for TLS). 

  • In the meantime, an IoT specific VLAN could be implemented to isolate the MQTT traffic to the same network as the MQTT broker and IoT devices, however this is not so feasible for home networks (and even though I have an EdgeRouter which supports such configuration, I would probably mess it up).