Wednesday, March 25, 2020

My first DevOps Journey - Part 2: Node.JS, Jenkins, PM2, and Docker

I decided to turn my Node-RED flow (from part 1) into a "production" application, and I knew that I wanted to use Node.js and Docker. What I didn't know, was how I was going to deploy it. I started looking at creating a dockerfile from scratch, but then I remembered PM2, a process manager for production Node.js applications. So I did the next thing that was logical to me: I googled "PM2 docker". I came across the Github page for the Official Docker Image for PM2 runtime. With this in mind, and knowing that I wanted to use Jenkins as my CI (Continuous Integration) tool, I pretty much had everything I needed to get started.


Now I don't "know" Node.js, but I had already used JavaScript a little bit for writing functions inside Node-RED. It was familiar enough to me that I felt really comfortable taking example code and StackExchange snippets and throwing something together. So... here is what I came up with (I used VSCode for this, and I called this project "darkstats" as a combination of DarkSky and StatsD)

Now if you use Node.js, you might easily be able point out how this code is flawed, or how it could be refactored (if this is the case, please let me know in the comments). Some noteworthy points about my experience:

1. I had to escape my quotes inside the JSONata expressions (\")
2. I'm using PM2 process metrics to measure total number of requests and request rate
3. If you're getting the errors "declaration or statement expected" or " ';' expected", you probably have an extra parentheses ")" in your code somewhere
4. I could not figure out how to use the Cheerio library for CSS-select for the life of me, but googleing it hard enough I was able to find a good example
5. I could have originally gone with subscribing to an RSS feed for one of my data sources, but learning how to scrape (some) webpages with CSS-select seemed to be a better use of my time


I found out about Jenkins a couple of years ago, from a co-worker who is a brilliant architect. I was about to get pulled into a project using it, but it turns out the documentation for the Solarwinds/Orion API was atrocious, so he ended up running with that. 

I'm running Jenkins on a dedicated build server, but it can be run in docker. As far as building a docker from within a docker [insert meme here], I am not sure about that so I am sticking with a server install. I started off by creating a local git repo in the directory where the dockerfile is, having followed the instructions from the PM2 official docker. From inside the directory, I ran "git init" to create the repo, and "git add -A" to add all files in the project folder, then populated the repository URL into the Jenkins project configuration. Any time I want to update the repo, I run "git add (either filename or -A)" and then "git commit -m "comment"" for example. 

I set up a build trigger, and added an alias to my .bashrc file but you should probably use bash_aliases

alias buildy='curl -u user:password;echo "Okay. Fine."'

I have it delete my workspace before starting, as a lazy way of cleaning up files from the last build

Configured the docker build command (it uses the dockerfile in the workspace directory)

Extract the docker image into a .tar file

And clean up the docker image from the local repository

Perform post-build actions

The exec command is:
docker load -i /mnt/user/appdata/darkstats/darkstats.0.$BUILD_NUMBER.tar && docker container rm -f DarkStats ; docker container create --name DarkStats darkstats:0.$BUILD_NUMBER && docker container start DarkStats && for image in $(docker images -q darkstats); do docker image rm  "$image"; done; rm /mnt/user/appdata/darkstats/*.tar

1. Loads the tar file into the local docker repository on my remote docker host
2. Removes the old container
3. Creates a new container with the new image
4. Starts the container
5. Deletes any old copies of the image (the current image will not be removed because it's running)
6. Clean up the tar file

To do the docker builds I had to enable the API on my docker host, and configure it at: Jenkins > Configure System > Docker Builder

I also had to configure the SSH server connection for artifact publishing and remote exec

Running the build, the docker image gets created successfully

And the post-actions complete (with an expected error)

The Docker container is replaced and deployed

The code is running, reporting metrics back to PM2 Keymetrics

The logs are visible as well

As seen in part 1, the data is visible in Graphite.

Well, that about wraps things up. With this workflow, it's pretty straightforward to get almost any data source into a time-series metric database for future analysis. This information can also be pulled into Grafana for even more interesting and informative dashboards. For example, I have this Unraid System Dashboard setup to monitor my Docker host. Thanks for reading!

My first DevOps Journey - Part 1: Node-RED, JSONata, StatsD, and Graphite

When I first started using Node-RED, a visual programming tool from IBM, in 2016, I had only been using it to experiment with microcontrollers and automation. As I started exploring other open source technologies, I found myself creating actually *functional* flows and wanting to do something more, so I figured I'd give DevOps a try.

In this project I use Node-RED, with JSONata expressions, to prototype a process to extract time-series metrics data from multiple sources and send it over to StatsD/Graphite for viewing. In the second part of this project I have created a DevOps pipeline to turn the prototype into a production quality Node.js application.

For part 1, I will go over the original pieces of this project and summarize how they fit together:
Domoticz, an open source home automation system
MQTT, a lightweight pub/sub message broker
Node-RED, a visual programming tool from IBM
JSONata, a query and transformation language for JSON data
StatsD - stats aggregation daemon created at Etsy
Graphite - time-series metrics graphing tool created at Orbitz

I am running all of these components as Docker containers on my UnRaid server (which I love for the built in update mechanism that seems to be superior to Portainer and Cockpit) :

Let us begin

1. Data sources


One of my data sources is Domoticz, which uses MQTT as a message broker. I have it connected to the DarkSky API. Update: sign-ups are no longer available, thanks Apple :(

I have a NodeMCU dev board with an HTU21D sensor running the ESPeasy firmware

Configured to connect to MQTT and publish to the domoticz/in topic

I also have a Shelly 1PM module with the Tasmota firmware flashed to it, but I had to deviate a little bit from this guide (if I recall correctly, I had to manually set it as a Shelly 2 device, then manually set the template settings from the guide). 

Tasmota has built in Domoticz parameters

Tasmota also has custom MQTT topic configuration which is what I use as the data source for this device, because it includes MCU temperature whereas the Domoticz topic only includes power usage


I have some other data sources that I interface with directly. I'm using the AirNow API which returns JSON output, and I'm scraping St. Louis County's Mold count iframe page with the HTML node (which uses CSS-Select), because I'm allergic to mold. 

2. Process


This Node-RED flow is the core piece of this project... if I did not have this tool at my disposal, I would not be interested in all of the other parts. 

Here is a breakdown of my flow:

Weather (I also have a humidifier controlled by my HTU21D sensor, which I will cover another time)
1. Subscribes to MQTT topic domoticz/out
2. Converts the JSON content to an object
3. Switches the output based on payload.dtype. One data type (lights/switches) is ignored here
4. Change nodes with JSONata expressions to convert the metrics data into StatsD format (this is where the magic happens)
5. Output to StatsD via UDP (as the gauge metric type)

Air Quality 

1. Timestamp nodes triggers this part of the flow every 15 minutes
2. HTTP Request nodes grab data either in JSON or or raw HTML format
3. A JSON node converts to an object for the AirNow data, and a HTML node uses CSS-select to grab the mold count
4. More change nodes with JSONata expressions
5. Output to StatsD via UDP

Laundry (I am also working on "your laundry is done" notifications, which I may cover another time)
1. Subscribes to Tasmotta MQTT telemetry topic
2. Switch node only outputs if it is sensor data being published (don't care about status)
3. JSON node converts it to an object
4. Change node with JSONata expression 
5. Output to StatsD via UDP


When I stumbled upon JSONata, and realized how powerful it is and easy to pick up, it's hard to describe how I felt. I am not a programmer and this just lowered my entry barrier to the programming world. I'll sum up how I get to this point:

1. I put a debug node at the end of my flow, outputting "complete msg object" instead of "msg.payload"

2. Copy the entire message object from the debug feed, by mousing over the right side of "object"

3. Paste the message object into the "test" tab of the JSONata expression editor, and click "format JSON"

4. Now it's a matter of building your expression, referring to the JSONata documentation. If I type "payload" as the expression, the result is the "payload" object

This is what one of my expressions looks like:".tmp:"&(

With StatsD you can chain metrics together with a newline (\n). The ouput looks like this (spaces sent to StatsD are converted to underscores):

"DarkSky THB.tmp:44.78|g\nDarkSky THB.hum:51|g\nDarkSky|g"

3. Results


And there we have it... the metrics were added to the dashboard by selecting values from paths starting with stats.gauges

I did end up making a lot of changes to the dashboard to better display the data. Overlaying certain data types in a second y-axis makes for some interesting correlations, such as rain/visibility

In part 2, I will go over the development/deployment phase, which includes the following components:
Node.js - a JavaScript runtime built on Chrome's V8 JavaScript engine.
Docker - a containerization platform
PM2 - a process manager for Node.js applications
Jenkins - an open source automation server