How to connect a Node MQTT app to AWS IoT Core without the SDK

AWS IoT Core is a powerful tool that allows you to easily setup & manage billions of devices, and trillions of events. AWS provide a wizard which guides you through configuring a device, and installing either the JavaScript, Java or Python SDK onto it.

As easy as this setup is, I wasn't able to find any resources with guidance on how to send events to IoT Core when using an SDK isn't an option. I experienced this recently when trying to setup an RFID reader. The reader comes with a set of pre-configured APIs that allow it to send events over HTTP Stream, MQTT, Kafka or Webhook. All of these APIs are great, but I couldn't get onto the device to install the IoT Core SDK, and just pointing the device at the IoT Core endpoint, didn't work.

To simplify the setup, I removed the hardware device, and decided to just create a "dumb" MQTT publisher in Node.

Pre-requisites

To follow this project, you'll need to have the latest version of Node.js installed, and you'll need an AWS account.

The sample project structure is very simple. Don't worry about the certificates folder yet, we'll get them from AWS:

app-name
  ├── index.js
  ├── package.json
  └── certs
      ├── device.cert.pem
      ├── device-key.private.key
      └── aws-root.cert.pem

Creating the Node MQTT App

This article won't focus too much on the MQTT app, as it's intentionally very simple. The focus here is getting the communication with AWS IoT Core working.

To start the project, create a new folder, and run npm init to create a basic NPM project. We can now install the MQTT dependency using npm i mqtt.

Create an index.js file, and add the following code:

// index.js

const fs = require("fs");
const mqtt = require('mqtt'); 

const ENDPOINT = "mqtts://<IoT Core URL>";
const TOPIC = "<Topic Name>" 

var rootCert = fs.readFileSync("./certs/amazon-root.cert.pem");
var certFile = fs.readFileSync("./certs/device-cert.cert.pem");
var keyFile = fs.readFileSync("./certs/device-key.private.key");

const client  = mqtt.connect(ENDPOINT, {
  clientId: "<Client ID>",
  keepalive: 60,
  reconnectPeriod: 0,
  clean: false,
  reschedulePings: true,
  protocolId: 'MQTT',
  protocolVersion: 4,
  connectTimeout: 30 * 1000,
  ca: [ rootCert ],
  cert: certFile,
  key: keyFile
}); 

client.on("connect", () => {   
  console.log("client connected"); 
  let count = 0;

  setInterval(() => {
    console.log("Publishing...");
    
    client.publish(TOPIC, JSON.stringify({
      data: count,
      timestamp: Date.now()
    }), {qos: 1, retain: true, contentType: "string"}, (PacketCallback, err) => { 
      if(err) { 
        console.log("Error: ", err) 
        return;
      } 

      console.log("Published");
    }) 

    count++;
  }, 1000);
}) 

client.on("error", (err) => { 
    console.log("Error: " + err);

    if(err.code == "ENOTFOUND") { 
        console.log("Network error, make sure you have an active internet connection") 
    } 
}) 

client.on("close", () => { 
    console.log("Connection closed by client") 
}) 

client.on("disconnect", () => { 
  console.log("Broker closed the connection") 
}) 

client.on("reconnect", () => { 
    console.log("Client reconnecting...") 
}) 

client.on("offline", () => { 
    console.log("Client is currently offline") 
})

There are 3 fields in the above code that we'll need to update:

<IoT Core URL> - We'll populate this field shortly, for now leave the placeholder
<Topic Name> - This is the name of the topic we'd like to send or receive events from. An example topic for this app might be poc/mqtt/node-app
<Client ID> - This is the ID we're going to identify the publisher we're creating by, an example might be my-laptop

And that's it, we should now have a semi-complete MQTT publisher.

Creating a Thing in IoT Core

In order for IoT Core to recognise our events, we need to create a Thing. A Thing is what IoT Core calls a connected device, whether it's a publisher or subscriber, if it's connected to IoT Core, it's a Thing.

To create a Thing:

Log into the AWS Console, and in the Navigation Search bar, search for IOT CORE
In the main body of the page, click Connect device
We will then be presented with a setup guide, in the lower-right corner, click Get started
In this example, we're not using an SDK to setup our IoT device, so the selection on this screen doesn't matter. To get through this screen, we will select Linux/OSX as my platform, and Node.js as my SDK. Click Next.
Give your thing a name, this should be a unique identifier. As this is just a demo, we'll name mine nyle-local. Click Next step.
We will now be presented with a generated policy, and a generated connection kit. The connection kit contains some snippets to help get started with the SDK. We aren't interested the snippets, but we are interested in the certificates IoT Core has generated for us. Download the connection kit. We'll come back to this in a moment.
Click Next step
And finally, click Done

We now have a Thing!

Setting the certificates

We now need to take the certificates from the connection kit we downloaded, and set them in our local MQTT project.

Unzip the connection package, and we should see the following:

connect_device_package
  ├── start.sh
  ├── <Thing Name>.cert.pem
  ├── <Thing Name>.public.key
  └── <Thing Name>.private.key

Open the MQTT project folder side-by-side so you can drag-and-drop the files over. Create a certs folder in the project root, and move the following files over:

Drag <Thing Name>.cert.pem and rename it to device-cert.cert.pem
Drag the <Thing Name>.private.key and rename it to device-key.private.key
Although we won't use it in our MQTT app, it's good to keep all of the certificates together, so drag <Thing Name>.private.key over, but there's no need to rename it

There's one more certificate we need. If we try to run the MQTT app without the root certificate, there's a good chance we'll receive an SSL error of Unable to get Local Issuer Certificate. This is because there's a gap in the certificate chain between the ones we have just downloaded, and the root trusted certificate. To complete this chain, we need Amazon's root certificate.

We can get the AWS root certificate from the Root Certificate Repository. This should look similar to:

In our MQTT project, create a New File called amazon-root.cert.pem, then copy & paste the contents of the certificate into the file.

We should now have all three pieces we need to connect to AWS securely.

Setting the URL

The final variable we need to update in our MQTT Node app is the <Iot Core URL>. We can also get this from the connection kit we downloaded. Open the start.sh file, and find the --host-name parameter passed into the run script at the bottom of the file. This should similar to a1aaaaaa11aa11-ats.iot.eu-west-1.amazonaws.com. Copy this URL, and replace the variable in the Node MQTT script.

Updating the Policy

The last thing we need to do is ensure that the Policy AWS created for us gives us the correct permissions. By default, the policy that IoT Core generates is focussed on giving access via the SDK.

To find the policy, on the left-hand navigation, go to Security > Policies, and we should see an item named <Thing Name>-Policy. In the top-right, click Edit active version. Remove or edit the policies so that the only permissions we have are:

Allow - iot:Publish - *
Allow - iot:Connect - *
Allow - iot:RetainPublish - *

This policy configuration is for demo purposes only. The * character should not be used in production.

Be sure to check the box for Set the edited version as the active version for this policy, otherewise the changes won't immediately take effect. Click Save as new version. The updated policy should be identical to the one bewlow:

AWS IoT Core Thing Policy

Running the MQTT Client

We can now run our MQTT client. In the package.json file, add a start script:

{
  "scripts": {
    "start": "node ./index.js"
  },
}

Run the script:

npm start

We should now see a feed of events being published to IoT Core in the Terminal.

To see them events appear in Iot Core, we can make use of the MQTT test client, which can be found in the left-hand navigation

Enter the topic that the MQTT Node app is publishing to, and press Subscribe. If all is well, we should see a feed of events appearing in the feed. The Node app is now connected to IoT Core!

Troubleshooting

Script is not starting

Ensure the mqtt npm package has been installed properly, and that all of the <Variables> have been updated in the MQTT Node app code.

The MQTT Node app isn't connecting to IoT Core

If the MQTT app is struggling to connect to IoT Core, or it is connecting & disconnecting, there are a number of AWS reserved topics that can assist in troubleshooting.

Two incredibly helpful topics we can subscribe to in the MQTT test client are:

$aws/events/presence/connected/<Client ID> - Which will publish an event when the device connects to IoT Core
$aws/events/presence/disconnected/<Client ID> - Which will publish an event when the device disconnects from IoT Core. This event contains reason codes which allow us to pinpoint issues.