Zigbee Device Monitoring Flow

april.brandt · November 16, 2022, 1:56am

Ok some quick instructions, so pay attention, kids.

In the check device health and global state, the “on start” tab is where you list your devices that you want checked outside of the default. Meaning, all zigbee devices will be monitored and checked every 2 hours according to the default set at the top. I’ve left mine in there. (which you will ultimately want to remove or comment out). The numbers are expressed in seconds. You can vary the length of time before it times out. REMEMBER this is expressed in seconds. Don’t edit anything that says DO NOT EDIT Unless you’re taking responsibility for your own code. This is proven to work. We used this early on in CORE development. It’s not clean. It’s not been improved, but CORE handles it like a champ.

Here is the first part.

[{"id":"1090fa3ec2993b28","type":"mqtt in","z":"20ad3b97e56645fd","d":true,"name":"","topic":"zigbee2mqtt/#","qos":"2","datatype":"auto","broker":"3f792c51bc3d4765","nl":false,"rap":true,"rh":0,"inputs":0,"x":190,"y":1100,"wires":[["29fd1b5b9e9f9a46","4d8933ef6de47276"]]},{"id":"bc0a4ca3ff6dbd3c","type":"comment","z":"20ad3b97e56645fd","name":"--- DEVICE HEALTH and GLOBAL STATE ---","info":"","x":250,"y":1000,"wires":[]},{"id":"29fd1b5b9e9f9a46","type":"function","z":"20ad3b97e56645fd","name":"Check Device Health & Save Global State","func":"// -- DO NOT EDIT BELOW THIS LINE --\n// Actual code\nconst defaultTimeout = context.get(\"defaultTimeout\");\nconst deviceOverrides = context.get(\"deviceOverrides\");\n\nif(msg.topic.startsWith(\"zigbee2mqtt/bridge/\")) {\n    // These messages we don't need, in general, but renames are important\n    if(msg.topic == \"zigbee2mqtt/bridge/response/device/rename\") {\n        node.log(msg);\n        const payloadObj = JSON.parse(msg.payload);\n        flow.set(\"dh*_\" + encodeURI(payloadObj.data.to.replace(/\\s/g, '_')), flow.get(\"dh*_\" + encodeURI(payloadObj.data.from.replace(/\\s/g, '_'))));\n        flow.set(\"dh*_\" + encodeURI(payloadObj.data.from.replace(/\\s/g, '_')), null);\n        flow.set(\"dh*_\" + encodeURI(payloadObj.data.from), null);\n    }\n    return null;\n} else {\n    if(msg.topic == \"PING\" || msg.topic == \"STATUS\") {\n        // Cleanup code, only used temporarily\n        // for(const savedDeviceNameEncoded of global.keys()) {\n        //     if(savedDeviceNameEncoded.includes('%20')) {\n        //         global.set(savedDeviceNameEncoded, undefined);\n        //     }\n        // }\n        let deviceStatus = {};\n        let deviceOffline = {};\n        let allOnline = true;\n        const now = new Date().getTime();\n        // const now = new Date().getTime();\n        msg.now = now;\n        const lastSeenTime = new Date(null);\n        for(const deviceNameEncoded of flow.keys()) {\n            if(deviceNameEncoded.startsWith(\"dh*_\")) {\n                if(deviceNameEncoded.includes(\"%20\")) {\n                    // This is an old and incorrect property\n                    flow.set(deviceNameEncoded, undefined);\n                } else {\n                    const lastSeen = flow.get(deviceNameEncoded);\n                    if(lastSeen !== null) {\n                        const deviceName = decodeURI(deviceNameEncoded.substring(4).replace(/_/g, ' '));\n                        let timeout = defaultTimeout;\n                        if(deviceOverrides.hasOwnProperty(deviceName)) timeout = deviceOverrides[deviceName];\n                        const timeLapsedNum = Math.round((now - lastSeen) / 1000)\n                        msg.timeLapsedNum = timeLapsedNum;\n                        lastSeenTime.setSeconds(timeLapsedNum);\n                        const timeLapsedStr = lastSeenTime.toISOString().substr(11, 8);\n                        if(timeout == -1) {\n                            deviceStatus[deviceName] = \"SKIPPED: \" + timeLapsedNum;\n                        } else if(timeLapsedNum < timeout) {\n                            deviceStatus[deviceName] = \"ONLINE: \" + timeLapsedNum + \", \" + lastSeen;\n                        } else {\n                            deviceStatus[deviceName] = \"OFFLINE: \" + timeLapsedNum + \", \" + lastSeen;\n                            // Clean up offline devices\n                            global.set(\"zigbee_motion_\" + encodeURI(deviceName.replace(/\\s/g, '_')), undefined);\n                            global.set(\"zigbee_contact_\" + encodeURI(deviceName.replace(/\\s/g, '_')), undefined);\n                            global.set(\"zigbee_\" + encodeURI(deviceName.replace(/\\s/g, '_')), undefined);\n                            allOnline = false;\n                            deviceOffline[deviceName] = timeLapsedNum;\n                        }\n                    }\n                }\n            }\n        }\n        if(allOnline) {\n            node.status({fill:\"green\",shape:\"dot\",text:\"online\"});\n        } else {\n            node.status({fill:\"red\",shape:\"ring\",text:\"offline\"});\n        }\n        msg.payload = deviceStatus;\n        if(Object.keys(deviceOffline).length > 0) {\n            msg.deviceOffline = deviceOffline;\n        } else {\n            msg.deviceOffline = null;\n        }\n        if(msg.topic == \"PING\") {\n            if(msg.deviceOffline === null) {\n                return null;\n            } else {\n                msg.topic = \"DeviceOffline\";\n            }\n        }\n        return msg;\n    } else {\n        const deviceNameRegExp = /zigbee2mqtt\\/([^\\/]*)/;\n        const deviceNameExtendedRegExp = /zigbee2mqtt\\/([^\\/]*)\\/([^\\/]*)/;\n        let deviceName = deviceNameRegExp.exec(msg.topic);\n        let deviceNameExtended = deviceNameExtendedRegExp.exec(msg.topic);\n        if(deviceName !== null && deviceName.length == 2 && (deviceNameExtended === null || deviceNameExtended.length < 3)) {\n            const deviceNameEncoded = encodeURI(deviceName[1].replace(/\\s/g, '_'));\n            if(msg.payload != '') {\n                let payload = JSON.parse(msg.payload);\n                if(payload.hasOwnProperty(\"last_seen\")) {\n                    flow.set(\"dh*_\" + deviceNameEncoded, payload.last_seen);\n                }\n                // Save the device state to global variables\n                if(payload.hasOwnProperty(\"occupancy\")) {\n                    global.set(\"zigbee_motion_\" + deviceNameEncoded, payload.occupancy);\n                    // Also make a combined state available\n                    if(payload.occupancy == true) {\n                        global.set(\"zigbee_any_motion\", true);\n                    } else {\n                        let motion = false;\n                        for(const savedDeviceNameEncoded of global.keys()) {\n                            if(savedDeviceNameEncoded.startsWith(\"zigbee_motion_\") && global.get(savedDeviceNameEncoded) == true) {\n                                motion = true;\n                                break;\n                            }\n                        }\n                        global.set(\"zigbee_any_motion\", motion);\n                    }\n                }\n                if(payload.hasOwnProperty(\"contact\")) {\n                    global.set(\"zigbee_contact_\" + deviceNameEncoded, payload.contact);\n                }\n                global.set(\"zigbee_\" + deviceNameEncoded, payload);\n            }\n        }\n    }\n}\nreturn null;","outputs":1,"noerr":0,"initialize":"// Code added here will be run once\n// whenever the node is started.\n// Settings\n// IMPORTANT: Device NAMES can't contain a forward slash \"/\"!!!\nconst defaultTimeout = 2 * 3600;     // 4 hours = 4 * 3600 seconds\nconst deviceOverrides = {\n    \"EXAMPLE Repeater - Balcony (IKEA)\": -1,    // -1 = Do not check for Device Health\n    \"EXAMPLE SillySensor\": 8 * 60 * 60,         // Checks in once every 8 hours expressed in seconds\n    \"BasementSpareLamp\": 4 * 3600, \n    \"MasterBedroomCloset1\": 4 * 3600,\n    \"MasterBedroomCloset2\": 4 * 3660,\n    \"KitchenCabinetsW\": 4 * 3600,\n    \"Doorbell\": 8 * 60 * 60,\n    \"RadioRGBW\": 4 * 3600,\n    \"SPAREBedroomLampL\": 4 * 3600,\n    \"BasementLivingRoomLampWest\": 4 * 3600,\n    // \"Button - Study Alice (Aqara)\": 4 * 3600,\n    // \"Balcony - Contact Door\": 4 * 3600,\n    // \"Study - GU10 Gledopto\": 24 * 3600,\n    // \"Livingroom - Yellow Lamp\": 24*3600,\n    // \"Study - Bulb (IKEA 1 color)\": -1,\n    // \"Study - Alice Lamp (IKEA)\": -1,\n    // \"AqaraCubeUnassigned2\": -1,\n    // \"T&H - Study (Aqara T1)\": -1,\n    // \"Livingroom - Coffee Machine Water Leak\": -1,\n    // \"Kitchen - Main Gas Valve\": 12*3600,\n    // \"Kitchen - Sink Water Sensor\": 4*3600,\n    // \"Kitchen - Contact Door\": 4*3600,\n    // \"Corridor - Temp & Humidity\": 24*3600,\n    // \"Entrance - Motion\": -1\n    // \"Study - iHorn Motion\": -1,\n}\n// -- DO NOT EDIT BELOW THIS LINE --\n// Actual code\ncontext.set(\"defaultTimeout\", defaultTimeout);\ncontext.set(\"deviceOverrides\", deviceOverrides);\nlet deviceStatus = {};\nlet deviceOffline = {};\nlet allOnline = true;\n// const now = new Date().getTime();\nconst now = new Date().getTime();\nfor(const deviceNameEncoded of flow.keys()) {\n    if(deviceNameEncoded.startsWith(\"dh*_\")) {\n        if(deviceNameEncoded.includes(\"%20\")) {\n            // This is an old and incorrect property\n            flow.set(deviceNameEncoded, undefined);\n        } else {\n            const lastSeen = flow.get(deviceNameEncoded);\n            if(lastSeen !== null) {\n                const deviceName = decodeURI(deviceNameEncoded.substring(4).replace(/_/g, ' '));\n                let timeout = defaultTimeout;\n                if(deviceOverrides.hasOwnProperty(deviceName)) timeout = deviceOverrides[deviceName];\n                if(timeout == -1) {\n                    deviceStatus[deviceName] = \"SKIPPED\";\n                } else if((now - lastSeen) < (timeout * 1000)) {\n                    deviceStatus[deviceName] = \"ONLINE\";\n                } else {\n                    deviceStatus[deviceName] = \"OFFLINE\";\n                    allOnline = false;\n                    deviceOffline[deviceName] = Math.round((now - lastSeen) / 1000);\n                }\n            }\n        }\n    }\n}\nif(allOnline) {\n    node.status({fill:\"green\",shape:\"dot\",text:\"online\"});\n} else {\n    node.status({fill:\"red\",shape:\"ring\",text:\"offline\"});\n}","finalize":"","libs":[],"x":520,"y":1100,"wires":[["ea547aeecca0720e"]]},{"id":"87626bcaf5cfe2bc","type":"debug","z":"20ad3b97e56645fd","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":970,"y":1180,"wires":[]},{"id":"8ede87446457a244","type":"inject","z":"20ad3b97e56645fd","d":true,"name":"Check Health Ping","props":[{"p":"topic","vt":"str"},{"p":"payload"}],"repeat":"7200","crontab":"","once":true,"onceDelay":"10","topic":"PING","payload":"PING","payloadType":"str","x":220,"y":1220,"wires":[["29fd1b5b9e9f9a46"]]},{"id":"dea944b0f42a56a7","type":"inject","z":"20ad3b97e56645fd","d":true,"name":"Get Health Status","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":"10","topic":"STATUS","x":200,"y":1180,"wires":[["29fd1b5b9e9f9a46"]]},{"id":"ea547aeecca0720e","type":"switch","z":"20ad3b97e56645fd","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"DeviceOffline","vt":"str"},{"t":"neq","v":"DeviceOffline","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":790,"y":1100,"wires":[["311bab92c731f793"],["87626bcaf5cfe2bc","6bd665e4060ea18d"]]},{"id":"b6f2a3e0c5dcff4c","type":"link out","z":"20ad3b97e56645fd","name":"->Pushover","links":["d9e4c8d5d6471847"],"x":1535,"y":1100,"wires":[]},{"id":"afab0efa86e28552","type":"function","z":"20ad3b97e56645fd","name":"Format Device Offline Message","func":"msg.topic = \"Device Offline!\";\nlet payload = \"Zigbee Device Health\\n\";\npayload +=    \"--------------------\\n\";\nconst lastSeenTime = new Date(null);\nfor(const [offlineDevice, lastSeen] of Object.entries(msg.deviceOffline)) {\n    lastSeenTime.setSeconds(lastSeen);\n    payload += offlineDevice + \": \" + lastSeenTime.toISOString().substr(11, 8) + '\\n';\n}\nmsg.payload = payload;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1330,"y":1100,"wires":[["87626bcaf5cfe2bc","b6f2a3e0c5dcff4c","6bd665e4060ea18d"]]},{"id":"4d8933ef6de47276","type":"debug","z":"20ad3b97e56645fd","name":"MQTT debug","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":450,"y":1060,"wires":[]},{"id":"6bd665e4060ea18d","type":"debug","z":"20ad3b97e56645fd","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":990,"y":1220,"wires":[]},{"id":"311bab92c731f793","type":"throttle","z":"20ad3b97e56645fd","name":"Limit Offline Messages","throttleType":"time","timeLimit":"30","timeLimitType":"minutes","countLimit":0,"blockSize":0,"locked":false,"x":1020,"y":1100,"wires":[["87626bcaf5cfe2bc","afab0efa86e28552","6bd665e4060ea18d"]]},{"id":"3f792c51bc3d4765","type":"mqtt-broker","name":"CORE Clean1","broker":"10.10.2.1","port":"1883","clientid":"","autoConnect":true,"usetls":false,"protocolVersion":"5","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"sessionExpiry":""}]

I’ve had the first nodes disabled, so you will want to enable them once you put your information in. Enter your core server information into your mqtt node. The inject nodes will allow you to check at any time. This outputs to the debug window.

I pushed the output to a pushover node. Easy enough.

Challenge accepted? @Mike @relaxton
GO!

relaxton · November 16, 2022, 3:59am

OK, so I chucked it in and it seems to be working. Do you use another flow for low battery warnings? I do have a few of devices offline that I need to fix, remove or add to the ignore list but like it so far.

april.brandt · November 16, 2022, 4:00am

I pushed it into pushover so that I could get the notification in a format I could walk around and fix the problems.

relaxton · November 16, 2022, 4:04am

Yes, I did the same, while I did not want to use Pushover for my notifications, I set it up quickly just to get something working. I also linked it in with my emergency mode (e.g. leaks detected) so that not only do all coloured lights go red, but I get the text of the specific emergency sent to my phone.

relaxton · November 16, 2022, 4:21am

Actually, most of the notifications were false positives. Is there any reason why this code is not using the availability data that zigbee2mqtt is maintaining? It looks like it is just doing a simple “elapsed time since last seen check” which is possibly a bit naive?

Am I missing something?

markus · November 16, 2022, 4:58am

At the time this was created that was not very useful. At this point in time that may be a valid appooach though.

This has worked well for me, once the elapsed time for some devices has been modified it has been very good. I’m sure it could use a lot of tweaking by now, but it still works for me. I’ve not needed it for a long time but enabled it now, looks to work as it did, if I see something which needs changing I’ll update it.

relaxton · November 16, 2022, 5:36am

Sweet. I will look at whether I could simplify by using the z2m availability and report back. I will also look at integrating battery reporting into the feed.

Definitely a good start, even if only because it inspired me to finally get my notification system working.

bobbles · December 6, 2022, 6:36pm

Hi @april.brandt
I’ve been running this OK but yesterday my hub picked up and paired a ‘Hive’ outlet. I haven’t tried to pair anything for days.
I haven’t got one so can only assume my neighbour has bought one and has been playing with it.
The only reason I know it has paired with my hub is because this flow picked it as an OFFLINE device.
I have deleted it OK.

This brings up 2 questions.

It keeps showing up as being offline in this flow. Is there a way to reset the stored list without deleting the flow and starting again.
When I looked at z2m it was in a state of permitting pairing. I thought I had disabled it but it should disable after 4 minutes anyway. Is there a bug in here somewhere?

This is all minor stuff I know but as we are in beta I thought I would let you guys know.

I can also confirm that my CC and all flows and devices are running with absolutely no issues. Not one reboot needed .

Keep up the good work guys and gals and I look forward to more integrations/innovations with the platform.

EDIT: Thinking about this I did upgrade the zigbee z2m last week. ( oll-z2m --update). Could this have left pairing ‘open’?

bobbles · December 6, 2022, 7:50pm

Found how to remove it from the list.

april.brandt · December 6, 2022, 7:53pm

Check settings that auto join is not checked in z2m. You probably did that, but others might need this info

Mike · January 26, 2023, 9:18am

So I missed this when it first was released, and only getting to it now.

I’m getting this error, which is probably because I’m in a different (earlier) time zone. It’s usually the problem with these types of errors.

Edit: I’m also getting these errors, in amongst the normal correct outputs. Looking through the errors, it appears the be the ----> ’ <------in Mike’s…would that be correct?

april.brandt · January 27, 2023, 2:25am

Line 48 is most likely where it’s breaking. And the change is to set now to a timezone. You’d have to see what data is coming in for now so it is not before the incoming data.

Yes, this is creating a global based on this name and a single quote is not allowed in globals. You would need to change the name of the device, or you would need to change the code to remove that. But doing that will make it very hard on yourself. The easy route is just changing the name of the device.

Mike · January 27, 2023, 2:31am

Thanks, for the second error, I’ll change the name.

For the first, this is what’s coming in.

"{"level":"info","message":"MQTT publish: topic 'zigbee2mqtt/PS - Garage Door', payload '{\"energy\":6.01,\"last_seen\":\"2023-01-27T02:29:34.997Z\",\"linkquality\":123,\"power\":0,\"state\":\"ON\"}'"}"

april.brandt · January 27, 2023, 2:34am

Are you running Z2M on CORE? Or on a separate machine?

Mike · January 27, 2023, 2:37am

Core…

and the timezone is set correctly, just checked.

april.brandt · January 27, 2023, 2:41am

Something’s up in with the code. Stand by.

april.brandt · January 27, 2023, 2:51am

line 37 try replacing that entire line with this:
but check that that line has the const lastSeen in case your line numbers are different.

 const lastSeen = flow.get(deviceNameEncoded) | 0;

Mike · January 27, 2023, 2:54am

That kinda worked, but now all my devices are reported as offline.

april.brandt · January 27, 2023, 2:57am

Last seen has not been saved as a global yet. If you’ve made other changes then it could be something different, but wait for last seen to be saved again and then see. Patience young Costanza.

Mike · January 27, 2023, 3:05am

Something’s not quite right, since some of these devices have checked in since the change.

This is the payload, which also looks a little strange??