Suddenly a bunch of zigbee devices went offline

Mike · April 17, 2023, 10:03pm

Suddenly yesterday all my devices went offline too. I was heading out, and all I could do quickly was restart z2m, which worked. What logs do you need @april.brandt ?

It was offline for 2 hours before I discovered (walking out the door, hmmm, why are automations not working???) and restarted z2m…and then when I came home, all devices had reconnected without re-pairing. Nice.

april.brandt · April 17, 2023, 10:05pm

The commands are listed in my post above

roberto · April 18, 2023, 8:30am

my silence is due to the fact that I haven’t had any problems for almost a week.
The only operations carried out were:
. disconnect a zigbee repeater (Aeotec Zi Range Extender)
. disconnect a led hue bulb that is not mine, i.e. belonging to some neighbour, which was connected by mistake.

Now I’ll have to do the counter-test: reconnect the router.
I suppose my logs are not relevant at this point.

april.brandt · April 18, 2023, 11:24pm

Harvest the logs if it happens.

bobbles · April 26, 2023, 3:57pm

Just got back from my hols and picked up with this thread again.
In case the info is still needed, here it is in my case.

epw · April 26, 2023, 4:08pm

Wow that’s old firmware on your coordinator! 76 devices might qualify it as large, but it’s not excessively large. There were improvements that Koen made in 2022 that should help with larger meshes. The Feb '22 build was solid, and I’ve been very happy with the Dec '22 build. Both are mainline releases and can be obtained on his git - here’s a link for the 20221226 build: Z-Stack-firmware/coordinator/Z-Stack_3.x.0/bin at master · Koenkk/Z-Stack-firmware · GitHub

bobbles · April 26, 2023, 4:13pm

If memory serves me right, I’ll have to do a search, I tried to update my co-ordinator and the update failed miserably.
I did post for advice in that thread but got no responses.
I had bought another co-ordinator so that I could update that without crashing things if it went wrong.
Just gave up in the end.

Here is the thread.

sburke781 · April 27, 2023, 12:43pm

Feel like I may have something to add to this topic after recent experience… But there are plenty of possible root causes, including NR rules, changes to the placement of my Core / Zigbee radio or updates to the Z2m container. Basically I have been seeing “weirdness” in responses to button presses for both my IKEA on/off switches and Hue Tap Dial devices paired to Core. I’ll see what I can work out and report back.

april.brandt · April 28, 2023, 2:48pm

update your coordinator with the TI updater. I just updated mine when my zigbee crashed two weeks ago. follow the instructions with the firmware. Be careful that you’re downloading the correct one. I have 3 sticks so far that i need to fix with a j-tag. Just no time at the moment to do so.

sburke781 · April 28, 2023, 11:44pm

Where are the instructions for updating the Zigbee firmware? Is there a post somewhere I should refer to?

Thanks,
Simon

april.brandt · April 28, 2023, 11:46pm

just google your model of coordinator and the instructions will be with it. The most important is to make sure you know your model.

sburke781 · April 29, 2023, 12:18am

Good tip. And don’t trust that the model on Amazon matches the actual model you receive. I originally purchased the Sonoff z3 Dongle-Plus, i.e. the “P” version, when I originally got my Core, then bought what I thought was a second “P” version in case I wanted another one for my second Core. That does not appear to have been the case, the spare one in my desk drawer is in fact the “E” version. Not a major problem, but would have been nice for them to be the same.

sburke781 · April 29, 2023, 2:36am

I’m like @roberto , the Zigbee stick is not recognised on my Windows laptop, and the Silabs driver download page that I have seen linked here and on other sites doesn’t actually list anything to download. This does feel a little bit complicated…

I may not technically need to upgrade the firmware, so might just plug it back into Core and look at some other things like NR to see where some of my issues might be.

april.brandt · April 29, 2023, 3:11am

which one are you updating? the E or the P?

sburke781 · April 29, 2023, 3:14am

I was attempting to upgrade the P one, that is the one I currently use on my Core. The E is the spare I have in a drawer.

april.brandt · April 29, 2023, 3:27am

you would need to download the serial drivers for it. I’ll look into it and let you know. I’ve been flashing so many boards i can’t remember off of the top of my head.

sburke781 · April 29, 2023, 3:31am

I might have made another step forward…

In Chrome I was not seeing anything on the SiLabs page, but I just opened it in FireFox and can now see them listed.

sburke781 · April 29, 2023, 4:23am

With a few bumps along the way I got there in the end

I don’t know if it’s related, but I think the boot time of the Core seemed a lot quicker after I had finished the firmware update…

roberto · May 10, 2023, 3:28pm

It happende again after few weeks. I tried to download the logs with filezilla. I cannot connect to core.
any advice?

EDIT: with oh-la the port is 22 instead of 21

roberto · May 10, 2023, 5:45pm

@markus I stoppd z2m, reset the dongle, and restarted z2m from GUI (I tried by cmd line getting the same result). Z2m does’nt restart. It recycles every minute or so giving the same log:

|2023-05-10 19:40:45 CEST|info|WARN: Config override! Config file "/home/oh-la/data/z2m/oll-z2m.conf" present and containing more than comments, possibly overriding defaults. Remove the file to disable overrides.|
|2023-05-10 19:40:45 CEST|info|---- CONFIG OVERRIDES START ----|
|2023-05-10 19:40:45 CEST|info|CONTAINER_REPO="koenkk/zigbee2mqtt:latest"|
|2023-05-10 19:40:45 CEST|info|----- CONFIG OVERRIDES END -----|
|2023-05-10 19:40:45 CEST|info|INFO: Running command: prepare|
|2023-05-10 19:40:45 CEST|info|INFO: Executing the first part of prepare as root|
|2023-05-10 19:40:45 CEST|info|INFO: Retrieving/updating the device images|
|2023-05-10 19:40:45 CEST|info|INFO: Skipped image updates since last update was less than 24 hours ago...|
|2023-05-10 19:40:45 CEST|info|INFO: Set permissions|
|2023-05-10 19:40:46 CEST|info|INFO: Done!|
|2023-05-10 19:40:46 CEST|info|INFO: We are root, becoming the oh-la user instead...|
|2023-05-10 19:40:46 CEST|info|INFO: Running "oll-z2m --prepare" as user oh-la|
|2023-05-10 19:40:46 CEST|info|(to oh-la) root on none|
|2023-05-10 19:40:46 CEST|info|pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)|
|2023-05-10 19:40:46 CEST|info|WARN: Config override! Config file "/home/oh-la/data/z2m/oll-z2m.conf" present and containing more than comments, possibly overriding defaults. Remove the file to disable overrides.|
|2023-05-10 19:40:46 CEST|info|---- CONFIG OVERRIDES START ----|
|2023-05-10 19:40:46 CEST|info|CONTAINER_REPO="koenkk/zigbee2mqtt:latest"|
|2023-05-10 19:40:46 CEST|info|----- CONFIG OVERRIDES END -----|
|2023-05-10 19:40:46 CEST|info|INFO: Running command: prepare|
|2023-05-10 19:40:46 CEST|info|INFO: Running as required user oh-la|
|2023-05-10 19:40:46 CEST|info|INFO: Finalizing preparing for the Zigbee2MQTT container as the podman user...|
|2023-05-10 19:40:47 CEST|info|WARN: Config override! Config file "/home/oh-la/data/z2m/oll-z2m.conf" present and containing more than comments, possibly overriding defaults. Remove the file to disable overrides.|
|2023-05-10 19:40:47 CEST|info|---- CONFIG OVERRIDES START ----|
|2023-05-10 19:40:47 CEST|info|CONTAINER_REPO="koenkk/zigbee2mqtt:latest"|
|2023-05-10 19:40:47 CEST|info|----- CONFIG OVERRIDES END -----|
|2023-05-10 19:40:47 CEST|info|INFO: Running command: defaults|
|2023-05-10 19:40:47 CEST|info|INFO: Running as required user oh-la|
|2023-05-10 19:40:47 CEST|info|INFO: Ensuring the ~/.systemd folder exists|
|2023-05-10 19:40:47 CEST|info|Settings already present in "/home/oh-la/data/z2m/configuration.yaml" for the Zigbee2MQTT container... No changes made!|
|2023-05-10 19:40:47 CEST|info|WARN: Config override! Config file "/home/oh-la/data/z2m/oll-z2m.conf" present and containing more than comments, possibly overriding defaults. Remove the file to disable overrides.|
|2023-05-10 19:40:47 CEST|info|---- CONFIG OVERRIDES START ----|
|2023-05-10 19:40:47 CEST|info|CONTAINER_REPO="koenkk/zigbee2mqtt:latest"|
|2023-05-10 19:40:47 CEST|info|----- CONFIG OVERRIDES END -----|
|2023-05-10 19:40:47 CEST|info|INFO: Running command: backup|
|2023-05-10 19:40:47 CEST|info|INFO: Running as required user oh-la|
|2023-05-10 19:40:47 CEST|info|INFO: Creating backup of essential Z2M files in /home/oh-la/data/z2m_backup...|
|2023-05-10 19:40:47 CEST|info|INFO: Creating backup of "configuration.yaml" in "/home/oh-la/data/z2m_backup".|
|2023-05-10 19:40:47 CEST|info|INFO: File "configuration.yaml" exist and is NOT empty!|
|2023-05-10 19:40:47 CEST|info|INFO: The latest backup is already up-to-date.|
|2023-05-10 19:40:47 CEST|info|INFO: Creating backup of "coordinator_backup.json" in "/home/oh-la/data/z2m_backup".|
|2023-05-10 19:40:47 CEST|info|INFO: File "coordinator_backup.json" exist and is NOT empty!|
|2023-05-10 19:40:47 CEST|info|INFO: The latest backup is already up-to-date.|
|2023-05-10 19:40:47 CEST|info|INFO: Creating backup of "database.db" in "/home/oh-la/data/z2m_backup".|
|2023-05-10 19:40:47 CEST|info|INFO: File "database.db" exist and is NOT empty!|
|2023-05-10 19:40:47 CEST|info|INFO: The latest backup is already up-to-date.|
|2023-05-10 19:40:47 CEST|info|INFO: Done!|
|2023-05-10 19:40:47 CEST|info|WARN: Config override! Config file "/home/oh-la/data/z2m/oll-z2m.conf" present and containing more than comments, possibly overriding defaults. Remove the file to disable overrides.|
|2023-05-10 19:40:47 CEST|info|---- CONFIG OVERRIDES START ----|
|2023-05-10 19:40:47 CEST|info|CONTAINER_REPO="koenkk/zigbee2mqtt:latest"|
|2023-05-10 19:40:47 CEST|info|----- CONFIG OVERRIDES END -----|
|2023-05-10 19:40:47 CEST|info|INFO: Running command: run|
|2023-05-10 19:40:47 CEST|info|INFO: Running as required user oh-la|
|2023-05-10 19:40:48 CEST|info|INFO: Running the Zigbee2MQTT container... Should ONLY be run through a systemd service!|
|2023-05-10 19:40:48 CEST|info|INFO: Exporting all serial devices to the container|
|2023-05-10 19:40:48 CEST|info|INFO: Mapping ports: 127.0.0.1:8080:8080|
|2023-05-10 19:40:48 CEST|info|INFO: Mapping folders: /home/oh-la/data/z2m:/app/data|
|2023-05-10 19:40:48 CEST|info|INFO: Mapping folders: /etc/ssl/certs:/etc/ssl/certs:ro|
|2023-05-10 19:40:48 CEST|info|INFO: Mapping folders: /usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro|
|2023-05-10 19:40:48 CEST|info|INFO: Mapping enviroment var: NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="/usr/bin/podman filtering at log level info"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="Setting parallel job count to 13"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="/usr/bin/podman filtering at log level info"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="Setting parallel job count to 13"|
|2023-05-10 19:40:48 CEST|info|2023-05-10 19:40:48.621814437 +0200 CEST m=+0.299707341 container create f03ce9b9b5d34644fd267957174c778d5719a143c77858870f1bfc04e32f8305 (image=docker.io/koenkk/zigbee2mqtt:latest, name=z2m, PODMAN_SYSTEMD_UNIT=podman-z2m.service)|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"|
|2023-05-10 19:40:48 CEST|info|time="2023-05-10T19:40:48+02:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"|
|2023-05-10 19:40:49 CEST|info|time="2023-05-10T19:40:49+02:00" level=info msg="Got Conmon PID as 2980736"|
|2023-05-10 19:40:49 CEST|info|2023-05-10 19:40:49.068049947 +0200 CEST m=+0.745942893 container init f03ce9b9b5d34644fd267957174c778d5719a143c77858870f1bfc04e32f8305 (image=docker.io/koenkk/zigbee2mqtt:latest, name=z2m, PODMAN_SYSTEMD_UNIT=podman-z2m.service)|
|2023-05-10 19:40:49 CEST|info|2023-05-10 19:40:49.085832712 +0200 CEST m=+0.763725616 container start f03ce9b9b5d34644fd267957174c778d5719a143c77858870f1bfc04e32f8305 (image=docker.io/koenkk/zigbee2mqtt:latest, name=z2m, PODMAN_SYSTEMD_UNIT=podman-z2m.service)|
|2023-05-10 19:40:49 CEST|info|f03ce9b9b5d34644fd267957174c778d5719a143c77858870f1bfc04e32f8305|
|2023-05-10 19:40:49 CEST|info|Using '/app/data' as data directory|
|2023-05-10 19:40:58 CEST|info|Zigbee2MQTT:info 2023-05-10 19:40:58: Logging to console and directory: '/app/data/log/2023-05-10.19-40-58' filename: log.txt|
|2023-05-10 19:40:58 CEST|info|Zigbee2MQTT:info 2023-05-10 19:40:58: Starting Zigbee2MQTT version 1.30.4 (commit #b2dd21e)|
|2023-05-10 19:40:58 CEST|info|Zigbee2MQTT:info 2023-05-10 19:40:58: Starting zigbee-herdsman (0.14.111)|
|2023-05-10 19:40:59 CEST|info|Zigbee2MQTT:error 2023-05-10 19:40:59: Error while starting zigbee-herdsman|
|2023-05-10 19:40:59 CEST|info|Zigbee2MQTT:error 2023-05-10 19:40:59: Failed to start zigbee|
|2023-05-10 19:40:59 CEST|info|Zigbee2MQTT:error 2023-05-10 19:40:59: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions|
|2023-05-10 19:40:59 CEST|info|Zigbee2MQTT:error 2023-05-10 19:40:59: Exiting...|
|2023-05-10 19:40:59 CEST|info|Zigbee2MQTT:error 2023-05-10 19:40:59: Error: Error while opening serialport 'Error: Error: No such file or directory, cannot open /dev/serial/by-path/platform-xhci-hcd_2_auto-usb-0_1_1_0-port0'|
|2023-05-10 19:40:59 CEST|info|at SerialPort.<anonymous> (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:146:28)|
|2023-05-10 19:40:59 CEST|info|at SerialPort._error (/app/node_modules/@serialport/stream/dist/index.js:76:22)|
|2023-05-10 19:40:59 CEST|info|at /app/node_modules/@serialport/stream/dist/index.js:112:18|
|2023-05-10 19:40:59 CEST|info|2023-05-10 19:40:59.963952092 +0200 CEST m=+0.153266671 container died f03ce9b9b5d34644fd267957174c778d5719a143c77858870f1bfc04e32f8305 (image=docker.io/koenkk/zigbee2mqtt:latest, name=z2m)|
|2023-05-10 19:41:00 CEST|info|2023-05-10 19:41:00.130195425 +0200 CEST m=+0.319509838 container remove f03ce9b9b5d34644fd267957174c778d5719a143c77858870f1bfc04e32f8305 (image=docker.io/koenkk/zigbee2mqtt:latest, name=z2m, PODMAN_SYSTEMD_UNIT=podman-z2m.service)|
|2023-05-10 19:41:00 CEST|info|Error: no arguments are needed with --latest or --cidfile|