I woke up this morning to the CORE failing once again to gracefully recover from a network outage. Long story short I manually got the network back and unplugged the CORE, plugged it back in and now a handful of containers are refusing to start citing something wrong with symlinks. The affected containers are Z2M, InfluxDB, and Uptime-Kuma.
This is an example log for InfluxDB. Very strange error after downloading the new image. Any guidance would be greatly appreciated @markus @april.brandt
...
Timestamp Lvl Message
2025-03-20 10:04:03 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:04:03 GMT−7 info Done!
2025-03-20 10:04:03 GMT−7 info pam_unix(su-l:session): session closed for user oh-la
2025-03-20 10:05:50 GMT−7 info INFO: Running command: prepare
2025-03-20 10:05:50 GMT−7 info Executing the first part of prepare as root
2025-03-20 10:05:50 GMT−7 info WARN: NOT replacing /home/oh-la/data/influxdb/influxdb.conf since there has been manual changes!
2025-03-20 10:05:50 GMT−7 info failed adding user 'telegraf', data deleted
2025-03-20 10:05:50 GMT−7 info Set permissions
2025-03-20 10:05:50 GMT−7 info About to update telegraf.conf
2025-03-20 10:05:51 GMT−7 info INFO: Overwriting /etc/telegraf/telegraf.conf since no manual changes have been made to it!
2025-03-20 10:05:51 GMT−7 info Updated telegraf.conf
2025-03-20 10:05:51 GMT−7 info Inserting the telegraf influxdb password into the telegraf config
2025-03-20 10:05:51 GMT−7 info Done!
2025-03-20 10:05:51 GMT−7 info INFO: We are root, becoming the oh-la user instead...
2025-03-20 10:05:51 GMT−7 info INFO: Running "oll-influxdb --prepare" as user oh-la
2025-03-20 10:05:51 GMT−7 info (to oh-la) root on none
2025-03-20 10:05:51 GMT−7 info pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)
2025-03-20 10:05:51 GMT−7 info INFO: Running command: prepare
2025-03-20 10:05:51 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:05:51 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:05:51 GMT−7 info Done!
2025-03-20 10:05:51 GMT−7 info pam_unix(su-l:session): session closed for user oh-la
2025-03-20 10:06:32 GMT−7 info INFO: Running command: prepare
2025-03-20 10:06:32 GMT−7 info Executing the first part of prepare as root
2025-03-20 10:06:32 GMT−7 info WARN: NOT replacing /home/oh-la/data/influxdb/influxdb.conf since there has been manual changes!
2025-03-20 10:06:32 GMT−7 info failed adding user 'telegraf', data deleted
2025-03-20 10:06:32 GMT−7 info Set permissions
2025-03-20 10:06:32 GMT−7 info About to update telegraf.conf
2025-03-20 10:06:32 GMT−7 info INFO: Overwriting /etc/telegraf/telegraf.conf since no manual changes have been made to it!
2025-03-20 10:06:32 GMT−7 info Updated telegraf.conf
2025-03-20 10:06:32 GMT−7 info Inserting the telegraf influxdb password into the telegraf config
2025-03-20 10:06:32 GMT−7 info Done!
2025-03-20 10:06:32 GMT−7 info INFO: We are root, becoming the oh-la user instead...
2025-03-20 10:06:32 GMT−7 info INFO: Running "oll-influxdb --prepare" as user oh-la
2025-03-20 10:06:32 GMT−7 info (to oh-la) root on none
2025-03-20 10:06:32 GMT−7 info pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)
2025-03-20 10:06:33 GMT−7 info INFO: Running command: prepare
2025-03-20 10:06:33 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:06:33 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:06:33 GMT−7 info Done!
2025-03-20 10:06:33 GMT−7 info pam_unix(su-l:session): session closed for user oh-la
2025-03-20 10:07:14 GMT−7 info INFO: Running command: prepare
2025-03-20 10:07:14 GMT−7 info Executing the first part of prepare as root
2025-03-20 10:07:14 GMT−7 info WARN: NOT replacing /home/oh-la/data/influxdb/influxdb.conf since there has been manual changes!
2025-03-20 10:07:14 GMT−7 info failed adding user 'telegraf', data deleted
2025-03-20 10:07:14 GMT−7 info Set permissions
2025-03-20 10:07:14 GMT−7 info About to update telegraf.conf
2025-03-20 10:07:14 GMT−7 info INFO: Overwriting /etc/telegraf/telegraf.conf since no manual changes have been made to it!
2025-03-20 10:07:14 GMT−7 info Updated telegraf.conf
2025-03-20 10:07:14 GMT−7 info Inserting the telegraf influxdb password into the telegraf config
2025-03-20 10:07:14 GMT−7 info Done!
2025-03-20 10:07:14 GMT−7 info INFO: We are root, becoming the oh-la user instead...
2025-03-20 10:07:14 GMT−7 info INFO: Running "oll-influxdb --prepare" as user oh-la
2025-03-20 10:07:14 GMT−7 info (to oh-la) root on none
2025-03-20 10:07:14 GMT−7 info pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)
2025-03-20 10:07:14 GMT−7 info INFO: Running command: prepare
2025-03-20 10:07:14 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:07:14 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:07:15 GMT−7 info Done!
2025-03-20 10:07:15 GMT−7 info pam_unix(su-l:session): session closed for user oh-la
2025-03-20 10:07:53 GMT−7 info INFO: Running command: prepare
2025-03-20 10:07:53 GMT−7 info Executing the first part of prepare as root
2025-03-20 10:07:53 GMT−7 info WARN: NOT replacing /home/oh-la/data/influxdb/influxdb.conf since there has been manual changes!
2025-03-20 10:07:53 GMT−7 info failed adding user 'telegraf', data deleted
2025-03-20 10:07:53 GMT−7 info Set permissions
2025-03-20 10:07:54 GMT−7 info About to update telegraf.conf
2025-03-20 10:07:54 GMT−7 info INFO: Overwriting /etc/telegraf/telegraf.conf since no manual changes have been made to it!
2025-03-20 10:07:54 GMT−7 info Updated telegraf.conf
2025-03-20 10:07:54 GMT−7 info Inserting the telegraf influxdb password into the telegraf config
2025-03-20 10:07:54 GMT−7 info Done!
2025-03-20 10:07:54 GMT−7 info INFO: We are root, becoming the oh-la user instead...
2025-03-20 10:07:54 GMT−7 info INFO: Running "oll-influxdb --prepare" as user oh-la
2025-03-20 10:07:54 GMT−7 info (to oh-la) root on none
2025-03-20 10:07:54 GMT−7 info pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)
2025-03-20 10:07:54 GMT−7 info INFO: Running command: prepare
2025-03-20 10:07:54 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:07:54 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:07:54 GMT−7 info Done!
2025-03-20 10:07:54 GMT−7 info pam_unix(su-l:session): session closed for user oh-la
2025-03-20 10:31:42 GMT−7 info INFO: Running command: prepare
2025-03-20 10:31:42 GMT−7 info Executing the first part of prepare as root
2025-03-20 10:31:42 GMT−7 info WARN: NOT replacing /home/oh-la/data/influxdb/influxdb.conf since there has been manual changes!
2025-03-20 10:31:42 GMT−7 info failed adding user 'telegraf', data deleted
2025-03-20 10:31:42 GMT−7 info Set permissions
2025-03-20 10:31:42 GMT−7 info About to update telegraf.conf
2025-03-20 10:31:42 GMT−7 info INFO: Overwriting /etc/telegraf/telegraf.conf since no manual changes have been made to it!
2025-03-20 10:31:42 GMT−7 info Updated telegraf.conf
2025-03-20 10:31:42 GMT−7 info Inserting the telegraf influxdb password into the telegraf config
2025-03-20 10:31:42 GMT−7 info Done!
2025-03-20 10:31:42 GMT−7 info INFO: We are root, becoming the oh-la user instead...
2025-03-20 10:31:42 GMT−7 info INFO: Running "oll-influxdb --prepare" as user oh-la
2025-03-20 10:31:42 GMT−7 info (to oh-la) root on none
2025-03-20 10:31:42 GMT−7 info pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)
2025-03-20 10:31:43 GMT−7 info INFO: Running command: prepare
2025-03-20 10:31:43 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:31:43 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:31:58 GMT−7 info INFO: Running command: defaults
2025-03-20 10:31:58 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:31:58 GMT−7 info INFO: Ensuring the ~/.systemd folder exists
2025-03-20 10:31:58 GMT−7 info INFO: Running command: run
2025-03-20 10:31:58 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:31:59 GMT−7 info INFO: Running the InfluxDB container... Should ONLY be run through a systemd service!
2025-03-20 10:31:59 GMT−7 info INFO: Mapping ports: 8086:8086
2025-03-20 10:31:59 GMT−7 info INFO: Mapping ports: 8089:8089/udp
2025-03-20 10:31:59 GMT−7 info INFO: Mapping folders: /home/oh-la/data/influxdb:/var/lib/influxdb
2025-03-20 10:31:59 GMT−7 info INFO: Mapping folders: /etc/ssl/certs:/etc/ssl/certs:ro
2025-03-20 10:31:59 GMT−7 info INFO: Mapping folders: /usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro
2025-03-20 10:31:59 GMT−7 info INFO: Mapping enviroment var: NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
2025-03-20 10:31:59 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_LOGGING_FORMAT=json
2025-03-20 10:31:59 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_REPORTING_DISABLED=true
2025-03-20 10:31:59 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_LOGGING_SUPPRESS_LOGO=false
2025-03-20 10:31:59 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_LOGGING_LEVEL=warn
2025-03-20 10:31:59 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_CONFIG_PATH=/var/lib/influxdb/influxdb.conf
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="/usr/bin/podman filtering at log level info"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="Setting parallel job count to 13"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="/usr/bin/podman filtering at log level info"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"
2025-03-20 10:31:59 GMT−7 info time="2025-03-20T10:31:59-07:00" level=info msg="Setting parallel job count to 13"
2025-03-20 10:31:59 GMT−7 info Resolving "influxdb" using unqualified-search registries (/etc/containers/registries.conf)
2025-03-20 10:31:59 GMT−7 info Trying to pull docker.io/library/influxdb:1.8...
2025-03-20 10:32:31 GMT−7 info time="2025-03-20T10:32:31-07:00" level=warning msg="failed, retrying in 1s ... (1/3). Error: Error initializing source docker://influxdb:1.8: Error reading manifest 1.8 in docker.io/library/influxdb: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit"
2025-03-20 10:33:03 GMT−7 info time="2025-03-20T10:33:03-07:00" level=warning msg="failed, retrying in 2s ... (2/3). Error: Error initializing source docker://influxdb:1.8: Error reading manifest 1.8 in docker.io/library/influxdb: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit"
2025-03-20 10:33:37 GMT−7 info time="2025-03-20T10:33:37-07:00" level=warning msg="failed, retrying in 4s ... (3/3). Error: Error initializing source docker://influxdb:1.8: Error reading manifest 1.8 in docker.io/library/influxdb: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit"
2025-03-20 10:34:13 GMT−7 info toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
2025-03-20 10:34:13 GMT−7 info Error: Error initializing source docker://influxdb:1.8: Error reading manifest 1.8 in docker.io/library/influxdb: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
2025-03-20 10:34:13 GMT−7 info time="2025-03-20T10:34:13-07:00" level=error msg="failed to kill 12596"
2025-03-20 10:34:14 GMT−7 info Error: no arguments are needed with --latest or --cidfile
2025-03-20 10:34:44 GMT−7 info INFO: Running command: prepare
2025-03-20 10:34:44 GMT−7 info Executing the first part of prepare as root
2025-03-20 10:34:44 GMT−7 info WARN: NOT replacing /home/oh-la/data/influxdb/influxdb.conf since there has been manual changes!
2025-03-20 10:34:44 GMT−7 info failed adding user 'telegraf', data deleted
2025-03-20 10:34:44 GMT−7 info Set permissions
2025-03-20 10:34:44 GMT−7 info About to update telegraf.conf
2025-03-20 10:34:44 GMT−7 info INFO: Overwriting /etc/telegraf/telegraf.conf since no manual changes have been made to it!
2025-03-20 10:34:44 GMT−7 info Updated telegraf.conf
2025-03-20 10:34:44 GMT−7 info Inserting the telegraf influxdb password into the telegraf config
2025-03-20 10:34:44 GMT−7 info Done!
2025-03-20 10:34:44 GMT−7 info INFO: We are root, becoming the oh-la user instead...
2025-03-20 10:34:44 GMT−7 info INFO: Running "oll-influxdb --prepare" as user oh-la
2025-03-20 10:34:44 GMT−7 info (to oh-la) root on none
2025-03-20 10:34:44 GMT−7 info pam_unix(su-l:session): session opened for user oh-la(uid=2000) by (uid=0)
2025-03-20 10:34:45 GMT−7 info INFO: Running command: prepare
2025-03-20 10:34:45 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:34:45 GMT−7 info Finalizing preparing for the InfluxDB container as the podman user...
2025-03-20 10:35:49 GMT−7 info INFO: Running command: defaults
2025-03-20 10:35:49 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:35:49 GMT−7 info INFO: Ensuring the ~/.systemd folder exists
2025-03-20 10:35:49 GMT−7 info INFO: Running command: run
2025-03-20 10:35:49 GMT−7 info INFO: Running as required user oh-la
2025-03-20 10:35:58 GMT−7 info INFO: Running the InfluxDB container... Should ONLY be run through a systemd service!
2025-03-20 10:35:58 GMT−7 info INFO: Mapping ports: 8086:8086
2025-03-20 10:35:58 GMT−7 info INFO: Mapping ports: 8089:8089/udp
2025-03-20 10:35:58 GMT−7 info INFO: Mapping folders: /home/oh-la/data/influxdb:/var/lib/influxdb
2025-03-20 10:35:58 GMT−7 info INFO: Mapping folders: /etc/ssl/certs:/etc/ssl/certs:ro
2025-03-20 10:35:58 GMT−7 info INFO: Mapping folders: /usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro
2025-03-20 10:35:58 GMT−7 info INFO: Mapping enviroment var: NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
2025-03-20 10:35:58 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_LOGGING_FORMAT=json
2025-03-20 10:35:58 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_REPORTING_DISABLED=true
2025-03-20 10:35:58 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_LOGGING_SUPPRESS_LOGO=false
2025-03-20 10:35:58 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_LOGGING_LEVEL=warn
2025-03-20 10:35:58 GMT−7 info INFO: Mapping enviroment var: INFLUXDB_CONFIG_PATH=/var/lib/influxdb/influxdb.conf
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="/usr/bin/podman filtering at log level info"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="Setting parallel job count to 13"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="/usr/bin/podman filtering at log level info"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"
2025-03-20 10:35:59 GMT−7 info time="2025-03-20T10:35:59-07:00" level=info msg="Setting parallel job count to 13"
2025-03-20 10:35:59 GMT−7 info Resolving "influxdb" using unqualified-search registries (/etc/containers/registries.conf)
2025-03-20 10:35:59 GMT−7 info Trying to pull docker.io/library/influxdb:1.8...
2025-03-20 10:36:01 GMT−7 info Getting image source signatures
2025-03-20 10:36:01 GMT−7 info Copying blob sha256:76475c2689e229fac9e8ba4a02e64decb7fd62b2a3e4ad65ba97f8e1a35471f2
2025-03-20 10:36:01 GMT−7 info Copying blob sha256:1deb2c2ef23607994f7238c8d97d113f5ebd868b8eb64a0c10d2e0983f036a39
2025-03-20 10:36:02 GMT−7 info Copying blob sha256:8fa0738214bb9b9c99a4687a36d2f88b1da7bee68cfb099439210663bc99d7d6
2025-03-20 10:36:02 GMT−7 info Copying blob sha256:71efbbe32e44303a3305fcc39de75a46e033b4ffd4176944c7e3c7eb7063148a
2025-03-20 10:36:02 GMT−7 info Copying blob sha256:822229d5942c09beb2b6b9cca4f78a6f49f33025a1e30feea735942abb7fb6a5
2025-03-20 10:36:02 GMT−7 info Copying blob sha256:b8ace858e7d3c6b17074be16ab9448e1241e907bef8fa72a4d53144e8a7f065a
2025-03-20 10:36:02 GMT−7 info Copying blob sha256:54df75daf565fb6d2ed179ea7b170f9692df1301c9c1234823572e780a758f19
2025-03-20 10:36:09 GMT−7 info Copying config sha256:7fe366b9a9af62c1d199e01e5898d650011bc52fc18be481851ff4f236aff284
2025-03-20 10:36:09 GMT−7 info Writing manifest to image destination
2025-03-20 10:36:09 GMT−7 info Storing signatures
2025-03-20 10:36:09 GMT−7 info time="2025-03-20T10:36:09-07:00" level=warning msg="Can't read link \"/home/oh-la/.local/share/containers/storage/overlay/l/6W3NDAZU6I53V5TAQYOVN3E7VC\" because it does not exist. Going through storage to recreate the missing symlinks."
2025-03-20 10:36:09 GMT−7 info time="2025-03-20T10:36:09-07:00" level=warning msg="Can't stat lower layer \"/home/oh-la/.local/share/containers/storage/overlay/l/6W3NDAZU6I53V5TAQYOVN3E7VC\" because it does not exist. Going through storage to recreate the missing symlinks."
2025-03-20 10:36:09 GMT−7 info error recreating the missing symlinks: error reading name of symlink for &{"1c0de0b539a9c2a4b6dd5d33ec23867a408dcd8493227734f03c6ce9334f7ffc" 'က' %!q(os.FileMode=2147484096) {%!q(uint64=668000040) %!q(int64=63814361496) %!q(*time.Location=&{Local [{LMT -28378 false} {PDT -25200 true} {PST -28800 false} {PWT -25200 true} {PPT -25200 true}] [{-2717640000 2 false false} {-1633269600 1 false false} {-1615129200 2 false false} {-1601820000 1 false false} {-1583679600 2 true true} {-880207200 3 false false} {-769395600 4 false false} {-765385200 2 false false} {-687967140 1 false false} {-662655600 2 false false} {-620838000 1 false false} {-608137200 2 false false} {-589388400 1 false false} {-576082800 2 false false} {-557938800 1 false false} {-544633200 2 false false} {-526489200 1 false false} {-513183600 2 false false} {-495039600 1 false false} {-481734000 2 false false} {-463590000 1 false false} {-450284400 2 false false} {-431535600 1 false false} {-418230000 2 false false} {-400086000 1 false false} {-386780400 2 false false} {-368636400 1 false false} {-355330800 2 false false} {-337186800 1 false false} {-323881200 2 false false} {-305737200 1 false false} {-292431600 2 false false} {-273682800 1 false false} {-260982000 2 false false} {-242233200 1 false false} {-226508400 2 false false} {-210783600 1 false false} {-195058800 2 false false} {-179334000 1 false false} {-163609200 2 false false} {-147884400 1 false false} {-131554800 2 false false} {-116434800 1 false false} {-100105200 2 false false} {-84376800 1 false false} {-68655600 2 false false} {-52927200 1 false false} {-37206000 2 false false} {-21477600 1 false false} {-5756400 2 false false} {9972000 1 false false} {25693200 2 false false} {41421600 1 false false} {57747600 2 false false} {73476000 1 false false} {89197200 2 false false} {104925600 1 false false} {120646800 2 false false} {126698400 1 false false} {152096400 2 false false} {162381600 1 false false} {183546000 2 false false} {199274400 1 false false} {215600400 2 false false} {230724000 1 false false} {247050000 2 false false} {262778400 1 false false} {278499600 2 false false} {294228000 1 false false} {309949200 2 false false} {325677600 1 false false} {341398800 2 false false} {357127200 1 false false} {372848400 2 false false} {388576800 1 false false} {404902800 2 false false} {420026400 1 false false} {436352400 2 false false} {452080800 1 false false} {467802000 2 false false} {483530400 1 false false} {499251600 2 false false} {514980000 1 false false} {530701200 2 false false} {544615200 1 false false} {562150800 2 false false} {576064800 1 false false} {594205200 2 false false} {607514400 1 false false} {625654800 2 false false} {638964000 1 false false} {657104400 2 false false} {671018400 1 false false} {688554000 2 false false} {702468000 1 false false} {720003600 2 false false} {733917600 1 false false} {752058000 2 false false} {765367200 1 false false} {783507600 2 false false} {796816800 1 false false} {814957200 2 false false} {828871200 1 false false} {846406800 2 false false} {860320800 1 false false} {877856400 2 false false} {891770400 1 false false} {909306000 2 false false} {923220000 1 false false} {941360400 2 false false} {954669600 1 false false} {972810000 2 false false} {986119200 1 false false} {1004259600 2 false false} {1018173600 1 false false} {1035709200 2 false false} {1049623200 1 false false} {1067158800 2 false false} {1081072800 1 false false} {1099213200 2 false false} {1112522400 1 false false} {1130662800 2 false false} {1143972000 1 false false} {1162112400 2 false false} {1173607200 1 false false} {1194166800 2 false false} {1205056800 1 false false} {1225616400 2 false false} {1236506400 1 false false} {1257066000 2 false false} {1268560800 1 false false} {1289120400 2 false false} {1300010400 1 false false} {1320570000 2 false false} {1331460000 1 false false} {1352019600 2 false false} {1362909600 1 false false} {1383469200 2 false false} {1394359200 1 false false} {1414918800 2 false false} {1425808800 1 false false} {1446368400 2 false false} {1457863200 1 false false} {1478422800 2 false false} {1489312800 1 false false} {1509872400 2 false false} {1520762400 1 false false} {1541322000 2 false false} {1552212000 1 false false} {1572771600 2 false false} {1583661600 1 false false} {1604221200 2 false false} {1615716000 1 false false} {1636275600 2 false false} {1647165600 1 false false} {1667725200 2 false false} {1678615200 1 false false} {1699174800 2 false false} {1710064800 1 false false} {1730624400 2 false false} {1741514400 1 false false} {1762074000 2 false false} {1772964000 1 false false} {1793523600 2 false false} {1805018400 1 false false} {1825578000 2 false false} {1836468000 1 false false} {1857027600 2 false false} {1867917600 1 false false} {1888477200 2 false false} {1899367200 1 false false} {1919926800 2 false false} {1930816800 1 false false} {1951376400 2 false false} {1962871200 1 false false} {1983430800 2 false false} {1994320800 1 false false} {2014880400 2 false false} {2025770400 1 false false} {2046330000 2 false false} {2057220000 1 false false} {2077779600 2 false false} {2088669600 1 false false} {2109229200 2 false false} {2120119200 1 false false} {2140678800 2 false false}] PST8PDT,M3.2.0,M11.1.0 1741514400 1762074000 0x4000162160})} {'\x1f' '\U00010265' '䇀' '\x03' '\x00' '\x00' '\x00' '\x00' 'က' 'က' '\x00' '\b' {%!q(int64=1741588746) %!q(int64=92565444)} {%!q(int64=1678764696) %!q(int64=668000040)} {%!q(int64=1678764696) %!q(int64=668000040)} ['\x00' '\x00']}}: open /home/oh-la/.local/share/containers/storage/overlay/1c0de0b539a9c2a4b6dd5d33ec23867a408dcd8493227734f03c6ce9334f7ffc/link: no such file or directory
2025-03-20 10:36:09 GMT−7 info Error: Error committing the finished image: error adding layer with blob "sha256:71efbbe32e44303a3305fcc39de75a46e033b4ffd4176944c7e3c7eb7063148a": error recreating the missing symlinks: error reading name of symlink for &{"1c0de0b539a9c2a4b6dd5d33ec23867a408dcd8493227734f03c6ce9334f7ffc" 'က' %!q(os.FileMode=2147484096) {%!q(uint64=668000040) %!q(int64=63814361496) %!q(*time.Location=&{Local [{LMT -28378 false} {PDT -25200 true} {PST -28800 false} {PWT -25200 true} {PPT -25200 true}] [{-2717640000 2 false false} {-1633269600 1 false false} {-1615129200 2 false false} {-1601820000 1 false false} {-1583679600 2 true true} {-880207200 3 false false} {-769395600 4 false false} {-765385200 2 false false} {-687967140 1 false false} {-662655600 2 false false} {-620838000 1 false false} {-608137200 2 false false} {-589388400 1 false false} {-576082800 2 false false} {-557938800 1 false false} {-544633200 2 false false} {-526489200 1 false false} {-513183600 2 false false} {-495039600 1 false false} {-481734000 2 false false} {-463590000 1 false false} {-450284400 2 false false} {-431535600 1 false false} {-418230000 2 false false} {-400086000 1 false false} {-386780400 2 false false} {-368636400 1 false false} {-355330800 2 false false} {-337186800 1 false false} {-323881200 2 false false} {-305737200 1 false false} {-292431600 2 false false} {-273682800 1 false false} {-260982000 2 false false} {-242233200 1 false false} {-226508400 2 false false} {-210783600 1 false false} {-195058800 2 false false} {-179334000 1 false false} {-163609200 2 false false} {-147884400 1 false false} {-131554800 2 false false} {-116434800 1 false false} {-100105200 2 false false} {-84376800 1 false false} {-68655600 2 false false} {-52927200 1 false false} {-37206000 2 false false} {-21477600 1 false false} {-5756400 2 false false} {9972000 1 false false} {25693200 2 false false} {41421600 1 false false} {57747600 2 false false} {73476000 1 false false} {89197200 2 false false} {104925600 1 false false} {120646800 2 false false} {126698400 1 false false} {152096400 2 false false} {162381600 1 false false} {183546000 2 false false} {199274400 1 false false} {215600400 2 false false} {230724000 1 false false} {247050000 2 false false} {262778400 1 false false} {278499600 2 false false} {294228000 1 false false} {309949200 2 false false} {325677600 1 false false} {341398800 2 false false} {357127200 1 false false} {372848400 2 false false} {388576800 1 false false} {404902800 2 false false} {420026400 1 false false} {436352400 2 false false} {452080800 1 false false} {467802000 2 false false} {483530400 1 false false} {499251600 2 false false} {514980000 1 false false} {530701200 2 false false} {544615200 1 false false} {562150800 2 false false} {576064800 1 false false} {594205200 2 false false} {607514400 1 false false} {625654800 2 false false} {638964000 1 false false} {657104400 2 false false} {671018400 1 false false} {688554000 2 false false} {702468000 1 false false} {720003600 2 false false} {733917600 1 false false} {752058000 2 false false} {765367200 1 false false} {783507600 2 false false} {796816800 1 false false} {814957200 2 false false} {828871200 1 false false} {846406800 2 false false} {860320800 1 false false} {877856400 2 false false} {891770400 1 false false} {909306000 2 false false} {923220000 1 false false} {941360400 2 false false} {954669600 1 false false} {972810000 2 false false} {986119200 1 false false} {1004259600 2 false false} {1018173600 1 false false} {1035709200 2 false false} {1049623200 1 false false} {1067158800 2 false false} {1081072800 1 false false} {1099213200 2 false false} {1112522400 1 false false} {1130662800 2 false false} {1143972000 1 false false} {1162112400 2 false false} {1173607200 1 false false} {1194166800 2 false false} {1205056800 1 false false} {1225616400 2 false false} {1236506400 1 false false} {1257066000 2 false false} {1268560800 1 false false} {1289120400 2 false false} {1300010400 1 false false} {1320570000 2 false false} {1331460000 1 false false} {1352019600 2 false false} {1362909600 1 false false} {1383469200 2 false false} {1394359200 1 false false} {1414918800 2 false false} {1425808800 1 false false} {1446368400 2 false false} {1457863200 1 false false} {1478422800 2 false false} {1489312800 1 false false} {1509872400 2 false false} {1520762400 1 false false} {1541322000 2 false false} {1552212000 1 false false} {1572771600 2 false false} {1583661600 1 false false} {1604221200 2 false false} {1615716000 1 false false} {1636275600 2 false false} {1647165600 1 false false} {1667725200 2 false false} {1678615200 1 false false} {1699174800 2 false false} {1710064800 1 false false} {1730624400 2 false false} {1741514400 1 false false} {1762074000 2 false false} {1772964000 1 false false} {1793523600 2 false false} {1805018400 1 false false} {1825578000 2 false false} {1836468000 1 false false} {1857027600 2 false false} {1867917600 1 false false} {1888477200 2 false false} {1899367200 1 false false} {1919926800 2 false false} {1930816800 1 false false} {1951376400 2 false false} {1962871200 1 false false} {1983430800 2 false false} {1994320800 1 false false} {2014880400 2 false false} {2025770400 1 false false} {2046330000 2 false false} {2057220000 1 false false} {2077779600 2 false false} {2088669600 1 false false} {2109229200 2 false false} {2120119200 1 false false} {2140678800 2 false false}] PST8PDT,M3.2.0,M11.1.0 1741514400 1762074000 0x4000162160})} {'\x1f' '\U00010265' '䇀' '\x03' '\x00' '\x00' '\x00' '\x00' 'က' 'က' '\x00' '\b' {%!q(int64=1741588746) %!q(int64=92565444)} {%!q(int64=1678764696) %!q(int64=668000040)} {%!q(int64=1678764696) %!q(int64=668000040)} ['\x00' '\x00']}}: open /home/oh-la/.local/share/containers/storage/overlay/1c0de0b539a9c2a4b6dd5d33ec23867a408dcd8493227734f03c6ce9334f7ffc/link: no such file or directory
I have the restic backups going, but based on that error I would have no clue what actually needs restoring. I have tried removing the image, redownloading, multiple reboots, but nothing has helped so far. The rest of my containers are fine which makes this even more puzzling.
Here is another datapoint that seems quite strange, if I podman rm --force the container ID, this goes away, but not liking the idea of that.
oh-la@Collective-Core:~$ podman ps
ERRO[0000] error joining network namespace for container a9b619aa191829efb56ee576784fc005930f2d62105be5880bc5b488c7579d52: error retrieving network namespace at /run/user/2000/netns/cni-803fe804-99de-e9d7-ce2b-f8929f57eb80: unknown FS magic on "/run/user/2000/netns/cni-803fe804-99de-e9d7-ce2b-f8929f57eb80": 1021994
Error: error joining network namespace of container a9b619aa191829efb56ee576784fc005930f2d62105be5880bc5b488c7579d52: error retrieving network namespace at /run/user/2000/netns/cni-803fe804-99de-e9d7-ce2b-f8929f57eb80: unknown FS magic on "/run/user/2000/netns/cni-803fe804-99de-e9d7-ce2b-f8929f57eb80": 1021994
Trying to stay positive but going to bed with everything humming along, to waking up with everything broken is quite frustrating. Now some stuff is fixed but Z2M being broken is a pretty big deal.