JamesT
July 16, 2024, 2:54pm
1
Welcome to the Lite-Node Bug Report forum! The primary function of this forum is to collect reports on issues currently affecting Lite-Nodes. Our goal is that this forum will become an extremely valuable tool to the development team, allowing all of us to help make diode the best network it can be.
The quality assurance team will be monitoring this forum continuously, looking for new issues and then adding them to our bug tracking database.
While you may not see us post here very often, please know that your reports are read and appreciated. In the event that the amount of information in a bug report proves to be insufficient during our investigation, you may be asked for additional information (like your hardware configuration or what time you encountered a specific issue)—so keep an eye out!
Please keep all posts on topic. This means threads should center around bug reports, workarounds, and discussion of impacting issues.
When providing information on a bug report, it is okay to post on an existing thread if your issue is exactly the same as the original post. If you are experiencing a similar issue with slight variations, please create your own topic.
If you find multiple bugs, create a separate thread for each one. While this may seem tedious, it is essential to our workflow as it allows our internal tools track which specific bugs have been investigated and it helps us share the workload among our team.
Do not hijack threads with your new topic. Posting an issue which is similar but different from the original topic means that your request may be missed or the original request ignored.
Please post a subject with a clear summary of your issue. Starting posts with “I need help!” or “Can’t play no matter what” only serves to make your issue harder to resolve.
We understand that emotions can run high while dealing with technical issues, but posting in an abusive or disparaging manner will not be tolerated, and does not advance our goal of providing assistance to the widest audience of customers in this forum.
Please do not post direct contact info for other companies. It is acceptable to post links to websites where contact info can be found, but posting the information here on the forum is not permitted.
As always, community forums work best when all participants treat each other with respect and courtesy before posting.
Thank you for supporting Diode.
JamesT
July 17, 2024, 2:30pm
2
@Knusperhaus when entering the shell and running the diode-node.info I get an error
/snap/bin/diode-node.shell
=ERROR REPORT==== 17-Jul-2024::14:27:44.824279 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
=ERROR REPORT==== 17-Jul-2024::14:27:44.887349 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
=ERROR REPORT==== 17-Jul-2024::14:27:44.927296 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
Erlang/OTP 26 [erts-14.2.3] [source] [64-bit] [smp:128:1] [ds:128:1:10] [async-threads:1] [jit:ns]
Interactive Elixir (1.15.7) - press Ctrl+C to exit (type h() ENTER for help)
iex(diode_node@nodelite001-1cpu-1gb-us-sjo1)1> diode-node.info
** (CompileError) cannot compile code (errors have been logged)
iex(diode_node@nodelite001-1cpu-1gb-us-sjo1)1> h
IEx.Helpers was not compiled with docs
*** ERROR: Shell process terminated! (^G to start new job) ***
iex(diode_node@nodelite001-1cpu-1gb-us-sjo1)2>
JamesT
July 17, 2024, 2:32pm
3
runnign from the path
/snap/bin/diode-node.info
=ERROR REPORT==== 17-Jul-2024::14:31:37.197599 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
=ERROR REPORT==== 17-Jul-2024::14:31:37.251027 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
=ERROR REPORT==== 17-Jul-2024::14:31:37.287794 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
== Diode Node 0x22612cf691c006cc656b0489eaXXXXXXXXXXXXXXX ==
Version : 1.2.1
Connected Devices: 2
Connected Peers : 8
Current Epoch : 664
Ticket Score : 0
Previous Epoch : 663
Ticket Score : 0
1 Like
Yes, confirmed. Looking at that
1 Like
Server crashes and restarts after this exception:
2024-07-19T18:58:30Z diode-node.diode-node[711532]: 18:58:30.252 [error] Process #PID<0.94431.0> on node :diode_node@vps38754 raised an exception
2024-07-19T18:58:30Z diode-node.diode-node[711532]: ** (MatchError) no match of right hand side value: {:error, :enoent}
2024-07-19T18:58:30Z diode-node.diode-node[711532]: (dets_plus 2.3.2) lib/dets_plus.ex:895: anonymous fn/7 in DetsPlus.spawn_sync_worker/1
EDIT: this happens pretty much exactly every 3 minutes
2 Likes
JamesT
July 21, 2024, 2:38pm
7
light nodes causing a client host error, error at web browser end resulting in error 500
**doConnectDevice() for 'media-stream' failed: GetServer(): wrong signature in server object &{Host:[57 52 46 50 51 55 46 50 46 49 54 56] EdgePort:41046 ServerPort:51055 Sig:[0 135 196 209 183 211 137 67 142 37 219 83 38 171 183 72 174 14 223 48 253 153 90 187 192 174 71 74 236 72 224 113 110 105 71 109 8 147 160 231 17 205 160 183 88 110 159 246 16 129 179 35 94 128 75 85 222 167 165 28 81 161 239 176 254] ServerPubKey:[4 170 175 49 75 113 70 225 30 44 47 24 2 46 12 180 57 254 168 61 173 150 50 66 61 65 185 80 116 63 147 62 90 149 59 87 117 14 155 106 34 2 198 196 129 170 62 150 51 110 178 37 34 16 153 219 22 201 112 66 135 88 238 233 222] Extra:map[block:{neg:false abs:[6678668]} name:{neg:false abs:[3275074798935961137 3558811779124520802 7811903006599819565 8315171310995989605 8315171473952368997 28025]} tickets:{neg:false abs:[]} time:{neg:false abs:[1721571447054126263]} uptime:{neg:false abs:[24683378]}]} with 2 candidates**
update on client side
INFO Retry to connect (1/3), waiting 5s server=94.237.2.168:41046
ERROR Couldn't add block 7597200 [0 0 12 245 94 103 146 111 80 168 80 61 106 28 44 175 118 123 6 19 226 161 251 51 238 158 193 128 247 137 84 143]: don't know direct parent of this block server=95.111.218.30:41046
DEBUG Added relay#4 [0x982039c3aaef4f40896793ec7ed2fc7afb0f6492] @ 94.237.2.168:41046
ERROR Couldn't add block 7597200 [0 0 12 245 94 103 146 111 80 168 80 61 106 28 44 175 118 123 6 19 226 161 251 51 238 158 193 128 247 137 84 143]: don't know direct parent of this block server=185.70.196.190:41046
ERROR Couldn't add block 7597200 [0 0 12 245 94 103 146 111 80 168 80 61 106 28 44 175 118 123 6 19 226 161 251 51 238 158 193 128 247 137 84 143]: don't know direct parent of this block server=95.111.218.30:41046
ERROR Couldn't add block 7597200 [0 0 12 245 94 103 146 111 80 168 80 61 106 28 44 175 118 123 6 19 226 161 251 51 238 158 193 128 247 137 84 143]: don't know direct parent of this block server=185.70.196.190:41046
ERROR Couldn't add block 7597200 [0 0 12 245 94 103 146 111 80 168 80 61 106 28 44 175 118 123 6 19 226 161 251 51 238 158 193 128 247 137 84 143]: don't know direct parent of this block server=95.111.218.30:41046
ERROR Couldn't add block 7597204 [0 0 12 220 33 174 142 107 88 36 221 135 93 118 248 150 193 187 172 109 254 70 125 193 173 70 156 63 212 31 225 156]: don't know direct parent of this block server=94.237.2.1
v1.2.5 is released and should a address two issues:
diode-node.info should will not issue these errors anymore:
=ERROR REPORT==== 17-Jul-2024::14:31:37.197599 ===
File operation error: eacces. Target: .. Function: list_dir. Process: application_controller.
@JamesT
DetsPlus related errors should auto-recover
@fluffycroutons
2 Likes
multiple light nodes causing a host error,
Are you saying you’re running multiple light nodes on the same server? Sorry I’m unclear on this…
JamesT
July 22, 2024, 11:24am
10
No all my light nodes with fallback are still producing a 500 error on the client side
JamesT
July 22, 2024, 11:26am
11
Knusperhaus:
diode-node.info
diode-node.info works perfect now
1 Like
DetsPlus error is gone now after update, but the node is still in a reboot loop:
<prints "coinbase" line with address>
2024-07-22T17:08:14Z diode-node.diode-node[788934]: 17:08:14.888 [info] Cron: Executing Broadcast Self...
2024-07-22T17:08:14Z diode-node.diode-node[788934]: 17:08:14.888 [info] Cron: Executing Reload Cert...
2024-07-22T17:08:14Z diode-node.diode-node[788934]: 17:08:14.893 [info] maybe_import_key: Checking ["/var/snap/diode-node/7/priv", "/var/snap/diode-node/7"]'
2024-07-22T17:08:14Z diode-node.diode-node[788934]: 17:08:14.935 [info] PeerHandlerV2 ceca2f@us1.prenet.diode.io: Creating connect worker
2024-07-22T17:08:14Z diode-node.diode-node[788934]: 17:08:14.937 [info] PeerHandlerV2 7e4cd3@us2.prenet.diode.io: Creating connect worker
2024-07-22T17:08:14Z diode-node.diode-node[788934]: 17:08:14.942 [info] PeerHandlerV2 68e0ba@as1.prenet.diode.io: Creating connect worker
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:14.948 [info] PeerHandlerV2 1350d3@as2.prenet.diode.io: Creating connect worker
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:14.989 [info] PeerHandlerV2 937c49@eu1.prenet.diode.io: Creating connect worker
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:14.994 [info] PeerHandlerV2 ae6992@eu2.prenet.diode.io: Creating connect worker
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.099 [info] PeerHandlerV2 d17005@localhost: Creating connect worker
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.955 [info] PeerHandlerV2 1350d3@as2.prenet.diode.io: hello from: fringe_quiz (0x1350d3b501d6842ed881b59de4b95b27372bfae8)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.955 [info] PeerHandlerV2 ae6992@eu2.prenet.diode.io: hello from: feel_panther (0xae699211c62156b8f29ce17be47d2f069a27f2a6)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.955 [info] PeerHandlerV2 937c49@eu1.prenet.diode.io: hello from: grain_silk (0x937c492a77ae90de971986d003ffbc5f8bb2232c)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.955 [info] PeerHandlerV2 68e0ba@as1.prenet.diode.io: hello from: catch_also (0x68e0bafdda9ef323f692fc080d612718c941d120)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.956 [info] PeerHandlerV2 7e4cd3@us2.prenet.diode.io: hello from: pause_chalk (0x7e4cd38d266902444dc9c8f7c0aa716a32497d0b)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.956 [info] PeerHandlerV2 d17005@127.0.0.1: hello from: hotel_story (0xd170058e208ac039b50c8342c45b9ed8e975a371)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.957 [info] PeerHandlerV2 ceca2f@us1.prenet.diode.io: hello from: enrich_turtle (0xceca2f8cf1983b4cf0c1ba51fd382c2bc37aba58)
2024-07-22T17:08:15Z diode-node.diode-node[788934]: 17:08:15.958 [info] PeerHandlerV2 d17005@localhost: hello from: hotel_story (0xd170058e208ac039b50c8342c45b9ed8e975a371)
2024-07-22T17:08:19Z diode-node.diode-node[788934]: 17:08:19.839 [info] EdgeV2 746b9e@180.131.145.33: accepted connection
2024-07-22T17:08:20Z diode-node.diode-node[788934]: 17:08:20.158 [info] PeerHandlerV2 24e22f@192.168.0.72: Creating connect worker
2024-07-22T17:08:22Z diode-node.diode-node[788934]: 17:08:22.160 [info] Timeout while getting a result from stool_dress (0x24e22f3e0843b1df1419397acdfb01bf1bd0aeb3)
2024-07-22T17:08:24Z diode-node.diode-node[788934]: 17:08:24.172 [info] Timeout while getting a result from stool_dress (0x24e22f3e0843b1df1419397acdfb01bf1bd0aeb3)
2024-07-22T17:08:25Z diode-node.diode-node[788934]: 17:08:25.162 [info] PeerHandlerV2 24e22f@192.168.0.72:51055: Connection failed in ssl.connect(): {:error, :timeout}
2024-07-22T17:08:25Z diode-node.diode-node[788934]: 17:08:25.162 [info] Node stool_dress (0x24e22f3e0843b1df1419397acdfb01bf1bd0aeb3) down for: :normal
2024-07-22T17:08:35Z diode-node.diode-node[788934]: 17:08:35.171 [info] PeerHandlerV2 24e22f@192.168.0.72: Creating connect worker
2024-07-22T17:08:37Z diode-node.diode-node[788934]: 17:08:37.172 [info] Timeout while getting a result from stool_dress (0x24e22f3e0843b1df1419397acdfb01bf1bd0aeb3)
2024-07-22T17:08:40Z diode-node.diode-node[788934]: 17:08:40.173 [info] PeerHandlerV2 24e22f@192.168.0.72:51055: Connection failed in ssl.connect(): {:error, :timeout}
2024-07-22T17:08:40Z diode-node.diode-node[788934]: 17:08:40.173 [info] Node stool_dress (0x24e22f3e0843b1df1419397acdfb01bf1bd0aeb3) down for: :normal
<crashes here and restarts>
I don’t see a crash message. You want to check dmesg -T – If it’s crashing because it’s running out of memory you should see a message there in the kernel log.
dmesg -T shows these references to diode-node but they don’t look like crashes:
[Tue Jul 23 14:14:56 2024] kauditd_printk_skb: 57 callbacks suppressed
[Tue Jul 23 14:14:56 2024] audit: type=1400 audit(1721744105.044:2541): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/proc/1260043/mountinfo" pid=1260043 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:14:56 2024] audit: type=1400 audit(1721744105.092:2542): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/proc/1260043/mountinfo" pid=1260043 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:14:56 2024] audit: type=1400 audit(1721744105.100:2543): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/proc/1260043/mountinfo" pid=1260043 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:14:57 2024] audit: type=1400 audit(1721744105.964:2544): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/home/node/" pid=1260043 comm="erts_dios_10" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:14:57 2024] audit: type=1400 audit(1721744106.060:2545): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/home/node/" pid=1260043 comm="erts_dios_10" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:14:57 2024] audit: type=1400 audit(1721744106.112:2546): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/home/node/" pid=1260043 comm="erts_dios_5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:15:05 2024] audit: type=1400 audit(1721744113.764:2547): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/proc/1260135/mountinfo" pid=1260135 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:15:05 2024] audit: type=1400 audit(1721744113.764:2548): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/proc/1260135/mountinfo" pid=1260135 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:15:05 2024] audit: type=1400 audit(1721744113.768:2549): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/proc/1260135/mountinfo" pid=1260135 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:15:05 2024] audit: type=1400 audit(1721744114.288:2550): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/home/node/" pid=1260135 comm="erts_dios_9" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:15:05 2024] audit: type=1400 audit(1721744114.336:2551): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/home/node/" pid=1260135 comm="erts_dios_9" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[Tue Jul 23 14:15:05 2024] audit: type=1400 audit(1721744114.400:2552): apparmor="DENIED" operation="open" profile="snap.diode-node.info" name="/home/node/" pid=1260135 comm="erts_dios_9" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
On v1.2.5 these crashes are happening again every few minutes:
2024-07-23T14:11:30Z diode-node.diode-node[788934]: 14:11:30.131 [error] Process #PID<0.2423365.0> on node :diode_node@vps38754 raised an exception
2024-07-23T14:11:30Z diode-node.diode-node[788934]: ** (MatchError) no match of right hand side value: {:error, :enoent}
2024-07-23T14:11:30Z diode-node.diode-node[788934]: (dets_plus 2.3.2) lib/dets_plus.ex:895: anonymous fn/7 in DetsPlus.spawn_sync_worker/1
JamesT
July 24, 2024, 3:10pm
16
I wrote a script for listing oom values, Just copy and paste the script into an executable script like oom.sh
#!/bin/bash
# This script retrieves and displays the OOM (Out Of Memory) score and the OOM adjusted score
# for each running process, sorted in descending order by the OOM score.
printf 'PID\tOOM Score\tOOM Adj\tCommand\n'
# Read each process ID and command, check if a corresponding oom_score file exists and its value is not zero.
# If so, print the process ID, OOM score, OOM adjusted score, and command.
while read -r pid comm
do
if [ -f /proc/$pid/oom_score ] && [ $(cat /proc/$pid/oom_score) != 0 ]
then
printf '%d\t%d\t\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$(cat /proc/$pid/oom_score_adj)" "$comm"
fi
done < <(ps -e -o pid= -o comm=) | sort -k 2nr
example output, this shows my beam.smp is using a lot of cpu, to avoid crashes I activated a swap file
PID OOM Score OOM Adj Command
300659 1186 0 beam.smp
329476 682 0 alloy
670 669 0 networkd-dispat
790 669 0 unattended-upgr
25289 668 0 systemd-resolve
328154 668 0 (sd-pam)
27131 667 0 rsyslogd
328147 667 0 sshd
328246 667 0 bash
49767 667 0 packagekitd
49775 667 0 udisksd
25283 666 0 systemd-network
26271 666 0 systemd-timesyn
300712 666 0 epmd
300716 666 0 erl_child_setup
301118 666 0 inet_gethost
301119 666 0 inet_gethost
301120 666 0 inet_gethost
301121 666 0 inet_gethost
301126 666 0 inet_gethost
328150 666 0 systemd
330982 666 0 bash
330991 666 0 htop
342204 666 0 sshd
342209 666 0 oom.sh
342210 666 0 oom.sh
342211 666 0 sort
49762 666 0 cron
49765 666 0 upowerd
49766 666 0 acpid
49773 666 0 polkitd
677 666 0 systemd-logind
731 666 0 login
25293 501 -250 systemd-journal
201676 72 -900 snapd
664 68 -900 dbus-daemon
my htop here shows diode grabbing swap and memory and cpu
hope it helps
JamesT
July 24, 2024, 3:14pm
17
whats your spec of build and os versions etc I am running ubuntu 22.04 and allis good with no crashes, just a memory leak and cpu leaks
I’m ubuntu 22.04 too. Maybe my 1GB memory isn’t enough.
@Knusperhaus original fix did seem to work at first, but now the same issue (the DetsPlus.spawn_sync_worker one) is back.
JamesT
July 24, 2024, 8:15pm
20
I am running the same memory config. Check to see if apparmor is denying beam.smp. also create a 1GB swap file to see if that stops the crashing.
What provider are you hosting on and what does dmesg -T show
