Logs are getting spammed with stacktraces "WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown:"

My lemmy-ui container is spiking cpu usage and looking into the logs the following shows up (900.000 lines in total). Any ideas what the issue could be? I’m on 0.19.1, everything seems to be working fine aside from the logs filling up and the usage piking sometimes.


<span style="color:#323232;">Stack backtrace:
</span><span style="color:#323232;">  44: clone
</span><span style="color:#323232;">   0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=lemmy.*.com http.target=/inbox otel.kind="server" request_id=26ca1ef9-41e8-4666-8d75-1143e141f3aa http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">2024-01-12T04:00:14.322787Z  INFO lemmy_server::scheduled_tasks: Done.
</span><span style="color:#323232;">2024-01-12T04:00:14.322817Z  INFO lemmy_server::scheduled_tasks: Updating banned column if it expires ...
</span><span style="color:#323232;">2024-01-12T04:01:40.121454Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown: Fetched remote object https://mastodon.online/users/amongusmerch which was deleted
</span><span style="color:#323232;">
</span><span style="color:#323232;">Stack backtrace:
</span><span style="color:#323232;">  44: clone
</span><span style="color:#323232;">   0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=lemmy.*.com http.target=/inbox otel.kind="server" request_id=5e8af779-1106-46e7-9c51-86a02226c80e http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">2024-01-12T04:02:10.254940Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown: Fetched remote object https://mastodon.online/users/caspermerch which was deleted
</span><span style="color:#323232;">
</span><span style="color:#323232;">Stack backtrace:
</span><span style="color:#323232;">  44: clone
</span><span style="color:#323232;">   0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=lemmy.*.com http.target=/inbox otel.kind="server" request_id=5436b893-9fd5-44b0-8eff-e6d719846f06 http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">2024-01-12T04:02:21.673184Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown: Fetched remote object https://mastodon.online/users/huntshowdownmerch which was deleted
</span><span style="color:#323232;">
</span><span style="color:#323232;">Stack backtrace:
</span><span style="color:#323232;">  44: clone
</span><span style="color:#323232;">   0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=lemmy.*.com http.target=/inbox otel.kind="server" request_id=13cbde1d-9cdf-46c9-bb5b-0500a09032e8 http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">2024-01-12T04:02:39.233929Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown: Fetched remote object https://mastodon.online/users/spacefrogsmerch which was deleted
</span><span style="color:#323232;">
</span><span style="color:#323232;">Stack backtrace:
</span><span style="color:#323232;">  44: clone
</span><span style="color:#323232;">   0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=lemmy.*.com http.target=/inbox otel.kind="server" request_id=dd6a8922-be67-4f46-97d5-8d43e97658c5 http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">2024-01-12T04:02:59.692759Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Unknown: expected value at line 2 column 5
</span>
willya,
@willya@lemmyf.uk avatar

Any reason for not going to 0.19.2?

haui_lemmy,

no, just that I’ve read that it is discouraged to put the compose files to “latest” so they get updated periodically. I updated a day or two before 0.19.2 came out I believe.

bjoern_tantau,
@bjoern_tantau@swg-empire.de avatar

0.19.0 and 0.19.1 had a bunch of nasty bugs. You should update.

haui_lemmy,

Okay! Thank you! Will do.

sunaurus,
@sunaurus@lemm.ee avatar

Did the upgrade solve it for you?

haui_lemmy,

I have started changing the compose file to upgrade but the changes are more extensive than I thought, more commands and other additions so I had to pause until I have more time (tomorrow or the day after I hope). Thanks for asking though. :)

haui_lemmy,

So I performed the upgrade and it bricked my install completely. The reason seems to be that something cant handle png icons and after about 4 hrs of working on it, searching and trying different approaches, the site is now back online. Now I need to take an extended break and tomorrow I might check if the error messages are gone. I did find some of the same in the log just a couple mins ago so I dont think this has helped. I might have to turn off backtraces so its less convoluted.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • lemmy_support@lemmy.ml
  • localhost
  • All magazines
  • Loading…
    Loading the web debug toolbar…
    Attempt #