I put it all under a spoiler tag because it’s a lot. Let me know if you’re interested!
inventory/specs# UPS Eaton 5SC 1000 full sine-wave inverter # Rack 13U enclosed rack w/casters and magnetic front door # Networking ## TP-Link EAP225 Wifi AP ## Aruba Networks S2500-24P-US Switch - 24 Port Gigabit Switch - PoE - 4x SFP+ 10Gbit ports # Servers ## Dell R720xd ### Components - 2x Intel Xeon E5-2667 v2 @ 3.3GHz (8-core CPUs) - 14x 8GB Samsung ECC 2Rx4 Dual Rank DDR3 10600R 1333MHz RAM (112 GB) - Intel 4P X520 NIC (2x SFP+ 10Gbit, 2x 1Gbit) - 2x 750W PSU - IDSDM 6YFN5 dual SD module - 2x Sandisk 16GB UHS-1 Extreme SDHC SD cards - PERC H710P Mini Host Bus Adapter - PERC H310 Host Bus Adapter - Dual 2.5" Hotswap Drive Backplane 0JDG3 - 2x Crucial MX500 500GB SSD - 2x 2.5" Dell Hotswap Drive Caddies/Trays - Front Hotswap HDD Backplane - 12x HGST Ultrastar KP06 6TB 7200RPM HDDs - 12x 3.5" Dell Hotswap Drive Caddies/Trays - Rack Rails (They hold the server in place, but they’re missing some bearings. If the server is pulled out on the rails it may not go back. Replacing these should be less than $50) - Locking front panel - iDRAC Module ### Notes - Runs ~235W at idle - Can handle many VMs and multiple simultaneous 4k Plex transcodes - This is basically the best set of parts for the xx20 series Dell servers and is more capable than a lot of the xx30 units ## Dell R710 ### Components - Rack Rails - Locking front panel - CD Drive to 2.5" Drive Adapter - Samsung 860 Evo 250GB SSD - Front Hotswap HDD Backplane - 6x 3.5" Dell Hotswap Drive Caddies/Trays - 2x 870W PSU - PERC H216 Host Bus Adapter - 2x Intel Xeon L5640 @2.26GHz (6-core CPUs) - 18x 4GB Samsung ECC 2Rx8 Dual Rank DDR3 10600R 1333MHz RAM (72GB) - Dell 0KJYDB 2xSFP+ 10Gbit NIC ### Notes - Set up to be an ideal backup server - Just add hard drives and it will be ready to go - iDRAC modules are available on eBay if you would like out of band management
Are you absolutely sure that NPM has an IP from the subnet 172.22.0.0/24? Is there any way you can remove the trusted_proxies setting from homeassistant and then check if it will accept the connection from NPM?
I have set it but it wont change anything. You can access the docker inspect here pastebin.com/t1T98RCwI can imagine that this problem is before homeassistant as even if I ignore the certificate error , it will not forward me to homeassistant but to my router / a warning page from my router saying it has blocked me.
If I test the server reachability inside nginx manager it will ask me if npm is configured correctly, so you might be onto something with NPM configuration …
I have now set up duckdns over docker instead of over my router, but it hasnt helped anything. My Duckdns IP is the same (and its correct, if I just open this IPV4 Address it will redirect to my nginx landing page).
Okay I think here is the error. AFter doing the Test Server Reachability the following will come up in the nginx-db logs: 2023-12-29 21:06:25 3 [Warning] Aborted connection 3 to db: ‘npm’ user: ‘npm’ host: ‘172.22.0.8’ (Got an error reading communication packets)
Now I have no clue why this is ( I think this is the end for today as my head is about to explode). Docker inspect nginx reveals that this request for sure came from nginx (as it has the .0.8 ip).
As far as motherboards go, you would probably be fine with any consumer desktop brand but you should probably look for something with dual NIC. If you want something a bit more robust AsRock Rack has some really great options. I’ve been using the X470D4U for about 4 years now without any issues.
As in, I have Nginx running on my server and use it as a reverse proxy to access a variety of apps and services. But can’t get it playing nicely with AIO Nextcloud.
When things lock up, will a kill -9 kill rsync or not? If it doesn’t, and the zpool status lockup is suspicious, it means things are stuck inside a system call. I’ve seen all sorts of horrible things with usb timeouts. Check your syslog.
Just tested, thanks for the suggestion! It killed a few instances of rsync, but there are two apparently stuck open. I issued reboot and the system seemed to hang while waiting for rsync to be killed and failed to unmount the zpool.
Syslog errors:
<span style="color:#323232;">Dec 31 16:53:34 halnas kernel: [54537.789982] #PF: error_code(0x0002) - not-present page
</span><span style="color:#323232;">Jan 1 12:57:19 halnas systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (file watch) being skipped.
</span><span style="color:#323232;">Jan 1 12:57:19 halnas systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (timer based) being skipped.
</span><span style="color:#323232;">Jan 1 12:57:19 halnas kernel: [ 1.119609] pcieport 0000:00:1b.0: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
</span><span style="color:#323232;">Jan 1 12:57:19 halnas kernel: [ 1.120020] pcieport 0000:00:1d.2: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
</span><span style="color:#323232;">Jan 1 12:57:19 halnas kernel: [ 1.120315] pcieport 0000:00:1d.3: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
</span><span style="color:#323232;">Jan 1 22:59:08 halnas kernel: [ 1.119415] pcieport 0000:00:1b.0: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
</span><span style="color:#323232;">Jan 1 22:59:08 halnas kernel: [ 1.119814] pcieport 0000:00:1d.2: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
</span><span style="color:#323232;">Jan 1 22:59:08 halnas kernel: [ 1.120112] pcieport 0000:00:1d.3: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
</span><span style="color:#323232;">Jan 1 22:59:08 halnas systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (file watch) being skipped.
</span><span style="color:#323232;">Jan 1 22:59:08 halnas systemd[1]: Condition check resulted in Process error reports when automatic reporting is enabled (timer based) being skipped.
</span><span style="color:#323232;">Jan 2 02:23:18 halnas kernel: [12293.792282] gdbus[2809399]: segfault at 7ff71a8272e8 ip 00007ff7186f8045 sp 00007fffd5088de0 error 4 in libgio-2.0.so.0.7200.4[7ff718688000+111000]
</span><span style="color:#323232;">Jan 2 02:23:22 halnas kernel: [12297.315463] unattended-upgr[2810494]: segfault at 7f4c1e8552e8 ip 00007f4c1c726045 sp 00007ffd1b866230 error 4 in libgio-2.0.so.0.7200.4[7f4c1c6b6000+111000]
</span><span style="color:#323232;">Jan 2 03:46:29 halnas kernel: [17284.221594] #PF: error_code(0x0002) - not-present page
</span><span style="color:#323232;">Jan 2 06:09:50 halnas kernel: [25885.115060] unattended-upgr[4109474]: segfault at 7faa356252e8 ip 00007faa334f6045 sp 00007ffefed011a0 error 4 in libgio-2.0.so.0.7200.4[7faa33486000+111000]
</span><span style="color:#323232;">Jan 2 07:07:53 halnas kernel: [29368.241593] unattended-upgr[4109637]: segfault at 7f73f756c2e8 ip 00007f73f543d045 sp 00007ffc61f04ea0 error 4 in libgio-2.0.so.0.7200.4[7f73f53cd000+111000]
</span><span style="color:#323232;">Jan 2 09:12:52 halnas kernel: [36867.632220] pool-fwupdmgr[4109819]: segfault at 7fcf244832e8 ip 00007fcf22354045 sp 00007fcf1dc00770 error 4 in libgio-2.0.so.0.7200.4[7fcf222e4000+111000]
</span><span style="color:#323232;">Jan 2 12:37:50 halnas kernel: [49165.218100] #PF: error_code(0x0002) - not-present page
</span><span style="color:#323232;">Jan 2 19:57:53 halnas kernel: [75568.443218] unattended-upgr[4110958]: segfault at 7fc4cab112e8 ip 00007fc4c89e2045 sp 00007fffb4ae2d90 error 4 in libgio-2.0.so.0.7200.4[7fc4c8972000+111000]
</span><span style="color:#323232;">Jan 3 00:54:51 halnas snapd[1367]: stateengine.go:149: state ensure error: Post "https://api.snapcraft.io/v2/snaps/refresh": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
</span>
Most of yhe subscribed communities seem to be working on your instance. When did you subscribe to permacomputing@lemmy.sdf.org? was it after the upgrade to 0.19.x?
I don’t have practical experience with ZFS, but my understanding is that it uses RAM a lot… if that’s new, it might be worth checking the RAM by booting up memtest (for example) and just ruling that out.
Maybe also worth watching the system with nmon or htop (running in another tmux / screen pane) at the beginning of the next session, then when you think it’s jammed up, see what looks different…
Awesome, thanks for giving some clues. It’s a new build, but I didn’t focus hugely on RAM, I think it’s only 32GB. I’ll try this out.
Edit: I did some reading about L2ARC, so pending some of these tests, I’m planning to get up to 64gb ram and then extend with an l2arc SSD, assuming no other hardware errors.
Just another thought… Maybe just format the drives as a massive EXT4 JBOD (just for a temp test) and copy the data again - just to see if ZFS is the problem… maybe it’s something else altogether? Maybe - and I hope not - the USB source drive is failing after long reads?
I believe there’s another issue. ZFS has been using nearly all RAM (which is fine, I only need RAM for system and ZFS anyway, there’s nothing else running on this box), but I was pretty convinced while I was looking that I don’t have dedup turned on. Thanks for your suggestions and links!
selfhosted
Top
This magazine is from a federated server and may be incomplete. Browse more on the original instance.