Symfony Profiler

{
    "@context": [
        "https:\/\/join-lemmy.org\/context.json",
        "https:\/\/www.w3.org\/ns\/activitystreams"
    ],
    "actor": "https:\/\/lemmy.world\/c\/selfhosted",
    "to": [
        "https:\/\/www.w3.org\/ns\/activitystreams#Public"
    ],
    "object": {
        "id": "https:\/\/lemmy.ca\/activities\/update\/ac8ea8ad-5dbd-4406-b922-113f661d0b82",
        "actor": "https:\/\/lemmy.ca\/u\/avidamoeba",
        "@context": [
            "https:\/\/join-lemmy.org\/context.json",
            "https:\/\/www.w3.org\/ns\/activitystreams"
        ],
        "to": [
            "https:\/\/www.w3.org\/ns\/activitystreams#Public"
        ],
        "object": {
            "type": "Note",
            "id": "https:\/\/lemmy.ca\/comment\/16817661",
            "attributedTo": "https:\/\/lemmy.ca\/u\/avidamoeba",
            "to": [
                "https:\/\/www.w3.org\/ns\/activitystreams#Public"
            ],
            "cc": [
                "https:\/\/lemmy.world\/c\/selfhosted",
                "https:\/\/lemmy.world\/u\/ZeDoTelhado"
            ],
            "content": "<p>That\u2019s really weird. I\u2019ve been using it for mobile-desktop-server-offsite sync for many years, with transfer sizes over 15TB, over WiFi, cellular, cable, fiber. I\u2019ve never seen data corruption. Conflicts, sometimes. Permission issues, sometimes. Wiping something accidentally, sometimes. It\u2019s even more weird because Syncthing performs computes hash values for the files it manages. I don\u2019t know if it performs hash validation after copying remotely but if not, it can be forced manually which would tell you what\u2019s fucked and be pulled from the source, if it still exists.<\/p>\n<p>Nevermind, <a href=\"https:\/\/docs.syncthing.net\/users\/syncing.html\" rel=\"nofollow\">it verifies the result:<\/a><\/p>\n<blockquote>\n<p>When a block is copied or received from another device, its SHA256 hash is computed and compared with the expected value. If it matches the block is written to a temporary copy of the file, otherwise it is discarded and Syncthing tries to find another source for the block.<\/p>\n<\/blockquote>\n<p>According to this, if you have data corruption it can only occur between copying\/moving a temporary file on your destination to another directory, or it could occur on the source itself. Both of those scenarios are a cause of concern and would likely persist with any utility. Moving or copying a file from one location to another on a sane machine should not corrupt it. If I were you I\u2019d ensure my server doesn\u2019t eat bits. If not the storage media, it could be bit rot, or bad RAM.<\/p>\n<p>Just in case everything seems fine, let me tell you what I dealt with. I had a Ryzen 5950X machine with 32GB of RAM. It worked well since inception with no signs of RAM or data corruption issues. I test every new machine with Memtest86+. At some point I migrated the storage from Ext4 on LVMRAID to ZFS. All good. Then I wrote an alarm for Prometheus to tell me if there\u2019s any issues in ZFS. A week later I get an email about a ZFS error. I check the system - says checksum errors, data has been corrected, applications unaffected, run a scrub to clear. I ran a scrub. A few more checksum errors found, all corrected, we\u2019re clean now. There was a strong solar storm around that time, probably that. A couple of weeks later I get another email. Same symptoms, same procedure. No solar storm. Shit. Memtest86+ - pass. Hm. A couple of weeks later I get another. Same thing. Memtest again - nothing. This went on for several months. Meanwhile the off-site backup sees nothing like that. While running Memtest on another machine I noticed that the test passes following the first took longer than the first, a lot longer. I thought something might be wrong with that machine. Dug into it, got into Memtest\u2019s source code and discovered that the first pass is shorter on purpose so that it quickly flags obviously bad RAM. Apparently if you want to detect less obvious issues, you have to run multiple passes. OK. Memtest the main server again, pass 1: OK, pass 2: OK, pass 3: OK, pass 4: FAIL. FUCK. Memtest each stick separately for 4 passes: OK. Memtest 2 at a time: OK. Memtest all 4: FAIL. Alright, now we know why ZFS keeps finding checksum errors. Long story short, this machine could not run this RAM in 4-DIMM config. Replaced it with another RAM that\u2019s rated to run in 4-DIMM config on that processor. No more checksum issues. If I was running the older Ext4-on-LVMRAID storage stack, I would have caught NONE of these and it would have happily corrupted files here and there. In fact it likely did and I have some corruption. Moral of the story - run <em>many<\/em> Memtest passes and use <em>checksumming<\/em> storage stack like ZFS or Btrfs. I strongly recommend ZFS since its stripe RAID works fine unlike Btrfs\u2019es. Start today, even if you\u2019re working with a single disk and add redundancy when you can. Only after change Syncthing for something else if you still somehow get corruption without ZFS\u2019es knowledge. And if ZFS tells you that you have checksum errors, you likely have bad hardware.<\/p>\n",
            "inReplyTo": "https:\/\/lemmy.world\/post\/30402224",
            "mediaType": "text\/html",
            "source": {
                "content": "That's really weird. I've been using it for mobile-desktop-server-offsite sync for many years, with transfer sizes over 15TB, over WiFi, cellular, cable, fiber. I've never seen data corruption. Conflicts, sometimes. Permission issues, sometimes. Wiping something accidentally, sometimes. It's even more weird because Syncthing performs computes hash values for the files it manages. I don't know if it performs hash validation after copying remotely but if not, it can be forced manually which would tell you what's fucked and be pulled from the source, if it still exists.\n\nNevermind, [it verifies the result:](https:\/\/docs.syncthing.net\/users\/syncing.html)\n\n> When a block is copied or received from another device, its SHA256 hash is computed and compared with the expected value. If it matches the block is written to a temporary copy of the file, otherwise it is discarded and Syncthing tries to find another source for the block.\n\nAccording to this, if you have data corruption it can only occur between copying\/moving a temporary file on your destination to another directory, or it could occur on the source itself. Both of those scenarios are a cause of concern and would likely persist with any utility. Moving or copying a file from one location to another on a sane machine should not corrupt it. If I were you I'd ensure my server doesn't eat bits. If not the storage media, it could be bit rot, or bad RAM.\n\nJust in case everything seems fine, let me tell you what I dealt with. I had a Ryzen 5950X machine with 32GB of RAM. It worked well since inception with no signs of RAM or data corruption issues. I test every new machine with Memtest86+. At some point I migrated the storage from Ext4 on LVMRAID to ZFS. All good. Then I wrote an alarm for Prometheus to tell me if there's any issues in ZFS. A week later I get an email about a ZFS error. I check the system - says checksum errors, data has been corrected, applications unaffected, run a scrub to clear. I ran a scrub. A few more checksum errors found, all corrected, we're clean now. There was a strong solar storm around that time, probably that. A couple of weeks later I get another email. Same symptoms, same procedure. No solar storm. Shit. Memtest86+ - pass. Hm. A couple of weeks later I get another. Same thing. Memtest again - nothing. This went on for several months. Meanwhile the off-site backup sees nothing like that. While running Memtest on another machine I noticed that the test passes following the first took longer than the first, a lot longer. I thought something might be wrong with that machine. Dug into it, got into Memtest's source code and discovered that the first pass is shorter on purpose so that it quickly flags obviously bad RAM. Apparently if you want to detect less obvious issues, you have to run multiple passes. OK. Memtest the main server again, pass 1: OK, pass 2: OK, pass 3: OK, pass 4: FAIL. FUCK. Memtest each stick separately for 4 passes: OK. Memtest 2 at a time: OK. Memtest all 4: FAIL. Alright, now we know why ZFS keeps finding checksum errors. Long story short, this machine could not run this RAM in 4-DIMM config. Replaced it with another RAM that's rated to run in 4-DIMM config on that processor. No more checksum issues. If I was running the older Ext4-on-LVMRAID storage stack, I would have caught NONE of these and it would have happily corrupted files here and there. In fact it likely did and I have some corruption. Moral of the story - run *many* Memtest passes and use *checksumming* storage stack like ZFS or Btrfs. I strongly recommend ZFS since its stripe RAID works fine unlike Btrfs'es. Start today, even if you're working with a single disk and add redundancy when you can. Only after change Syncthing for something else if you still somehow get corruption without ZFS'es knowledge. And if ZFS tells you that you have checksum errors, you likely have bad hardware.",
                "mediaType": "text\/markdown"
            },
            "published": "2025-05-28T20:29:39.335302Z",
            "updated": "2025-05-28T20:56:05.287078Z",
            "tag": [
                {
                    "href": "https:\/\/lemmy.world\/u\/ZeDoTelhado",
                    "name": "@ZeDoTelhado@lemmy.world",
                    "type": "Mention"
                }
            ],
            "distinguished": false,
            "language": {
                "identifier": "en",
                "name": "English"
            },
            "audience": "https:\/\/lemmy.world\/c\/selfhosted",
            "attachment": []
        },
        "cc": [
            "https:\/\/lemmy.world\/c\/selfhosted",
            "https:\/\/lemmy.world\/u\/ZeDoTelhado"
        ],
        "tag": [
            {
                "href": "https:\/\/lemmy.world\/u\/ZeDoTelhado",
                "name": "@ZeDoTelhado@lemmy.world",
                "type": "Mention"
            }
        ],
        "type": "Update",
        "audience": "https:\/\/lemmy.world\/c\/selfhosted"
    },
    "cc": [
        "https:\/\/lemmy.world\/c\/selfhosted\/followers"
    ],
    "type": "Announce",
    "id": "https:\/\/lemmy.world\/activities\/announce\/update\/3e19197c-7f1d-4c5d-a58f-338cb39261a8"
}

{"@context":["https://join-lemmy.org/context.json","https://www.w3.org/ns/activitystreams"],"actor":"https://lemmy.world/c/selfhosted","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"id":"https://lemmy.ca/activities/update/ac8ea8ad-5dbd-4406-b922-113f661d0b82","actor":"https://lemmy.ca/u/avidamoeba","@context":["https://join-lemmy.org/context.json","https://www.w3.org/ns/activitystreams"],"to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"type":"Note","id":"https://lemmy.ca/comment/16817661","attributedTo":"https://lemmy.ca/u/avidamoeba","to":["https://www.w3.org/ns/activitystreams#Public"],"cc":["https://lemmy.world/c/selfhosted","https://lemmy.world/u/ZeDoTelhado"],"content":"<p>That’s really weird. I’ve been using it for mobile-desktop-server-offsite sync for many years, with transfer sizes over 15TB, over WiFi, cellular, cable, fiber. I’ve never seen data corruption. Conflicts, sometimes. Permission issues, sometimes. Wiping something accidentally, sometimes. It’s even more weird because Syncthing performs computes hash values for the files it manages. I don’t know if it performs hash validation after copying remotely but if not, it can be forced manually which would tell you what’s fucked and be pulled from the source, if it still exists.</p>\n<p>Nevermind, <a href=\"https://docs.syncthing.net/users/syncing.html\" rel=\"nofollow\">it verifies the result:</a></p>\n<blockquote>\n<p>When a block is copied or received from another device, its SHA256 hash is computed and compared with the expected value. If it matches the block is written to a temporary copy of the file, otherwise it is discarded and Syncthing tries to find another source for the block.</p>\n</blockquote>\n<p>According to this, if you have data corruption it can only occur between copying/moving a temporary file on your destination to another directory, or it could occur on the source itself. Both of those scenarios are a cause of concern and would likely persist with any utility. Moving or copying a file from one location to another on a sane machine should not corrupt it. If I were you I’d ensure my server doesn’t eat bits. If not the storage media, it could be bit rot, or bad RAM.</p>\n<p>Just in case everything seems fine, let me tell you what I dealt with. I had a Ryzen 5950X machine with 32GB of RAM. It worked well since inception with no signs of RAM or data corruption issues. I test every new machine with Memtest86+. At some point I migrated the storage from Ext4 on LVMRAID to ZFS. All good. Then I wrote an alarm for Prometheus to tell me if there’s any issues in ZFS. A week later I get an email about a ZFS error. I check the system - says checksum errors, data has been corrected, applications unaffected, run a scrub to clear. I ran a scrub. A few more checksum errors found, all corrected, we’re clean now. There was a strong solar storm around that time, probably that. A couple of weeks later I get another email. Same symptoms, same procedure. No solar storm. Shit. Memtest86+ - pass. Hm. A couple of weeks later I get another. Same thing. Memtest again - nothing. This went on for several months. Meanwhile the off-site backup sees nothing like that. While running Memtest on another machine I noticed that the test passes following the first took longer than the first, a lot longer. I thought something might be wrong with that machine. Dug into it, got into Memtest’s source code and discovered that the first pass is shorter on purpose so that it quickly flags obviously bad RAM. Apparently if you want to detect less obvious issues, you have to run multiple passes. OK. Memtest the main server again, pass 1: OK, pass 2: OK, pass 3: OK, pass 4: FAIL. FUCK. Memtest each stick separately for 4 passes: OK. Memtest 2 at a time: OK. Memtest all 4: FAIL. Alright, now we know why ZFS keeps finding checksum errors. Long story short, this machine could not run this RAM in 4-DIMM config. Replaced it with another RAM that’s rated to run in 4-DIMM config on that processor. No more checksum issues. If I was running the older Ext4-on-LVMRAID storage stack, I would have caught NONE of these and it would have happily corrupted files here and there. In fact it likely did and I have some corruption. Moral of the story - run <em>many</em> Memtest passes and use <em>checksumming</em> storage stack like ZFS or Btrfs. I strongly recommend ZFS since its stripe RAID works fine unlike Btrfs’es. Start today, even if you’re working with a single disk and add redundancy when you can. Only after change Syncthing for something else if you still somehow get corruption without ZFS’es knowledge. And if ZFS tells you that you have checksum errors, you likely have bad hardware.</p>\n","inReplyTo":"https://lemmy.world/post/30402224","mediaType":"text/html","source":{"content":"That's really weird. I've been using it for mobile-desktop-server-offsite sync for many years, with transfer sizes over 15TB, over WiFi, cellular, cable, fiber. I've never seen data corruption. Conflicts, sometimes. Permission issues, sometimes. Wiping something accidentally, sometimes. It's even more weird because Syncthing performs computes hash values for the files it manages. I don't know if it performs hash validation after copying remotely but if not, it can be forced manually which would tell you what's fucked and be pulled from the source, if it still exists.\n\nNevermind, [it verifies the result:](https://docs.syncthing.net/users/syncing.html)\n\n> When a block is copied or received from another device, its SHA256 hash is computed and compared with the expected value. If it matches the block is written to a temporary copy of the file, otherwise it is discarded and Syncthing tries to find another source for the block.\n\nAccording to this, if you have data corruption it can only occur between copying/moving a temporary file on your destination to another directory, or it could occur on the source itself. Both of those scenarios are a cause of concern and would likely persist with any utility. Moving or copying a file from one location to another on a sane machine should not corrupt it. If I were you I'd ensure my server doesn't eat bits. If not the storage media, it could be bit rot, or bad RAM.\n\nJust in case everything seems fine, let me tell you what I dealt with. I had a Ryzen 5950X machine with 32GB of RAM. It worked well since inception with no signs of RAM or data corruption issues. I test every new machine with Memtest86+. At some point I migrated the storage from Ext4 on LVMRAID to ZFS. All good. Then I wrote an alarm for Prometheus to tell me if there's any issues in ZFS. A week later I get an email about a ZFS error. I check the system - says checksum errors, data has been corrected, applications unaffected, run a scrub to clear. I ran a scrub. A few more checksum errors found, all corrected, we're clean now. There was a strong solar storm around that time, probably that. A couple of weeks later I get another email. Same symptoms, same procedure. No solar storm. Shit. Memtest86+ - pass. Hm. A couple of weeks later I get another. Same thing. Memtest again - nothing. This went on for several months. Meanwhile the off-site backup sees nothing like that. While running Memtest on another machine I noticed that the test passes following the first took longer than the first, a lot longer. I thought something might be wrong with that machine. Dug into it, got into Memtest's source code and discovered that the first pass is shorter on purpose so that it quickly flags obviously bad RAM. Apparently if you want to detect less obvious issues, you have to run multiple passes. OK. Memtest the main server again, pass 1: OK, pass 2: OK, pass 3: OK, pass 4: FAIL. FUCK. Memtest each stick separately for 4 passes: OK. Memtest 2 at a time: OK. Memtest all 4: FAIL. Alright, now we know why ZFS keeps finding checksum errors. Long story short, this machine could not run this RAM in 4-DIMM config. Replaced it with another RAM that's rated to run in 4-DIMM config on that processor. No more checksum issues. If I was running the older Ext4-on-LVMRAID storage stack, I would have caught NONE of these and it would have happily corrupted files here and there. In fact it likely did and I have some corruption. Moral of the story - run *many* Memtest passes and use *checksumming* storage stack like ZFS or Btrfs. I strongly recommend ZFS since its stripe RAID works fine unlike Btrfs'es. Start today, even if you're working with a single disk and add redundancy when you can. Only after change Syncthing for something else if you still somehow get corruption without ZFS'es knowledge. And if ZFS tells you that you have checksum errors, you likely have bad hardware.","mediaType":"text/markdown"},"published":"2025-05-28T20:29:39.335302Z","updated":"2025-05-28T20:56:05.287078Z","tag":[{"href":"https://lemmy.world/u/ZeDoTelhado","name":"@ZeDoTelhado@lemmy.world","type":"Mention"}],"distinguished":false,"language":{"identifier":"en","name":"English"},"audience":"https://lemmy.world/c/selfhosted","attachment":[]},"cc":["https://lemmy.world/c/selfhosted","https://lemmy.world/u/ZeDoTelhado"],"tag":[{"href":"https://lemmy.world/u/ZeDoTelhado","name":"@ZeDoTelhado@lemmy.world","type":"Mention"}],"type":"Update","audience":"https://lemmy.world/c/selfhosted"},"cc":["https://lemmy.world/c/selfhosted/followers"],"type":"Announce","id":"https://lemmy.world/activities/announce/update/3e19197c-7f1d-4c5d-a58f-338cb39261a8"}