poVoq here. I am starting to think I am cursed, because everything works fine (or is at least fixable with a remote server restart) until the second I go on a longer business trip and can't easily access the hardware.

Remote access is mostly lost, but I was able to triangulate the issue and of course it is something I had on my list of potential issues, but as a low priority item 🫠 At least it isn't the same problem again as in 2025.

What exactly triggered the issue is unclear though. I was still able to access the reverse-proxy server yesterday (shortly before it went down) and tried to do some updates for that php security problem in a VM that runs on the same machine. I thought I had those covered, but there is a slight chance that it is related and someone chained another exploit to escape the VM.

Anyways, it looks like the realistic / not overly optimistic scenario is that the server will stay down until christmas. I did have someone who could restart the servers on site until this summer, but they had to relocate to mainland Portugal for job reasons (tourism sector) and so they will also only return for christmas.

So yeah, not great at all, and I will have to think more seriously about how compatible this kind of self-hosting is with my current job, which limits my access to the servers for a few weeks at least once a year. It was easier during the pandemic times when I started all this, as deployments were shorter and more seldom back then.

poVoq here. I am starting to think I am cursed, because everything works fine (or is at least fixable with a remote server restart) until the second I go on a longer business trip and can't easily access the hardware.

Remote access is mostly lost, but I was able to triangulate the issue and of course it is something I had on my list of potential issues, but as a low priority item 🫠 At least it isn't the same problem again as in 2025.

What exactly triggered the issue is unclear though. I was still able to access the reverse-proxy server yesterday (shortly before it went down) and tried to do some updates for that php security problem in a VM that runs on the same machine. I thought I had those covered, but there is a slight chance that it is related and someone chained another exploit to escape the VM.

Anyways, it looks like the realistic / not overly optimistic scenario is that the server will stay down until christmas. I did have someone who could restart the servers on site until this summer, but they had to relocate to mainland Portugal for job reasons (tourism sector) and so they will also only return for christmas.

So yeah, not great at all, and I will have to think more seriously about how compatible this kind of self-hosting is with my current job, which limits my access to the servers for a few weeks at least once a year. It was easier during the pandemic times when I started all this, as deployments were shorter and more seldom back then.

Thank you for your service, and thanks for the update. Would an IP KVM help here? Know some of them can cycle power remotely, and they're getting a lot cheaper

I bought a chinese PiKVM copy last time it happened, but ran into issues with the usb otg connection which I procastrinated on to fix, so it currently only shares the screen but not the input. But I also connected it to the main router which is supposed to run the ssh tunnel and reverse-proxy eventually but right now those run on a seperare server and that one went down 🫠

So yeah, I am victim of my own half finished stuff as well as security hardening that now locked myself out pretty much...

poVoq here. I am starting to think I am cursed, because everything works fine (or is at least fixable with a remote server restart) until the second I go on a longer business trip and can't easily access the hardware.

Remote access is mostly lost, but I was able to triangulate the issue and of course it is something I had on my list of potential issues, but as a low priority item 🫠 At least it isn't the same problem again as in 2025.

What exactly triggered the issue is unclear though. I was still able to access the reverse-proxy server yesterday (shortly before it went down) and tried to do some updates for that php security problem in a VM that runs on the same machine. I thought I had those covered, but there is a slight chance that it is related and someone chained another exploit to escape the VM.

Anyways, it looks like the realistic / not overly optimistic scenario is that the server will stay down until christmas. I did have someone who could restart the servers on site until this summer, but they had to relocate to mainland Portugal for job reasons (tourism sector) and so they will also only return for christmas.

So yeah, not great at all, and I will have to think more seriously about how compatible this kind of self-hosting is with my current job, which limits my access to the servers for a few weeks at least once a year. It was easier during the pandemic times when I started all this, as deployments were shorter and more seldom back then.

So you're saying this might all be because of AI (which found the security flaw in PHP, which caused the maintenance, which happened sometime around when the server went down). Sounds fair to me.

Seriously, thanks for the update and I know sysadmining can be a stressy job. Take care of yourself.

I don't think it was actually the php issue that took it down, but I don't wan't to rule it out entirely. But I had some issues with spotty remote connection for a while now that that is likely due to LLM scraper load.

poVoq here. I am starting to think I am cursed, because everything works fine (or is at least fixable with a remote server restart) until the second I go on a longer business trip and can't easily access the hardware.

Remote access is mostly lost, but I was able to triangulate the issue and of course it is something I had on my list of potential issues, but as a low priority item 🫠 At least it isn't the same problem again as in 2025.

What exactly triggered the issue is unclear though. I was still able to access the reverse-proxy server yesterday (shortly before it went down) and tried to do some updates for that php security problem in a VM that runs on the same machine. I thought I had those covered, but there is a slight chance that it is related and someone chained another exploit to escape the VM.

Anyways, it looks like the realistic / not overly optimistic scenario is that the server will stay down until christmas. I did have someone who could restart the servers on site until this summer, but they had to relocate to mainland Portugal for job reasons (tourism sector) and so they will also only return for christmas.

So yeah, not great at all, and I will have to think more seriously about how compatible this kind of self-hosting is with my current job, which limits my access to the servers for a few weeks at least once a year. It was easier during the pandemic times when I started all this, as deployments were shorter and more seldom back then.

Bad luck for Ecospheres community