wppnx maintenance

Incident Report for vulpineawoo

Resolved

looks like everything is running fine now

Posted Mar 28, 2022 - 20:41 EDT

Update

the tor hidden service is now back online

Posted Mar 28, 2022 - 20:36 EDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 28, 2022 - 20:19 EDT

Update

replacement hardware has been secured, everything should be back once we deploy the software

Posted Mar 28, 2022 - 17:05 EDT

Identified

womp womp. wppnx's storage failed fsck.

luckily the services db was fully recoverable, so there will be no gap since the last backup once we get it up and running again

Posted Mar 27, 2022 - 18:32 EDT

Investigating

the maintenance is done, but now it appears that wppnx has had an unrelated hardware failure

Posted Mar 27, 2022 - 17:37 EDT

Identified

we are maintenancing

Posted Mar 27, 2022 - 12:35 EDT

This incident affected: irc servers (irc.wppnx.pii.at) and atheme services.