Author: Adrian Alexdre (FurrIX, Network Operations Center)

Upkeep avali for FurrIX, typically hides in the NOC

ToS Updated

With FurrIX being spun off into its own ordeal, and Ty Dwagon taking over MFN, our terms of service has been update to carve out the FurrIX related provisions. Also, FurrIX has generated their own terms of service based off ours, using the relevant pieces for a networking provider.

Decoupling FurrIX from MFN

It is going to take a bit of time to do so, but our volunteers are working on the steps needed to spin FurrIX off into its own project. Doing so will give that project the ability to focus on just networking and being a transit/connectivity provider and will transition MFN into a game services, name server and VPS hosting provider.

FurrIX will be getting its own NMP, TOS and goals as part of this move. That project will be responsible entirely for all of the shared routing gear, network design and transit services. MFN will retain control of the IP allocations but will essentially delegate full network administration to FurrIX for the next two years of operations with some conditions on use and expectations regarding service to Marbled Fennec Networks.

As part of this, Adrian will be giving her position on the board for MFN to Ty Dwagon as they will taking control of Marbled Fennec Networks. Adrian will be moving into a lead position with FurrIX to only handle networking and transit going forward.

SSL Certs Updated

The certs for NS1, NS2 and the web server have been updated. Will update the mail server in a bit.

Partial Outages Resolved

For the past couple of days there have been partial outages occurring in the form of network endpoint failures and routing ceasing to function for five to ten minute intervals, along with high latency. Unfortunately, this had occurred during my travels for work the past few days and neither I nor the other tech that helps with the NOC had time to triage and figure out what had happened.

After finally getting a moment to take a look at the network, I found that three of our routers were maxing out their allotted CPU limits and were failing to route traffic. What I discovered during a closer look at the affected routers is that during the last firmware upgrade, our logging settings got reset to defaults which means that all routers on our network were starting to experience an issue with running out of disk space, causing them to malfunction in a way that would max out their other resources. This also causes the GUI to go offline, meaning nothing could be done to fix this until I was able to tunnel in and pop a shell.

The fix to this issue was to remote into to each router and delete the offending log data and then reboot the routers one by one. Once that was handled, log limits were reconfigured in the GUI and services were brought back online and tested to ensure that all parts of the network are functional again. Furthermore, alert rules we placed into the NMS to fire off both email (SMS) and Discord notifications should any router’s disk become over 65% used. This should allow us to catch any future issues like this before they affect the network’s usability.