Category: Issues

UPDATE OCT 24TH 2024@1110EST:
It appears that the attack has stopped and we have removed the drop rules for the affected domains. We have also restored our usual name query limits.

UPDATE OCT 22ND 2024@0950EST:
The attack is still ongoing and our team will continue to drop queries for ‘cloudflare.com’ for the time being. We will perform another check around 1500EST.

UPDATE OCT 21ST 2024@2250EST:
It appears that the attack is largely focused on various regions in Brazil. While our team cannot ID the source of the attack, the destination for the requested data is pretty localized to that region. As of this time, the attackers seem to be settling for bogus TXT lookups against cloudflare.com and our team will begin dropping traffic with the hex for that domain until sometime tomorrow morning. This may break things on a temp basis as DNS queries for their domain will not resolve for users hitting our servers.

—
We are seeing what looks like a DNS attack at the moment, appears to have started around 1345EST until it died off near 1420EST, and then picked up again around 1700EST and it still ongoing at the time of posting. The majority of the request are TXT records for only a handful of domains. Our team will be keeping an eye on this.

We have already stepped in to significantly lower the responses per second allowed and widen the tracking bitmask to /24 for v4 and /58 for v6. These lower limits on responses will remain in effect until six hours after the attack ends in order to limit our impact without dropping our service entirely.

While the RPS is not super high, it is out of the norm for our name servers and we are reacting to it in order to limit the outgoing traffic amount and keep our name servers accessible.

There were some upstream issues that resulted in us loosing power a few times. Our team will be checking disk images and slowly getting connectivity back online, please have a bit of patience with us while we work on this.

–updates–

Core and Catos routers were restored from local images and should be alive again, we are still working on Ikus to see if it was damaged or not.

Ikus and CX were also damaged, Ikus has been restored from local. CX might need to be rolled back.

IPv6 Edge router is fine and is working as should. Web, email, NMS and Minecraft all seem to be okay as well.

So, when we connect project members to our networks, we enroll the interface on our side into our NMS for accounting. Every project member or end user can make use of up to 600GB of transit each month before being placed into a lower QoS bucket. Or so that it how things are supposed to work.

Our NMS applied its updates on its own daily not needing any real oversight from our team until about four days ago. An update failed to apply correctly which resulted in a complete failure of accounting and a mix up of interface names not matching with their respective subnets. So, in a nutshell, there was zero traffic accounting for about four days because none of the data that was accounted was correct.

This has been sorted out as of this morning and everyone’s monthly usage was reset.