Hey guys, It’s been a while! The last few weeks have been pretty rough on me, and I’d like to give an update to all the readers out there who might be curious of what was going on! Here goes! We have one main topic on today’s MSL- Site Downtime: What caused it, how it affected us and you, what happened after?
“Where are you, FFXIVGuild!?”
What happened, in a nutshell
Basically, an automated virus that finds weak-spots in websites or web hosts and injects funky stuff into our code. The type of ‘funky stuff’ that happened to us specifically were either: Redirecting you when you visit or Spammy blocks of text with links appearing above our guides.
How did it affect us, and you?
It is extremely unlikely that YOU or your local PC got infected when visiting us during the time our site was up and infected. The viruses actually target is the site itself – Redirects their sites to gain more $$$, and the spammy links appear for the same reason. In hopes you would click on it and go to “god knows where”. I’m sorry but any actions you took in Lalafell pr0n pages are not in our scope of responsibility…
What did we do once we found out infection was imminent?
It was so damn hard to track the issues down because they were difficult to recreate! Not all pages were infected (that’s TOO obvious), and the infected ones didn’t all have the same problem (There were 2 virus types). Ultimately, we also had to track how they got in. The decision was obvious… It was heartbreaking but…
The site was ultimately shutdown for YOUR protection
Many thanks to ALL PEOPLE who helped us out on FB/elsewhere who sent us screenshots/examples/links to the whatever they found was infected! It helped a ton. Anyways – We decided it’s time to fix many underlying back-end issues too, and make security as tight as as Lala-sphincter.
How the admin ‘team’ responded
Let me preface this by saying we’re just two guys. I write 99% of the content, and about half the back-end stuff. Maruko helps me out with e-mails and other communication lines, while helping 50% of the back-end too. I’d like to remind the public two things:
1) We’re not a bunch of suits. We’re not even a “bunch”.
2) We’re 2 brothers who made a fansite.
So yes obviously that comes with a ton of limitations (more on that in the next journal!). I am quite an experienced “online guy” that’s done a ton of work online, managing a large website is a great learning experience. Sadly, some things are only learned the hard way. Security, in this case.
Maruko “The Unseen” San
I’d like to remind the viewers, and give a shout-out to the ultimate big brother. He’s really busy, but sacrifices a lot of his free time to help keep FFXIVGuild running! I dont know how, maybe via comment on FB, or comment here – let us all give thanks to Maruko San! 100% sure, I cannot have made it past this annoying roadblock without him.
The Virus Minigame
Thing were looking’ pretty bad, I wont lie. So me and Maruko began our two-fold quest to restore the site. After some time, it became obvious this was looking pretty similar to something…
We decided to think of this security threat as a raid boss.
Yeaaaahp. So our battle had three phases. (I could write a long post about how we did this, but you guys might get bored!)
Phase 1: Cleaning
Phase 2: Sealing vulnerabilities
Phase 3: Testing (repeat until enrage)
Our first problem was we COULD NOT RESTORE A BACKUP. Why? It wouldn’t do anything! The vulerabilities would still exist! So it became obvious that before the site could go up, we had to clean and patch as many known weak-spots we had. This was NOT as easy as it sounds.
So the site comes back up six days later… and the real war began…
This is how our process worked. First we would check our logs for any suspicious activity. Then we would take steps to put up as many walls for any similar TYPE of attack, or outright blocking them. We furiously researched and applied best practices of securing high-traffic websites.
When we recieved more messages from people saying our website was hacked, our investigations turned up they were loading CACHED versions of the site! To think even a thing as this might happen! Some cached files were still stuck in our “CDN”!
For a few days, I would wake up every 2 hours and check our server logs. “Is there anything I can do better?”, “Is there a better way to secure the viewers?”, more and more… every implementation I felt more and more positive!
To beef up the security, this ‘raid boss mentality’ really worked! We were possessed! Before long, under this mindset, Maruko and I were in a frenzy! Like any raid, each time we shut the door on another type of vulnerability – we celebrated!
Finally, I can say… at least for now – we have won. We also fixed a lot of underlying site issues. As an great ‘side-effect’, our site loads much faster!
Summary
This wave of attacks apparently affected to A LOT of small to medium website owners. Hacking HOSTS isn’t actually as uncommon as it sounds! Ultimately, FFXIVGuild will have to move servers soon… we’ve GROSSLY OUTGROWN our type of hosting, and this will come with many changes in the near future. Which I’ll be discussing in the next journal!
To everyone who reached out saying “I need you guys back!”, you the real MVP
You know, running a fansite ain’t easy. But seriously thinking of those of you who took time out of their day to message us, and help us by showing screenshots and whatnot… I sincerely thank you! You were my inspiration to SMASH this threat!
Anyways, I’ve always wanted to have a ‘periodical’ type of post in FFXIVGuild, and I think having a personal blog regarding in-game or in-site matters would be the best. I’ve tried in the past as you may have noticed, but I think this is it. Mahiko Sans Log is here to stay! It’s a great way for me to communicate with you guys, which I think I want to do more of! And helps me let off some steam.
I’m guessing the next issue will contain what’s in store for FFXIVGuilds future – Please look forward to it(tm). Before all of this BS happened, I needed to refine all BiS lists… So I guess it’s back to business! I can finally work on FFXIVGuild in peace… Mahiko San signing out!
Keep up the good work!
Ganbatte Maruko duo brothers san =D
I am really impressed. Good job you two :)
Thank you both for all your hard work!