Jump to content

Announcement: Data Loss & Forums Rollback


Recommended Posts

Good evening, everybody,

As you've no doubt noticed, the forums have lost several days' worth of data. The data loss was a result of error earlier this afternoon when I was troubleshooting a critical failure in our blogging software. I ran a recovery procedure to repair the blog, and erroneous code in my process deleted forums' database data. I later consulted Nayleen about our options and what went wrong, and unfortunately the only option available to us was to roll back to our last uncorrupted backup (which was last weekend).

The data loss is my fault. I wrote my recovery programs and did not understand that two lines of code mixed-up database directories and -- ultimately -- would destroy the forums' database.

I take data loss very seriously and I've tried to be responsible: I backup the site every few days and store the files in at least three places: the server, my personal cloud, and in an offline, enterprise-grade hard drive I obtained for this purpose. Today's data loss was not due to negligence in making backups: I had backups made as recently as yesterday; the problem is that the most-recent backups were corrupted in my recovery process today. I ultimately had to go back and pull a backup from my offline HDD to restore the forums. 

I want to apologize for the frustration many of you are feeling. Providing this community with a safe, reliable online home is something I take very seriously, and I fully understand that the loss of data amounts to an astounding amount of work. As soon as I realized the breadth of the problem I cleared my schedule and got to work. 

The only thing I can control is how things will change going forward. I've gone through all my recovery documents and programs and rigorously checked and fixed mis-directed database queries. As soon as I identify other helpful SOP changes, I will make them.

Lastly -- I realize that this is yet another example of the need for technical assistance. I'm working on this actively, getting bids, and I'll post more about this soon in my community updates.

Thank you for your patience. I'm committed to making Fuwanovel the reliable community we all want it to be, and I'm deeply sorry for this incident.

    - Tay

Link to post
Share on other sites

I would like to make it 100% clear. When I liked Tay's post, I'm indicating my happiness, and delight that several days of data (and likes ho ho) was lost, not my happiness that Tay spent so long trying to recover everybody's likes :) 

Thank you for the hard work Tay, we all appreciate it. My sympathies to everybody who's lost so many, many likes during this terrible event :) 

Link to post
Share on other sites
21 hours ago, Tay said:

Good evening, everybody,

As you've no doubt noticed, the forums have lost several days' worth of data. The data loss was a result of error earlier this afternoon when I was troubleshooting a critical failure in our blogging software. I ran a recovery procedure to repair the blog, and erroneous code in my process deleted forums' database data. I later consulted Nayleen about our options and what went wrong, and unfortunately the only option available to us was to roll back to our last uncorrupted backup (which was last weekend).

The data loss is my fault. I wrote my recovery programs and did not understand that two lines of code mixed-up database directories and -- ultimately -- would destroy the forums' database.

I take data loss very seriously and I've tried to be responsible: I backup the site every few days and store the files in at least three places: the server, my personal cloud, and in an offline, enterprise-grade hard drive I obtained for this purpose. Today's data loss was not due to negligence in making backups: I had backups made as recently as yesterday; the problem is that the most-recent backups were corrupted in my recovery process today. I ultimately had to go back and pull a backup from my offline HDD to restore the forums. 

I want to apologize for the frustration many of you are feeling. Providing this community with a safe, reliable online home is something I take very seriously, and I fully understand that the loss of data amounts to an astounding amount of work. As soon as I realized the breadth of the problem I cleared my schedule and got to work. 

The only thing I can control is how things will change going forward. I've gone through all my recovery documents and programs and rigorously checked and fixed mis-directed database queries. As soon as I identify other helpful SOP changes, I will make them.

Lastly -- I realize that this is yet another example of the need for technical assistance. I'm working on this actively, getting bids, and I'll post more about this soon in my community updates.

Thank you for your patience. I'm committed to making Fuwanovel the reliable community we all want it to be, and I'm deeply sorry for this incident.

    - Tay

I never make serious threads. I made one literally five minutes before the backup. This is unbelievable...

On the other hand, you can say things are peaceful when you can safely claim that the biggest danger to the site are its backups.

Link to post
Share on other sites

I don't understand how do you keep using the same forum system after having this amount of serious problems, IPS has showed that it is a crappy forum with a really crappy support and really crappy to do anything from upgrading to installing a new theme (with a really really lack of hacks aka the spoiler system is just crap), why don't you just migrate into another system already? like vbulletin or xen. Having constantly data lost, losing the main theme, updating and then having bigger issues than before doing the update etc those are not acceptable when you are paying for something. And yes I'm saying this because I was an admin of 2 very big forums in the past. (I did offer to give you guys the vbulletin suit I own for free if money was an issue)

Link to post
Share on other sites

With regards to the backup situation, whats tools are being used for this? Would be possible to implement an automated backup solution? At work I use Backup Exec for physical servers and Veeam for virtual. (Veeam now offers a physical version too which is stripped down but still excellent). Might be able to get free or NFR versions.

Link to post
Share on other sites
1 hour ago, Deep Blue said:

I don't understand how do you keep using the same forum system after having this amount of serious problems, IPS has showed that it is a crappy forum with a really crappy support and really crappy to do anything from upgrading to installing a new theme (with a really really lack of hacks aka the spoiler system is just crap), why don't you just migrate into another system already? like vbulletin or xen. Having constantly data lost, losing the main theme, updating and then having bigger issues than before doing the update etc those are not acceptable when you are paying for something. And yes I'm saying this because I was an admin of 2 very big forums in the past. (I did offer to give you guys the vbulletin suit I own for free if money was an issue)

If memory serves, we did try to migrate the forums to another software at one point. It was a disaster. 

Link to post
Share on other sites
15 minutes ago, solidbatman said:

If memory serves, we did try to migrate the forums to another software at one point. It was a disaster. 

I migrated from phpbb to smf to vbulletin free(cracked version which it will be hacked by someone eventually no matter what) to vbulletin paid and yes it was a pain but it is doable specially if you know what you are doing, many things will be lost that is a fact but that's the price you need to pay for moving into a better forum system with a better support, more stable and robust.
In the long term is the best choice you can make. Nothing can be worst than having than what is happening now, having a forum that needs constantly to use old back ups because of losing post is the worst problem a forum can have.
Edit: vbulletin even have a tool to do the conversion, again things will be lost in the process (not post) but imo is worth it in the long term.

Link to post
Share on other sites

I lost some valuable posts harassing TexasDice for dropping the greatest VN of all time, but Tay, if you could just harass him for me a bit, I'd call it square.

In all seriousness: Shit happens, and the important thing is owning it and taking the right steps moving forward, which you're clearly currently doing. Thanks for all the hard work.

Link to post
Share on other sites
7 hours ago, Deep Blue said:

I don't understand how do you keep using the same forum system after having this amount of serious problems, IPS has showed that it is a crappy forum with a really crappy support and really crappy to do anything from upgrading to installing a new theme (with a really really lack of hacks aka the spoiler system is just crap), why don't you just migrate into another system already? like vbulletin or xen. Having constantly data lost, losing the main theme, updating and then having bigger issues than before doing the update etc those are not acceptable when you are paying for something. And yes I'm saying this because I was an admin of 2 very big forums in the past. (I did offer to give you guys the vbulletin suit I own for free if money was an issue)

From what I understand this wasn't an IPB problem.  Tay and others do a lot of customization, and one of his custom scripts broke the database.  We're always nagging them to do these customizations, so such catastrophic failures are a result of straying from the beaten path.

Forum dataloss actually isn't that uncommon.  MangaGamer lost its entire forum post history for several years.  They treated it as not a big deal, and they're a "professional" company and not just a fansite.  But well, problems like this are why I wouldn't host anything super-important here...

Link to post
Share on other sites
3 hours ago, sanahtlig said:

From what I understand this wasn't an IPB problem.  Tay and others do a lot of customization, and one of his custom scripts broke the database.  We're always nagging them to do these customizations, so such catastrophic failures are a result of straying from the beaten path.

Forum dataloss actually isn't that uncommon.  MangaGamer lost its entire forum post history for several years.  They treated it as not a big deal, and they're a "professional" company and not just a fansite.  But well, problems like this are why I wouldn't host anything super-important here...

for once if the forum doesnt provide them with a method of dumping the db or even the entire forum in some easy way to do regular back ups then that's a huge issue right there (one that it seems common here in fuwa) 
second the only thing they seem to touch is the blog because there isnt much to do besides the forum and the blog which is part of the forum itself... this blog http://blog.fuwanovel.net is independent of this forum or it should be because it's not sharing anything with this db so if it is then that's a mistake.
This forum system sucks, just go to see the modding section, it's really bad, compare it with xeno or even vbulletin... the spoiler tag should be one of the easiest mod to install in a forum but you see...it's not that simple here and we are stuck with a shitty version.

Spoiler

 

Fuwa could really benefit from having let's say a cms integrated with the forum itself, sharing the users or even the blog.
Dataloss is uncommon when you just need to update your forum and dont have weird mods installed. If you have a forum and you actually care about it and you do things properly, which I think is the case here of fuwa (tay puts a lot of dedication into the forum anyone can see that) but they are still making mistakes that shouldnt happen and a big part of it is because of the forum system itself which is really bad. 
I can't comment on mangagamer because i dont know what happened but from what you are saying that's not how a site manager should behave or act and because they do it doesnt mean it's the right way to do things, doesnt matter if it is a fansite or a professional one.
 

Link to post
Share on other sites
23 hours ago, solidbatman said:

If memory serves, we did try to migrate the forums to another software at one point. It was a disaster. 

As I read it, the goal isn't to move to a new server software, but rather to get something, which can be backed up easily and have minimal risk of data loss. Secondary goals are low maintenance and user friendliness/user features. If the current software doesn't live up to the requirements, then fix it or find some new software, which does.

 

If it is decided to move to new software, then copy the server to some local server, convert it and check the result. If it breaks stuff, delete it and start over. Repeat until you get it right. Once you know how to do it, backup the server and switch the software. If it unexpectedly breaks stuff, you still have the backup you can restore.

 

A semi related note on this: quite a number of servers today are virtual servers and with good reason. You can back up the entire HD disk image, run whatever update you want and if it breaks, you can restore the old HD by simply copying a single (big) file. Having the entire server on a disk image also mean it can be copied to other servers if needed. Chances are that the server is already a virtual server unless there is a dedicated server hardware, which is paid to not run anything else.

 

I feel a bit weird saying this as "the new guy". I hate when people show up out of nowhere and tell how to do something and demand it to be done that way (usually ignoring that there is a reason why it isn't so already). I will not demand anything and have tried to be more objective about this post. Still I would like to request not giving up just because of one failed attempt at some point. It is almost to be expected that the first attempt fails, which is why you should sandbox your tests until you get something working.

 

If I have to say my opinion on this matter, then I would say I would prefer different forum software. I hate the post comment interface with all the mandatory mouseclicks and while it is supposed to be intuitive, I get lost in it and still haven't figured out how to make posts look like I want. However this isn't a critical issue. The dataloss is and if the current software has an unfixable design flaw, which makes it prone to dataloss, then I strongly wish for something better.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...