Forums are frequently down

[ReadOnly]

Once or twice a day in the last 2 weeks forums are down for a 5-10 minutes.

Tried through web-proxy like hidemyass - all the same. It's not my provider.

Last one was 5 minutes ago.

Is this a normal behavior?

[TerminX]

Yes, this is the server backup process that runs every 12 hours, scheduled to run at 11:30 AM and PM, Central time.

[ZuljinRaynor]

We don't have a sticky for this?

[Joe Siegler]

No, because if we did, people would ignore it and ask anyway. I've learned that over the years. I could stick it in the FAQ, but would anyone read that?

Yeah, it's a bit bitter sounding, but 16 years of being a forum admin and being ignored will do that to you. :)

[ZuljinRaynor]

Heh, understandable. :p

[fast-1]

Right on. ;)

[Yatta]

What happens if the backup process occurs while the server/forum is online, instead?

[SplatterHappy]

In my experience (from working in a datacenter for 18 yrs), trying to back up an active database is almost never a good idea.

Worse case scenario: entire DB and your backup is corrupted, leading to a much longer downtime while you restore from a previous backup (and thereby losing all the changes made since that backup).

While dynamic (online) backups are possible and even demanded in the business, if it's not manditory that the database (forum in this case) be up, the tried and true method of taking the db offline is often the best.

Please feel free to correct me if I'm off-base on any of this, Joe. :)

[Joe Siegler]

What happens if the backup process occurs while the server/forum is online, instead?

You don't want to do that.

[Bludd]

Couldn't a placeholder page be displayed instead of just going offline? Something like "Pizza, brb"?

[Bludd]

Yeah, whatever just to show that the forum isn't exactly down. Just take the DBMS down and let the web server show a message telling us it won't be long now.

[Joe Siegler]

Couldn't a placeholder page be displayed instead of just going offline? Something like "Pizza, brb"?

Apache is down too at that time, because even with the forums "closed" there is still database access.

If you want a "true" backup, you have to keep people out of the database, and the only way to do that is to totally disable access to it.

[ReadOnly]

Why not make a message like "backuping, be back in 5 minutes" when you try to access forums? Or since the entire site(forums.3drealms.com) is down it's not possible? Or isn't just worth a time to implement.

[d3ad connection]

Why not make a message like "backuping, be back in 5 minutes" when you try to access forums? Or since the entire site(forums.3drealms.com) is down it's not possible? Or isn't just worth a time to implement.

Since they take Apache (web server) down when they backup, it isn't possible.

[Steve]

I don't see the big deal :confused: If it's down, it's down. It will return whenever :o

[ReadOnly]

It's down when I'm starting to read internets on work. Annoying. Though, I can live with it of course.

[Chimera]

In my experience (from working in a datacenter for 18 yrs), trying to back up an active database is almost never a good idea.

Worse case scenario: entire DB and your backup is corrupted, leading to a much longer downtime while you restore from a previous backup (and thereby losing all the changes made since that backup).



You've never used SQL Server then... :) I can backup a database during the middle of the day and no-one will notice. With regard to kicking people out first - it backs up a portion of the transaction log too, so it captures anything that's going on at the time. Full drive backups don't capture files that are in use, so as long as flat file backups are taken of the databases all is well. Recover the drive from the drive backup, then restore the databases.

As an example, the other week the data drive on one of our servers failed. The new drive was restored minus the databases, we rebuilt the system databases to get SQL Server started and then restored their backups over the top, then restored the user databases.

With SQL 2008, you can even restore a database and let users in whilst it's still restoring. Don't think I'd go that far, but still. Shows what can be done nowadays...

[TerminX]

Backing up a live database (e.g. anything that isn't explicitly read-only or offline during the backup process) isn't a great idea. Just browsing the forums will still have to write to the db in order to update access times, et cetera. I'm sure it's also a lot easier to automate bringing Apache down during the backup than it is to automate switching the forum over to read-only database access.

[Joe Siegler]

Backing up a live database (e.g. anything that isn't explicitly read-only or offline during the backup process) isn't a great idea. Just browsing the forums will still have to write to the db in order to update access times, et cetera. I'm sure it's also a lot easier to automate bringing Apache down during the backup than it is to automate switching the forum over to read-only database access.

I considered stuff like moving the entire forum directory to another name, then copying a temp to the live vb directory during the 3-4 minutes of downtime, then moving it all back.

Seems like a lot of work - we're talking a total of 7 minutes out of 24 hours a day, I figure it really isn't that big a deal. Perhaps someday if I'm really bored and have nothing better to do, I'll make that happen. Just got filed under "really isn't that important".

The only important part was getting a true database dump, hence the current conditions during backup. it was easier to just down apache as part of the backup script. :)

Oh Chimera.. I can of course back the db up with the forums closed. That's not the problem. I wanted a true backup of data. As TerminX said, even with the forums closed, there are still accesses. Folks have to see the closed message, and anyone who has an account and is logged in will get the closed msg, but their acct info gets updated with "last login", that kind of stuff. So it's not a "true" backup in that regard. Just no new posts during "forums closed".

[Scream]

We back up a dozen or more Oracle and MSSQL databases live every day using a variety of tools. I do my Exchange database every day while it's up and running, and have been doing so for 10 years. You just need the right backup software.

In this case though, I agree, it's not worth the effort or expense to save a few minutes of downtime with a bunch of users you don't have any SLA with anyway. ;)

[Bludd]

What? You don't have an SLA? I forced Joe to sign an SLA which promises 99.99999% uptime! This forum is business critical for me! :cool:

[Scream]

Ha, the way some people talk about it you'd think it was an essential service for them. ;)

[Steve]

Ha, the way some people talk about it you'd think it was an essential service for them. ;)

You would think that :o

/tries to hide post count.

[Superczar]

I have noticed lately that when the forums go down at 12:30 (Eastern) for routine backup, they come back up, run fine for about 10 minutes, and then like clockwork "hang" for a few minutes at about 12:50 (Eastern)...

[Joe Siegler]

I have noticed lately that when the forums go down at 12:30 (Eastern) for routine backup, they come back up, run fine for about 10 minutes, and then like clockwork "hang" for a few minutes at about 12:50 (Eastern)...

I don't know anything about the "hang". The 11:30 AM & PM (CST) backups are sql dumps where Apache is downed during the interim.

The only other backup is the one daily one where all the files on the drive are physically backed up, but apache is not downed during that. I seem to recall that's somewhere around 3 or 4 AM, but I forget offhand where that is.

This hang you speak of I have no knowledge of. All I can suggest is this. If you're around when it happens, see what the first page display number is at the bottom when it comes back. The bit where it says "Page generated in 0.27052593 seconds".. Tell me what the first one of those back is.

If it's some out of whack number, then it's a server thing, but if it's a reasonable number, then it's likely not coming from this end.

[Llama Gibbz]

I can confirm that too,its like clock work.

[Superczar]

Like clockwork it happened again at 12:50 am here, worked fine in the 10 minutes or so between the shutdown and 12:50, then it hung for over 2 minutes; here is the page load stat:

Page generated in 138.64313889 seconds (100.00% PHP - 0% MySQL) with 19 queries

[Joe Siegler]

I have noticed lately that when the forums go down at 12:30 (Eastern) for routine backup, they come back up, run fine for about 10 minutes, and then like clockwork "hang" for a few minutes at about 12:50 (Eastern)...

Is this right after both 11:30 backups (AM & PM), or just one of them?

[Superczar]

I can't give you a definite yes/no - I didn't really keep track of which half of the day it was doing it, other than a certainty that it did it at the nightly backup last night, whether it does in teh day - I'll have to find out at work if I get time next 2 days.

EDIT: It did not seem to do it this afternoon, maybe it is only doing it after the nightly backup???

[Joe Siegler]

OK, I witnessed this tonight. I'll look into it tomorrow.

Edit: I checked the logs right after I saw this, and the only thing going on was a bunch of Xbox Live Leaderboard updates. The thing is they started at 11:38PM, which if they were the cause of the delay is 12 minutes earlier than the 11:50 report time.

[Joe Siegler]

It should be fixed.

Let us know if this crops up again.

[Thewho]

What was the cause? Just out of curiousity.

[Joe Siegler]

There was sql dump coming from somewhere else in the backup scripts that was forgotten about. That one did NOT take down apache like the normal 11:30 ones, where apache is taken down.

This addtl dump has been removed from the cron jobs list.

[Superczar]

Did you change the time to 11:05 (central)?

[ReadOnly]

Forums were offline at the begging of the hour(see my time of writing).

P.S. It's 8:18 here. UTC+3.

[Joe Siegler]

Did you change the time to 11:05 (central)?

I have no idea what you're talking about there. If this is a new question, then it should be it's own thread. I don't see how this relates to the question.

[ReadOnly]

The forums were down at that time(11 pm central) yesterday. Couldn't check today at that time, though. Gonna check tomorrow.

[Joe Siegler]

The forums were down at that time(11 pm central) yesterday. Couldn't check today at that time, though. Gonna check tomorrow.

They shouldn't have changed - what's the server time say at the bottom of the page you're looking at? Is it right?

I'll check the logs tomorrow when I get back to the office, make sure the sql dumps are still happening at the same time.

[ReadOnly]

It says "All times are GMT +3. The time now is 08:41 AM".

Btw, I haven't noticed the forums being down at 11:30. Though, I can't say the exact time I was trying. Maybe backup was already complete.

[Joe Siegler]

Logs and cron all show 11:30. There's nothing to indicate anything about 11:00.

[ReadOnly]

Well, all is ok at 11pm. False alarm, it seems.

[Inanimate Carbon Rod]

Would there be anyway to move the backup time to later at night?


Here on the west coast its 9 something when the forms go down.

[Joe Siegler]

Would there be anyway to move the backup time to later at night?

Here on the west coast its 9 something when the forms go down.

It has to be SOMETIME, and since the servers are in Central, that's when we stage things around. If we were in Eastern, we'd do it then.

You're in the wrong time zone. Move back. ;)

[Joe Siegler]

Yes, this is the server backup process that runs every 12 hours, scheduled to run at 11:30 AM and PM, Central time.

Was looking around the control panel today, and ran across something I am surprised I never thought of using before. The Notices Manager. It's the same thing that had that notice of moving the forums up for a bit.

I set up two notices that will get displayed for 10 minutes leading up to the downtime. They cannot be dismissed, and should (if I set up my UTC times right) display for the last 10 minutes leading up to the downtime. When the forums come back, they won't be displayed again until the next 10 minutes before the next backup.

I wish I would have thought of this before today's first backup, which occurred 50 minutes ago as I type this. I'll have to wait and see what happens with the next one later tonight.

[ADM]

Ahh that's right. Good idea Joe.

[Joe Siegler]

I wish I would have thought of this before today's first backup, which occurred 50 minutes ago as I type this. I'll have to wait and see what happens with the next one later tonight.

Damn. Missed it. Was anyone online to see if this worked right?

[peoplessi]

Now shows:

Attention peoplessi. The forums will be going down for a database backup at the bottom of the hour (11:30AM CST). This backup will take no longer than 10 minutes, during which time the forums will be inaccessible. When the backup is complete, the forums will be restored.

[Joe Siegler]

The name is customized to everyone logged in, but that's the message. The other one is the same except it says 11:30PM instead. I take it it worked the right way?

[peoplessi]

Well, my time here shows in GMT+2, so I didn't check, but at the time of my post it showed that message. I guess it works :) Very neat feature.

[Scotty]

I happened to be on at the right time (12:28 pm EST), and it worked.

[Joe Siegler]

Cool. Thanks.!

[Malgon]

Good idea Joe! :)

[Joe Siegler]

I saw it myself at 11:25 yesterday morning, so this should be a non issue now. Although I might turn the text red, so it's more noticeable.

[ReadOnly]

I'm seeing it now. Yes, the red text will be more noticable.

[Joe Siegler]

I'm seeing it now. Yes, the red text will be more noticable.

I made it red and bold this morning. :)

[ReadOnly]

Can't wait to see it! ;)

[Montykoro!]

Wow...today it's a 15/25 min downtime...

[Joe Siegler]

Wow...today it's a 15/25 min downtime...

Separate issue. Servers are being rearranged on our network here.

[Montykoro!]

http://img242.imageshack.us/img242/3366/foro2ny0.th.jpg (http://img242.imageshack.us/my.php?image=foro2ny0.jpg)

Nice... a little script and presto screenshoot before de bckp :)

Thanks Joe for the explanation...sorry for the "offtopic"

[Scream]

I made it red and bold this morning. :)

This works well. Good idea. Should save you from having to answer the question "Why are the forums down?" as often. :)

[Rellik66]

Hmm, seems the daylight savings time switch screwed up the message, now the message appears an hour after the backup.

[Joe Siegler]

It was just brought to my attention that the big red notice has been wrong since daylight savings came into effect recently. I looked at the code for the notice, and it says

"Display notice if... "The time is between 17:20 & 17:30 (UTC / Universal Time) (Use hh:mm 24 hour format)

"Display notice if... "The time is between 05:20 & 05:30 (UTC / Universal Time) (Use hh:mm 24 hour format)

Those were the two triggers for the two notices. I made 'em say 16:20/16:30 and 04:20/04:30. This should correct the problem. Someone let me know if it triggers at the right time now, please. Thank you.

[Superczar]

It's 1:30pm here, and when I just fired up the forums - I got the message... also my time is set right but server says it's 12:30, not 1:30.

[Joe Siegler]

It's 1:30pm here, and when I just fired up the forums - I got the message... also my time is set right but server says it's 12:30, not 1:30.

Where is "here"? That would be helpful.

[Superczar]

Eastern time zone, PA. Sorry, I kinda figured most people knew by now. :D :doh:

It seems to be working right now anyways, so no big deal.

[Rellik66]

I can confirm that the message has shown up at the proper time.

[Joe Siegler]

I can confirm that the message has shown up at the proper time.

Cool. I forgot to look, tell you the truth, and I was online at the right time, too. :)

[Rellik66]

It also looks like I made that post with mere seconds to spare!

[Talos]

17:26 CET here in Germany and I see the message. I was kind of afraid when I first read "Attention Talos"... :D

[Scream]

17:26 CET here in Germany and I see the message. I was kind of afraid when I first read "Attention Talos"... :D

You're banned!!!! ;)

[Steve]

The name is customized to everyone logged in, but that's the message. The other one is the same except it says 11:30PM instead. I take it it worked the right way?

It scared the shit out of me when I first seen it. I almost read the first few words as

Attention Steve, stop posting crap

:o

[ReadOnly]

Actually, you read it right.

[Steve]

Actually, you read it right.

Yet I'm still here! :p