PDA

View Full Version : Server Crash Today



paman
28-04-10, 00:04
:ranger:
Hi all,

As some of you may know, they EF server was offline for a brief period of time this morning. The reason was due to a hung process taking up 400% of the system resources, thus locking up the server. After the server was rebooted and the process was killed, service was restored.

Although no database corruptions were detected, I still invite you to send me a message if you notice anything going bonkers since this incident.

Please let me know if you have any questions. Thanks.

bridge
28-04-10, 00:28
Thanks for your quick help.
This explains why I can't access this site all day long.

Davita
28-04-10, 00:30
Yes, and thank you for the notice.

David

donfuego
28-04-10, 03:16
Was it the fault of the hosting service or entirely self inflicted? :)

Gratilla
28-04-10, 06:13
Was it the fault of the hosting service or entirely self inflicted? :)


The reason was due to a hung process taking up 400% of the system resources, thus locking up the server.

That is, a bug in a program that sends the CPU into a frantic loop.

Also, a yuphemis..., ufami... , hyufemi..., same thing as someone not paying the bill.

TheBrightBlue
28-04-10, 13:37
I need to wait up to 1.5 hours to open this forum site today (was trying to log in at 12pm but just now i can log in). Mmmmm... Anyone gets the same problem or only me?

kingwilly
28-04-10, 13:50
Paman, I seem to be missing about 200 reputation points, can you please remedy

paman
28-04-10, 14:11
Was it the fault of the hosting service or entirely self inflicted? :)

Self-inflicted. After further investigation it appears that the resources on the server panics whenever we get a spike in traffic. I'm optimizing configurations right now to help prevent this in the future.


Paman, I seem to be missing about 200 reputation points, can you please remedy

You need to have 200 points removed? I'm on it! ;)

paman
28-04-10, 14:20
k. I've reconfigured Apache (webserver) and MySQL (database) settings. It has greatly reduced usage (down to 90-100MB from 180MB++). As long as we don't get further major spikes, we should be just fine.

atlantis
28-04-10, 16:18
I need to wait up to 1.5 hours to open this forum site today (was trying to log in at 12pm but just now i can log in). Mmmmm... Anyone gets the same problem or only me?
I had the same. Could not open the forum. Tried half an hour and gave up.

stt_cibubur
28-04-10, 16:36
I have problem accessing the forum using Firefox ...is it only me or anybody else?
IE and Chrome are OK.

waarmstrong
28-04-10, 17:06
Same, here, Atlantis. Gave up after 40 minutes. Actually got some real work done during the interim. Scary: the crash is getting in the way of my effort to be unproductive in retirement

atlantis
28-04-10, 17:28
Same, here, Atlantis. Gave up after 40 minutes. Actually got some real work done during the interim. Scary: the crash is getting in the way of my effort to be unproductive in retirement
Hahaha. I did the same. I've spent the whole day working in our land, cutting grass, planting trees and carrying tons of goods from one place to another. Our gardener sure prefers to see me working on my computer. He has finished the day exhausted and so do I. Please Paman, do something. :smile2:

waarmstrong
29-04-10, 12:10
What ever fix was in, apparently was less than perfect as I was getting some sort of "Data Base" error throughout most of this Jakarta morning. I hope the latest effort is not in the tidak apa apa spirit.

gffgold
29-04-10, 12:16
Not a very auspicious start to the new hosting arrangements.

paman
30-04-10, 02:06
What ever fix was in, apparently was less than perfect as I was getting some sort of "Data Base" error throughout most of this Jakarta morning. I hope the latest effort is not in the tidak apa apa spirit.

The original problem was handled with. The database issue is a new thing I need to figure out. I'll keep you guys up-to-date.


Not a very auspicious start to the new hosting arrangements.

Yeah haha. The thing was that before we had a shared account that was pretty much was handled by the hosting company. Now since we are on the VPS box, I'm the one who has to figure out the problems. It may take a little longer for me to respond. For example I found out about the database issue while I was out with some mates, so it took me awhile before I was back at a computer.

waarmstrong
30-04-10, 09:39
I guess will need to chain that boy to the box.

kingwilly
30-04-10, 10:09
i find that bashing the keyboard repeatedly until the software lets you in tends to help, shout if necessary.

paman
30-04-10, 14:37
We've upgraded our server to add additional memory. We should have more then enough memory now to handle all requests. This was one our biggest issues that I thought we could overcome by optimizing the config files, but apparently that wasn't the case.

waarmstrong
30-04-10, 16:24
I can sympathize, being one who usually tries to get by on the cheap, before bank-rolling a more expensive fix.