Problems at my other host

[bellgamin]

Here are a couple of interesting forum announcements by my other host -

Last Wed...
We are experiencing severe problem with the *servername* server (Mail & MySQL) due to a DOS attack occurring against the facility.

The attack is not directed at *nameof*Host or any of it's clients.

The reason it is only affecting this server is because this server turns out to be only on a single-homed system and not on the multi-homed systems that the remainder are on. I was under the impression that the computers in this facility were all multi-homed.

Last Thurs...
We have to change the MySQL, Mail & CP server (*servername*) from the single-homed line to the multi-homed line to stop the problems with DOS attacks. We will be doing it this evening at an estimated time of between 3-5 am Pacific Standard Time.

We will be changing the following IP to:
XX.28.158.18 -> XX.154.89.33

Some users may experience a short downtime and if you are using IP settings for connectivity with your mail or MySQL connectivity, you will need to make the modifications manually. :(
~~~~~~~
Thought some of you might be interested.
Regards.......Bell

[bellgamin]

I received following message from otherhost on 9/18/02...

~~~~~~~~
<quote>As you probably noticed we had few cases of increased heavy server load. After a more thorough investigation we have discovered there is some misconfigured script on the system that floods MySQL with queries.

We also know that the script uses a CGI version of PHP. If you are using this method, please let us know so that we can check which script(s) are causing problems.

What we would like to do is find the problem script and either fix it or convert it to standard PHP usage. We cannot have scripts causing a runaway set of queries against the MySQL server.

While we are still trying to isolate the exact problem script, we have determined that it is a script using the CGI version of PHP

The stopping of use of the CGI version of PHP will be a last resort only if we cannot pinpoint the offending script.

Thank you very much for your cooperation.

Support Staff
*otherHost*</quote>
~~~~~~~~~
By the way, if you folks think I should NOT post this kind of stuff, let me know & I'll desist. I just thought you might be interested.
Bell

[admin]

While I hate to see anyone having problems, it makes for quite interesting reading. Keep us updated.

[Gnomercy]

*nod*

The techie in me is loving reading this... I must say, I wish you weren't experiencing these issues, but I also find it interesting reading. I'd be interested in knowing what the script is as well, if you happen to find out.

[Nedani]

Yep. I am curious too. I don’t think it is too hard to track a runaway script but Danny can tell you exactly ;) . Any host has problems; the difference is how the problems are solved.

[bellgamin]

Interesting posts on otherhost's forum 9/18/02

<quote>
Question by Client A - What is the purpose of running php scripts in cgi? To make it more secure or something? I'm asking because I have wondered this, seen it in some scripts before as an option.

Answer by Client B - I think the purpose for some people is to get around PHP being in safe mode. I think I remember reading some threads about PHP scripts that wouldn't run correctly under PHP safe mode, so they were going to try to run the script as a CGI. If I recall correctly, I think Gallery was one of those scripts.
</quote>

As you no doubt have gathered, otherhost ALWAYS runs PHP in safe mode [whatever that means], despite requests by several clients. Some or all of those clients may have departed. They used to post to otherhost's forum a lot, & now they are silent.

Otherhost has drawn several fairly substantial DoS in the slightly more than 1 year since start-up.

I think this is at least partly because otherhost had some *dark side* clients whose forums attracted a lot of bad-a*ses, 4-letter flame sessions, hacker threats, etc.

In any event, the founders sold the store in mid August & now, as you see, the new owner has his own headaches.

Bell

[Nedani]

The safe mode option in PHP allows control of which files can be accessed from a script using functions that access the file system, such as include(), fopen(), readfile(), and so on. The restriction is as follows: A file can be accessed either if it is owned by the same user ID as the script that is trying to access it, or if the file is in a directory to which the user of the running script has access.safe_mode allows the administrator to control which users are allowed to run which functions and also entirely disable functions for security purposes. Many scripts using file uploads are having problems because of this. If you get an error like: "open_basedir restriction in effect", "Safe Mode Restriction in effect.", or "function has been disabled for security reasons" then it is a case of safe_mode :) . From what I see , safe_mode for AOH is off.

[bellgamin]

Originally posted by Nedani
From what I see , safe_mode for AOH is off.

Is that a good thing or a bad thing, I wonder?

Cheeks clamped together,
Bellgamin

[Nedani]

I think it is a good thing. You can use more premade scripts. If safe_mode is enabled then you have enough tools to do the same (bad) things using php in cgi mode, Perl, shell etc. You will not gain any extra security but you will get some extra headaches when trying to install a script.

[bellgamin]

Nedani - I checked thy site again. Wow! I hope those scripts you use don't crash the servers.

:D
Cheeks still clamped,
Bell

[Nedani]

:D No scripts in use yet. :D

safe_mode is used just to check what files can you use (read/write) from php. It has nothing to do with crashing.

I am testing all the scripts on my home server so ... if it is something to crash ... my PC will crash first and I will have nothing to upload to AOH :D .

[admin]

We use openbasedir directives to keep users in their own directories. This prevents a very large portion of the issues associated with user installed PHP scripts.

[bellgamin]

forum posts at otherhost...
~post1~~~~~~~~
9/19 by client F
This is now three days in a row at roughly the same time each day. Disable CGI PHP, cron or both. It took 8 1/2 minutes for THIS page to load to be able to even post this message.

Hermes is in the toilet again:
As of 12:10pm up 6 days, 6:03
load average: 11.36 10.97 8.25

I think this is the point were drastic measures ARE warranted.
xxxxxx

Bellgamin note: otherhost has mysql & email ALL on server hermes in TX, where all the trouble is happening. Host owner lives in CA & is trying to build his own data center in a storage building near his home.

~post2~~~~~~~~~~~
9/19by client S
load average: 17.89 15.32 10.95

bah, you guys should put mysql and email on each server like it is comes by standard so not all the customers suffer the slowdowns
xxxxxxx

~post3~~~~~~
9/19 by host's owner
Yes, I agree with you. The problem is and has always been that I cannot do anything with the servers where they are now.

We are going to start moving the servers the first part of next week to my shop. Once we get it there (on to much more powerful machines) we will build a couple of additional machines to offload the MySQL and Mail to separate servers.

In the meantime we are continuously trying to figure out what scripts are causing so much run-up.
xxxxx
~post4~~~
9/19 by client L
All 3 days I've been at school when the slow down has happened, so it leads me to believe someone has a cron that runs this program. I'm at school from 6:45am to around 2:45pm, PST. Maybe you could check and make sure it's not a cron between that time.
xxxxxxx
~post 5~~~~~~
9/19 by host's owner
We are currently going thru all the crons to see what time they run and what they run.
xxxxx

And the beat goes on.:bawling:

[Nedani]

I am curious why you don't move your site to AOH.

[bellgamin]

I am curious why you don't move your site to AOH.

I have several sites. I moved my main site to AOH. I have 2 sites still at otherhost, paid in advance through June 2003. Besides, most of the folks over there are really nice people. They're just having some problems right now.

Shaloha,
Bellgamin

[bellgamin]

otherhost is "putting priority" on relocating servers from TX data center [leased, I suppose} to owner's OWN data center in CA [owned, I suppose.]

~~~~~~~
Owner's post 9/20/02
I was asked about the equipment we are going to be moving to. Here is our old configurations and what will be the new ones.

OLD
Zeus (web.*host.com): 1.2GHZ Celeron | 512MB SDRAM | 40GB EIDE HDD
Poseidon (web2.*host.com): 1.2GHZ Celeron | 512MB SDRAM | 40GB EIDE HDD
Hermes/KRAKEN (cp.*host.com): 1.2GHZ Celeron | 512MB SDRAM | 40GB EIDE HDD
Apollo (web3.*host.com): 1GHZ PIII Copermine | 512MB SDRAM | 40GB EIDE HDD

NEW
Zeus (web.*host.com): 1.8aGHZ P4 | 512MB SDRAM | 60GB EIDE HDD (Hardware RAID 0)
Poseidon (web2.*host.com): 1.8aGHZ P4 | 512MB SDRAM | 60GB EIDE HDD (Hardware RAID 0)
Hermes/KRAKEN (cp.*host.com): 1.8aGHZ P4 | 512MB SDRAM | 60GB EIDE HDD (Hardware RAID 0)
Apollo (web3.*host.com): 1.8aGHZ P4 | 512MB SDRAM | 60GB EIDE HDD (Hardware RAID 0)
xxxxxxxxxxxx

Bell sez: You'll recall that otherhost has all MYSQL & eMail on hermes. A very expert fellow-client posted the following comment...
~~~~~~~~
Client wj 9/20/02
Up Hermes ram to 1 gig and go Dual Athlon MP's. Much better options for the money. They will give you more power!
xxxxxxxx

~~~~~~~
Owner 9/21/02
When I build the MySQL machine, I will build a very good box with 1+gb of mem. I do not know if I will build a dual AMD.

As I've told everyone at introduction time, my experiences is mostly with Windows. Believe it or not, MYSQL runs better on a single processor with large amounts of mem, compared to a dual processor with the same mem.
xxxxxxxxxx

~~~~~~~
client wj 9/21/02
MySQL Performs SIGNIFICANTLY better with 2 processors and AMD seems to run it even better. And a Dual Athlon XP 1900+ would cost the same as a P4 1.8.
xxxxxx

& that's 30 for now,
Bellgamin

[admin]

Originally posted by bellgamin
Believe it or not, MYSQL runs better on a single processor with large amounts of mem, compared to a dual processor with the same mem.



:eek: :eek:

Ok, you are right, I don't believe it.

[bellgamin]

If you read the forum posts carefully, notice how owner magically changes how much RAM he will put into new hermes. He raises it from 512K to 1+gig, like that had been his intention all along, with never so much as a nod in wj's direction.

As to Danny's comment concerning owner's strange idea that 1>2, I can only hope & pray that owner will put aside his pride & give thought to wj's recommendation.

Beyond here be dragons

[Nedani]

<edit>I removed something stupid.</edit>

Running MySQL, mail, apache on each machine would work much better and the two processors can be used at their full potential.

Keeping all the databases and all e-mails on one machine requires a great level of trust (like paying in advance for a year of hosting ;) ) and I don’t understand why he is trying the same solution after this failure.

It was dos attack, it was a runaway script or the MySQL machine is not performing as it should? Better clear these things before making any other move.

I hope we will never have this kind of discussion about another host (you know which one :D ).

[admin]

I am not sure I agree. According to my research, the threads are able to run on multiple cpu's.

[Nedani]

Yes, Danny is right. Thank you for update. I checked and I see that MySQL is running on a single cpu only on FreeBSD. I am throwing away my courses notes, the teacher didn’t bother to update, I didn’t bother to check … and now I am saying stupid things here. Sorry. :bawling:

[admin]

No reason to appologize. Thanks for keeping me honest. ;)

[bellgamin]

~~~~
Client F 9/22 0831 hrs
I've just lost my entire site right in the middle of a special event!!!!

This is getting ridiculous. Answers please!
xxxxxxx
~~~~~
Owner 9/22 0902
The Data Centers are having problems again. I still have not been given any answers as to what the problem is.
xxxxxxxxxx
~~~~~~
Client wj 9/22 1009 hrs
At this point it looks like routing problems. Ping times from zeus to hermes are in the 500 ms range.

Client wj 1011 hrs
Traceroute from Zeus to Hermes shows drop off at global compass. More than likly a DDOS preventing Hermes and Zeus from communicating properly or a router problem.
xxxxxx
~~~~~
Owner 9/22 1012
It is router in the chain to Hermes that is having a problem. I just got word on it. Right now we are working on a problem with Apollo's Apache not responding.
xxxxxxx

Another thread - about owner's new data center
~~~~
Owner 9/22
I have been going with MSI for single processor boards for the past 2 years with very good success. For dual I usually go with either Tyan or Asus
xxxxxxxx
~~~~~~
wj 9/22
I have ECS! ECS RoX0r!
But still. AMD just have that more power and L1 cache!
xxxxx

Aloha............Bell

[admin]

I think the new data center needs:

:chainsaw:



Just kidding. Hopefully they will have you up soon.

[bellgamin]

Pssst... I just obtained *Secret Photos* (http://www.alohabible.net/otherhost.html) of the heart of otherhost's former Data Center in Texas.

Danny, now that other host is moving out, I think these guys will give you a very good price.

Uhhh... be sure & let me know if you decide to take advantage of their offer. :eek:

[bellgamin]

9/24 Owner says...
To make everyone happy (and give me a change to evaluate them) I'm setting up a dual Athalon 2000+Mhz server for the MySQL server.

~~~~~~~
Client wj 9/25 responded...
Good Choice Craig. Just because the clock speed is 1.63 doesn't mean it isnt faster then a p4 2 ghz.
This is the true difference between the p4 and the Athlon MP:
CACHE!
Athlon MP:
L1: 128k
L2: 256k

P4:
L1: 8k
L2: 256k

The L2 is the same but the Athlon beats the P4 hands down on L1 which means MORE POWER!

[bellgamin]

Previous partner-owners of other host did their own Tech Spt. New owner isn't as skilled, & couldn't keep up.

A few weeks ago new owner hired 3 Tech Spt people -- 2 are in Russia, 1 is in a 3rd-world nation I haven't figured out yet.

Except for Tanya [1 of the Russians] their English isn't the greatest, but [from client comments I am now reading at otherhost's forum] they are EXTREMELY skilled. According to *insider info*, new owner got these 3 for = or < than what he would pay for 1 USA techie.

It looks like new owner is slowly pulling his chestnuts out of the fire. I now am seeing positive and congratulatory forum-postings by several clients with top-cost [platinum] accounts who previously said they would leave. Sounds like Tanya [via PMs, I suppose] is administering TLC by the pound, over & above mere tech spt. No innuendo here -- I'm fairly certain she still resides in Russia.

[bellgamin]

Otherhost is in final stages of moving to his OWN data center. Owner's progress report...
<q>
Monday, 30 September 2002:
5:30 am PST/8:30 am EST
Data transfer to the new servers has started. At this moment all ***Host customers can be using their hosting accounts as usually. We will inform everyone when there is a temporary need (from 12 to 24 hours) to stop changing settings of the Control Panels and contents of the websites so that we can successfully finish the transfer.
</q>

[bellgamin]

Okay, the move of otherhost to its own data center is complete.

My websites with that host never stopped working for a minute. Traceroute from Chicago is just 12 hops! Ping times from various locations in US, Europe & Asia are astoundingly fast. Even though the numbers changed, I still have my dedicated IP and it is zzzzzip-fast! It looks like the guy REALLY pulled it off. :D

The set-up in his new ceter is...
Zeus - web.***host.com - DNS1 & Web
Poseidon - web2.***host.com - DNS2 & Web
Hermes - cp.***host.com - CP & Mail
Apollo - web3.***host.com - Web
MySQL1 - MySQL & PostgreSQL

Otherhost has been in biz <1 year. Newowner has been at the helm slightly <2 months.

The otherhost folks are [& have been] on my site's prayer list. So also is AOH.

End transmission.... Bellgamin

[admin]

I am glad to hear everything worked out ok bell. I congratulate your host on their move.