bellgamin
04-15-2003, 01:04 AM
My other host -- the teen-ager I told you about in another thread -- got hacked something awful - 4/12-4/14.
Poor kid. Blood in the water. The sharks are circling now. Here is the fellow's post about what happened. I have modified it just enough to protect his identity. I'll refer to the host as XYZ.com
~~~~~~~~~~~~~~~
An amazing set of events has happened over the past day and a half. Totally proving that Murphy's Law is alive and well.
Yesterday morning our newest server (#7) was compromised. An intruder came into the server through a mistakenly enabled shell account, and setup a program which gave them superuser (root) access to the server. Thankfully, there are currently very few websites on the server, belonging to just two users, and as of the night before this, XYZ.com's own site had been placed onto the server.
I began to research what was done to break in, what had been done once they broke in, and how to clear up the damage. What I believed to be the only damage, was a sniffer that was setup on the server, to watch for incoming and outgoing passwords. From the bash history file the intruder left on the server, this looked to be all that was done. It was left on the server, logging and trying to catch passwords. (of which it did log a single password, however, the intruder did not come back to check their log, and I switched the password as soon as it had shown up in the log of the sniffer)
This is very complicated, so please bear with me as I try to explain this as simply as possible. So far, they have compromised the system, broke in without permission, and used their newfound access to try and retrieve passwords for other webservers.
Shortly after I realized what had happened, I checked a couple logs to see if I could determine the IP address of the intruder, however, this has been my first experience with such an intrusion, and I did not check as many logs as I could have. I was more worried about getting the remnants of the sniffer off of the server.
I ran the built-in uninstaller that had been so kindly included with the sniffer files. Uninstall was successful. It seemed all was well.
I went to install an SSL certificate for a domain on the server, and Apache fails to restart. I check the logs, and found that the ports Apache uses for displaying webpages, were in use.
I figured that all was well, I just needed to restart the server to clear out any hidden processes that may have still been running from the sniffer files. About five minutes later, realizing that the server was not going to come back online, I began to of course, panic.
I read up more on the intrusion method, and the sniffer, and found that it had been associated with a corruption in a file named 'ifconfig' -- this file deals with network settings. Apparently... even though this was what I believe was causing the port issues, more had been done to destroy the server.
As of right now, we only know that the linux configuration files were corrupted beyond any comprehension. The server would not boot, and was showing about one sign of life: the power on light.
Ok, so, let's just fix it, and get everything cleared up, and get back on with hosting. Well... here comes Murphy's Law:
I sign on to AOL Instant Messenger to chat with my provider, as this newly added server is in a different datacenter than our other servers, and I've only dealt with them through messenger thus far.
The two representatives are nowhere to be found. Well, there is absolutely nothing I can do at this point, so I just stay on AIM to let people know what has happened, and that things should be cleared up as soon as possible.
A side note: Earlier I had installed a firewall onto this server, and in doing so, disabled the pinging response from the server. The datacenter monitors the server in this way, and got in touch as soon as it was not pingable. However, this day, the day after I had messed with the firewall configuration, they obviously expected, just as I would have in their shoes, that it was the firewall once again.
About six hours after my failed attempt at restarting the server, support signs on. I ask them if they can fix the server, at this point I was still thinking that it was a simple problem, only requiring a little intervention by a real person to get it booted.
I was then informed, that the one person who could help get the server running once again, was at the hospital. Just so happened to be the day his wife delivered their new baby boy. Any other day, support would have been available immediately, but Murphy's Law decided to intervene, and keep that from happening.
Delivery of the baby was prolonged until earlier the next morning, and the support guy that can help, is of course, running back and forth between the hospital for his wife. Once again, in such shoes, I would be doing the same.
That's the summary of what happened to the server, now on to possible downtimes.
The nameservers of XYZ.com had been pointing to the new server since yesterday morning. In light of the nameserver settings, XYZ.com's DNS entries were not being read, since the DNS server that XYZ.com's site actually resides on, was down. Thus, any subdomains of XYZ.com, and anyone trying to use server #2 to connect to a server via FTP, for cPanel, etc, would have been unable to access in this way.
This outage only affected a handful of sites which were on the new server, along with any hosted subdomains. It also affected any users that were on ISPs which do not follow the RFC Standards. Basically a matter of how the ISP looks up a domain's nameservers. If your ISP is in accordance with the RFC Standards, then it would look up the nameserver IP addresses through the nameserver itself, via the WHOIS database, which all of our nameservers are registered with. Those which do not follow these standards, will look up the nameservers via it's parent domain's DNS server. Which in this case, was offline. If you have experienced an outage with your site over the past 36 hours, I'd highly recommend switching to a new ISP, that abides by the RFC Standards. However, we will be taking measures to ensure redundancy with our site and DNS, as soon as we get everything cleared.
All explanation out of the way, on to the better news, and policies, which must be set in place to prevent this from happening on another of our servers.
We are bringing on a long-time Linux consultant, to improve server stability, and most importantly, server security. We will also be installing firewalls onto all our servers. (For those running HLStats, do not worry, we will be checking the ports that all HLStats users are running through, and leave them opened)
From now on, there will not be any accounts automatically setup with SSH access. Any account that includes SSH access, will be granted SSH access, upon our receiving an image of a valid driver's license, or another verifiable form of identification. The price of SSH access, is also to increase back to $XX.
This of course has brought a drastic downtime upon XYZ.com's own site, which I am sure has lost us a lot in terms of our reliability factor, with many potential users. Our reliability checker on FindMyHosting is showing around 90% right now, but, that's only for our main site, and is for sure going to improve, as I get a redundant system in place to make certain this does not happen again. We are working very hard to improve a lot of things, security and stability are definitely at the top of the list, along with 100% functionality and uptime with our main support area, the forums and XYZ.com's own site.
For those who are wanting to leave because of this, I would like to strongly urge you not to. Not from a business standpoint, but because we are working very hard to improve things, and will improve things.
Also, until new server comes back online and gets re-setup with cPanel, signups will not work. I'm expecting the new server to be back online sometime between midnight, and 4AM, CST.
Regards,
/s/ the lad's name
~~~~~~~~~~~~~~~~~~~
I do hope the young fellow makes it. I am NOT deserting this guy at a time like this. I must admit, however, that I am sooo glad my main sites are here at AOH.
Poor kid. Blood in the water. The sharks are circling now. Here is the fellow's post about what happened. I have modified it just enough to protect his identity. I'll refer to the host as XYZ.com
~~~~~~~~~~~~~~~
An amazing set of events has happened over the past day and a half. Totally proving that Murphy's Law is alive and well.
Yesterday morning our newest server (#7) was compromised. An intruder came into the server through a mistakenly enabled shell account, and setup a program which gave them superuser (root) access to the server. Thankfully, there are currently very few websites on the server, belonging to just two users, and as of the night before this, XYZ.com's own site had been placed onto the server.
I began to research what was done to break in, what had been done once they broke in, and how to clear up the damage. What I believed to be the only damage, was a sniffer that was setup on the server, to watch for incoming and outgoing passwords. From the bash history file the intruder left on the server, this looked to be all that was done. It was left on the server, logging and trying to catch passwords. (of which it did log a single password, however, the intruder did not come back to check their log, and I switched the password as soon as it had shown up in the log of the sniffer)
This is very complicated, so please bear with me as I try to explain this as simply as possible. So far, they have compromised the system, broke in without permission, and used their newfound access to try and retrieve passwords for other webservers.
Shortly after I realized what had happened, I checked a couple logs to see if I could determine the IP address of the intruder, however, this has been my first experience with such an intrusion, and I did not check as many logs as I could have. I was more worried about getting the remnants of the sniffer off of the server.
I ran the built-in uninstaller that had been so kindly included with the sniffer files. Uninstall was successful. It seemed all was well.
I went to install an SSL certificate for a domain on the server, and Apache fails to restart. I check the logs, and found that the ports Apache uses for displaying webpages, were in use.
I figured that all was well, I just needed to restart the server to clear out any hidden processes that may have still been running from the sniffer files. About five minutes later, realizing that the server was not going to come back online, I began to of course, panic.
I read up more on the intrusion method, and the sniffer, and found that it had been associated with a corruption in a file named 'ifconfig' -- this file deals with network settings. Apparently... even though this was what I believe was causing the port issues, more had been done to destroy the server.
As of right now, we only know that the linux configuration files were corrupted beyond any comprehension. The server would not boot, and was showing about one sign of life: the power on light.
Ok, so, let's just fix it, and get everything cleared up, and get back on with hosting. Well... here comes Murphy's Law:
I sign on to AOL Instant Messenger to chat with my provider, as this newly added server is in a different datacenter than our other servers, and I've only dealt with them through messenger thus far.
The two representatives are nowhere to be found. Well, there is absolutely nothing I can do at this point, so I just stay on AIM to let people know what has happened, and that things should be cleared up as soon as possible.
A side note: Earlier I had installed a firewall onto this server, and in doing so, disabled the pinging response from the server. The datacenter monitors the server in this way, and got in touch as soon as it was not pingable. However, this day, the day after I had messed with the firewall configuration, they obviously expected, just as I would have in their shoes, that it was the firewall once again.
About six hours after my failed attempt at restarting the server, support signs on. I ask them if they can fix the server, at this point I was still thinking that it was a simple problem, only requiring a little intervention by a real person to get it booted.
I was then informed, that the one person who could help get the server running once again, was at the hospital. Just so happened to be the day his wife delivered their new baby boy. Any other day, support would have been available immediately, but Murphy's Law decided to intervene, and keep that from happening.
Delivery of the baby was prolonged until earlier the next morning, and the support guy that can help, is of course, running back and forth between the hospital for his wife. Once again, in such shoes, I would be doing the same.
That's the summary of what happened to the server, now on to possible downtimes.
The nameservers of XYZ.com had been pointing to the new server since yesterday morning. In light of the nameserver settings, XYZ.com's DNS entries were not being read, since the DNS server that XYZ.com's site actually resides on, was down. Thus, any subdomains of XYZ.com, and anyone trying to use server #2 to connect to a server via FTP, for cPanel, etc, would have been unable to access in this way.
This outage only affected a handful of sites which were on the new server, along with any hosted subdomains. It also affected any users that were on ISPs which do not follow the RFC Standards. Basically a matter of how the ISP looks up a domain's nameservers. If your ISP is in accordance with the RFC Standards, then it would look up the nameserver IP addresses through the nameserver itself, via the WHOIS database, which all of our nameservers are registered with. Those which do not follow these standards, will look up the nameservers via it's parent domain's DNS server. Which in this case, was offline. If you have experienced an outage with your site over the past 36 hours, I'd highly recommend switching to a new ISP, that abides by the RFC Standards. However, we will be taking measures to ensure redundancy with our site and DNS, as soon as we get everything cleared.
All explanation out of the way, on to the better news, and policies, which must be set in place to prevent this from happening on another of our servers.
We are bringing on a long-time Linux consultant, to improve server stability, and most importantly, server security. We will also be installing firewalls onto all our servers. (For those running HLStats, do not worry, we will be checking the ports that all HLStats users are running through, and leave them opened)
From now on, there will not be any accounts automatically setup with SSH access. Any account that includes SSH access, will be granted SSH access, upon our receiving an image of a valid driver's license, or another verifiable form of identification. The price of SSH access, is also to increase back to $XX.
This of course has brought a drastic downtime upon XYZ.com's own site, which I am sure has lost us a lot in terms of our reliability factor, with many potential users. Our reliability checker on FindMyHosting is showing around 90% right now, but, that's only for our main site, and is for sure going to improve, as I get a redundant system in place to make certain this does not happen again. We are working very hard to improve a lot of things, security and stability are definitely at the top of the list, along with 100% functionality and uptime with our main support area, the forums and XYZ.com's own site.
For those who are wanting to leave because of this, I would like to strongly urge you not to. Not from a business standpoint, but because we are working very hard to improve things, and will improve things.
Also, until new server comes back online and gets re-setup with cPanel, signups will not work. I'm expecting the new server to be back online sometime between midnight, and 4AM, CST.
Regards,
/s/ the lad's name
~~~~~~~~~~~~~~~~~~~
I do hope the young fellow makes it. I am NOT deserting this guy at a time like this. I must admit, however, that I am sooo glad my main sites are here at AOH.