Responding to a security incident
If your system has been compromised, used to attack another site, or has a network sniffer installed, your system is now out of your control. This is a call to action, but *not* a cause for panic. Unless you were told your system is actively attacking another site, you do not need to act immediately. The attackers may have been in your machine for days, weeks, possibly even months, so another few hours won't be that bad. (If your system is actively attacking another site, you may be asked to disconnect it from the network right away, or UW Technology may be forced to block your connectivity to the Internet to stop the attack.)
Consider what you must do
What you need to do right now is understand the steps necessary to regain control of your system, prepare for and take those steps, ensure that you have tightened the security of your system so this won't happen again right away, and then recover from the incident to bring your system back to its prior working state.
Think of this as like preparing for a multi-day hike in the mountains. You don't just decide to do a multi-day hike and jump in your car. You need to know what to bring, what trails you will follow, how to use that little camp stove, and what kinds of obstacles you will encounter along the way.
Please note that it is critical that you spend the time to know how to respond appropriately, otherwise you will do more harm than good. There is no "quick fix". Rushing and NOT READING and understanding the material referenced here usually means you get to do the job twice and prevent others from doing their jobs (by destroying evidence).
Another thing you must consider now is, is it worth it to leave the system connected to the Internet? Even though this system may be your department's Web server, email server, etc., is it REALLY more important to stay online as it is to regain control? What use is it to spend days of work trying to clean up and recover (even with your Web pages still available) only to have the attackers delete the entire file system, making you start all over again, because they continued to have complete access to your system via the network?
At various points, like when re-installing and re-configuring the operating system before you go to the network to get patches (or if you need to leave for a while to get food or to sleep; you can't just do this for two straight days, after all), you might want to disconnect the system from the network so that your friends don't come back before you've finished your work.
Where to begin?
Start right now by taking a few minutes to read an overview of the incident response cycle. The most critical things you need to do are:
- Take good notes, including who did what, where, how, and when. Keep track of total time spent for all involved, for use in calculating damage estimates.
- Notify the right people to get help. This may be any/all of the following: your own department's network administrators, security@u.washington.edu, cert@cert.org, and local/state/federal law enforcement.
- Preserve evidence, in case the intrusion is more sophisticated than you think (and you have no way of knowing this, so assume the worst). If you fail to do this, you may ruin any chance of a complete resolution to this incident. At the first sign of a full system compromise, work to image the system (don't continue to dig around and especially DO NOT DELETE FILES you find).
- Enforce a "need to know" policy. Limit who is told about the incident, not so much to cover it up as to keep control of what is interpreted as "facts" from things that are really speculation or just plain false. Refer all questions up your management chain.
- Use "out of band" communications. Sending reports from the compromised system, or from accounts that may have been stolen by the intruder, may let the intruder know (and react) to your efforts to contain them.
- Contain the problem and eradicate it. This requires spending the time to learn the full scope of the intrusion, rather than just quickly cleaning up your own system. A sniffer may expose hundreds of other accounts and let the intruder "dig in" very deeply.
Note that both the bulleted list above and the list at the incidence response link above recommend taking notes to remember what you have and haven't done yet (and how much time it took) and taking a snapshot of the system to ensure you preserve evidence. If you are not used to taking notes, now is a good time to learn that habit. Same thing for doing tape backups. If you don't have backups of your home directories and data files, the job of regaining control will be a bit harder, but you can still get the job done.
When making backups for analysis, you might want to consider how they will be used. If you use a program like "ufsdump", which makes backups of file systems, it may not be possible for the person who is going to analyze the backup to read it, especially if they don't have the same operating system version as you are using. File system neutral backups (e.g., "tar", "cpio", etc.) can be read on other systems, but realize that these programs sometimes do not backup special files (e.g., device files, sockets, etc.) and they do not backup inactive (deleted) files.
You may need to make two backups (one with "ufsdump" or "tar", another with "dd"), in order to both preserve ALL DATA on the system and to provide a snapshot of the files -- locations, time stamps, etc. -- that can be analyzed.
Also consider the media. DLT, 8mm, 4mm, QIC tape, CD-ROM, ... These drives may not be at hand for those who will analyze the backup. When in doubt, ask what media type and format is required for analysis. (When sending data to security@u.washington.edu, we can read 4mm (DDS-3) and 8mm tapes (DDS-1), and ISO 9660 format CD-ROMs, or raw "dd" image files transferred over the network as a last resort. We prefer GNU tar formatted archives for operating system independence when transferring individual files from the system, but "dd" image copies are best for preservation of evidence -- including deleted files -- when analysis cannot be done immediately or where the possibility of prosecution exists.)
Also, don't forget that if the attackers have full access to your system, they can read your mail and will know when you report the incident and what response you get. Do your communication from another system or by phone.
Next, review one of these documents to learn about what you should look for in determining what has happened on your system, but don't start trying to fix things yet:
In many cases, the intruders will have gained access to the root account (i.e., you have lost control of the system) and you may not be able to detect the full extent of what is happening on your system. If you are feeling absolutely compelled to do something at this point, and you can still get access to the root account, now would be a good time to do a complete backup. Use the "dd" program and make sure you get copies of all active partitions on the system. If you have never used "dd", see the examples, or contact security@u.washington.edu for assistance.
While the tape is being written, read the Steps for recovering from a Unix or NT system compromise. Also, you should know that the more skilled intruders have tools that allow them to hide their presence on your system, making the system lie to you. These tool kits are called "root kits", and they make life very hard for the novice administrator. Read more about root kits.
Remember that a skilled intruder can hide from you, so don't assume that if you don't see a lot of signs of intrusion, your system was not compromised. ANYTHING strange found on the system that would require system administrator privileges is enough to assume full system compromise and to start acting cautiously so as to not destroy what little evidence is still there. If you can't determine how someone broke in, you cannot prevent it happening again.
Ready, set, go to work!
Things will start to sound really familiar now. You will begin to get the picture about what you need to do. Use this better understanding to now build your plan of what steps you will start to take to deal with this incident. When CERT mentions security patches, figure out where these will come from and how you apply them. If you just re-install the operating system, but don't apply the necessary security patches, you're likely to have the attackers right back in your system again in no time. You DON'T want this.
In some case, it is easiest just to re-install the operating system from scratch. Make sure you have good backups of your users' home directories and data files, since a re-installation may wipe out the entire hard drive and you'll have to put your home directories back on later. This is where that copy of the /etc/passwd file, /etc/group file, and so forth, come in handy.
In other cases, you may be able to mount the original operating system CD-ROM and compare files by hand (make sure you use a program other than "sum", since the CERT advisory on "ongoing network monitoring attacks" that you read already told you that attackers modify this program to hide their "trojan horse" versions of programs like "ps" and "netstat". A good program to use is md5 (e.g., in Linux, you can use the "md5sum" program to get an MD5 hash. See "man md5sum".).
Now start planning for setting up the system a little tighter than it was before. Take another few minutes and read the following:
Consider how you set up your accounts, how you choose your passwords, whether it really is a good idea to run sendmail on your workstation instead of using an email server on a centrally administered system, and what kinds of network services you leave enabled. This document refers you to a checklist for securing your system written by AUSCERT, the Australian CERT organization. UW Technology has also produced a similar checklist, although not quite as comprehensive and detailed as that of AUSCERT.
Now that you know what you need to do, make a short checklist and get your notes ready, then go ahead and get to work. You've got a lot of things to do, but you should have a pretty good picture in your mind of what is ahead of you.
If you cannot even get into the root account to start, you will probably need to boot the system from a floppy or CD-ROM. If you have any questions about this, or need any other assistance, send email to security@u.washington.edu.
