Lesson 9 Incident Response & Reporting

55 minutes

Incident Response & Reporting

Effective incident response is crucial for minimizing damage from security breaches and maintaining business continuity. This lesson covers incident response frameworks, processes, and professional reporting techniques.

Incident Response Lifecycle

1. Preparation

Establish IR team, policies, and procedures

2. Identification

Detect and analyze potential incidents

3. Containment

Limit damage and prevent spread

4. Eradication

Remove threats and vulnerabilities

5. Recovery

Restore systems and operations

6. Lessons Learned

Review and improve processes

NIST Incident Response Framework

The NIST Computer Security Incident Handling Guide (SP 800-61) provides a comprehensive framework for incident response:

Key Activities:

Establish incident response team and roles
Develop incident response policies and procedures
Set up communication channels and escalation paths
Deploy monitoring and detection tools
Create incident response toolkit
Conduct training and tabletop exercises

Documentation Required:

Incident Response Plan
Contact lists and communication templates
Network diagrams and asset inventories
Baseline system configurations

Detection Methods:

Automated monitoring systems (SIEM, IDS/IPS)
User reports and help desk tickets
Threat intelligence feeds
External notifications (vendors, law enforcement)

Analysis Process:

Validate the incident
Determine incident scope and impact
Classify incident severity
Document initial findings
Notify appropriate stakeholders

Containment Strategies:

Short-term: Immediate isolation of affected systems
Long-term: Temporary solutions while preparing permanent fixes
Network segmentation and access controls
System quarantine and traffic blocking

Eradication and Recovery:

Remove malware and close vulnerabilities
Apply security patches and updates
Restore systems from clean backups
Implement additional monitoring
Gradually return systems to production

Incident Classification and Severity

Severity	Impact	Response Time	Examples
Critical	Severe business disruption, data breach	Immediate (0-1 hours)	Ransomware, active data exfiltration
High	Significant service impact	4 hours	System compromise, malware infection
Medium	Limited service impact	24 hours	Policy violations, suspicious activity
Low	Minimal or no service impact	72 hours	Failed login attempts, spam

Digital Forensics and Evidence Collection

Evidence Collection

Volatile Data: Memory dumps, network connections
System Images: Complete disk copies
Log Files: System, application, and security logs
Network Data: Packet captures, flow records
Documentation: Screenshots, photographs

Chain of Custody

Document who, what, when, where, why
Use tamper-evident storage and labeling
Maintain continuous documentation
Limit access to authorized personnel
Use cryptographic hashes for integrity

Python Incident Response Toolkit

#!/usr/bin/env python3
import os
import json
import hashlib
import subprocess
import platform
from datetime import datetime
import psutil
import logging

class IncidentResponseToolkit:
    def __init__(self, case_id, output_dir="ir_investigation"):
        self.case_id = case_id
        self.output_dir = f"{output_dir}_{case_id}"
        self.evidence_log = []
        
        # Create output directory structure
        self.create_directory_structure()
        
        # Set up logging
        logging.basicConfig(
            level=logging.INFO,
            format='%(asctime)s - %(levelname)s - %(message)s',
            handlers=[
                logging.FileHandler(f"{self.output_dir}/investigation.log"),
                logging.StreamHandler()
            ]
        )
        self.logger = logging.getLogger(__name__)
        
        self.logger.info(f"Incident Response investigation started for case: {case_id}")
    
    def create_directory_structure(self):
        """Create standard IR directory structure"""
        directories = [
            self.output_dir,
            f"{self.output_dir}/evidence",
            f"{self.output_dir}/logs",
            f"{self.output_dir}/memory",
            f"{self.output_dir}/network",
            f"{self.output_dir}/system",
            f"{self.output_dir}/reports"
        ]
        
        for directory in directories:
            os.makedirs(directory, exist_ok=True)
    
    def collect_system_info(self):
        """Collect basic system information"""
        self.logger.info("Collecting system information...")
        
        system_info = {
            'case_id': self.case_id,
            'timestamp': datetime.now().isoformat(),
            'hostname': platform.node(),
            'operating_system': platform.system(),
            'os_version': platform.version(),
            'architecture': platform.architecture(),
            'processor': platform.processor(),
            'boot_time': datetime.fromtimestamp(psutil.boot_time()).isoformat(),
            'current_users': [user.name for user in psutil.users()]
        }
        
        # Save system info
        with open(f"{self.output_dir}/system/system_info.json", 'w') as f:
            json.dump(system_info, f, indent=2)
        
        self.add_evidence_entry("System Information", "system/system_info.json", "System metadata collection")
        return system_info
    
    def collect_running_processes(self):
        """Collect information about running processes"""
        self.logger.info("Collecting running processes...")
        
        processes = []
        for proc in psutil.process_iter(['pid', 'name', 'username', 'cmdline', 'create_time', 'memory_info']):
            try:
                proc_info = proc.info
                proc_info['create_time'] = datetime.fromtimestamp(proc_info['create_time']).isoformat()
                proc_info['memory_rss'] = proc_info['memory_info'].rss if proc_info['memory_info'] else 0
                del proc_info['memory_info']  # Remove complex object
                processes.append(proc_info)
            except (psutil.NoSuchProcess, psutil.AccessDenied):
                continue
        
        # Save process list
        with open(f"{self.output_dir}/system/processes.json", 'w') as f:
            json.dump(processes, f, indent=2)
        
        # Create human-readable process list
        with open(f"{self.output_dir}/system/processes.txt", 'w') as f:
            f.write("RUNNING PROCESSES\n")
            f.write("=" * 50 + "\n")
            f.write(f"{'PID':<8} {'Name':<20} {'User':<15} {'Memory (MB)':<12} {'Command'}\n")
            f.write("-" * 80 + "\n")
            
            for proc in sorted(processes, key=lambda x: x.get('memory_rss', 0), reverse=True):
                memory_mb = proc.get('memory_rss', 0) / (1024 * 1024)
                cmdline = ' '.join(proc.get('cmdline', [])) if proc.get('cmdline') else ''
                f.write(f"{proc.get('pid', 'N/A'):<8} {proc.get('name', 'N/A'):<20} "
                       f"{proc.get('username', 'N/A'):<15} {memory_mb:<12.2f} {cmdline[:40]}\n")
        
        self.add_evidence_entry("Running Processes", "system/processes.json", f"{len(processes)} processes captured")
        return processes
    
    def collect_network_connections(self):
        """Collect active network connections"""
        self.logger.info("Collecting network connections...")
        
        connections = []
        for conn in psutil.net_connections(kind='inet'):
            conn_info = {
                'fd': conn.fd,
                'family': str(conn.family),
                'type': str(conn.type),
                'local_address': f"{conn.laddr.ip}:{conn.laddr.port}" if conn.laddr else None,
                'remote_address': f"{conn.raddr.ip}:{conn.raddr.port}" if conn.raddr else None,
                'status': conn.status,
                'pid': conn.pid
            }
            
            # Try to get process name
            if conn.pid:
                try:
                    proc = psutil.Process(conn.pid)
                    conn_info['process_name'] = proc.name()
                except (psutil.NoSuchProcess, psutil.AccessDenied):
                    conn_info['process_name'] = 'Unknown'
            
            connections.append(conn_info)
        
        # Save network connections
        with open(f"{self.output_dir}/network/connections.json", 'w') as f:
            json.dump(connections, f, indent=2)
        
        # Create human-readable connections list
        with open(f"{self.output_dir}/network/connections.txt", 'w') as f:
            f.write("ACTIVE NETWORK CONNECTIONS\n")
            f.write("=" * 50 + "\n")
            f.write(f"{'Protocol':<8} {'Local Address':<22} {'Remote Address':<22} {'Status':<12} {'PID':<8} {'Process'}\n")
            f.write("-" * 90 + "\n")
            
            for conn in connections:
                protocol = "TCP" if "SOCK_STREAM" in conn['type'] else "UDP"
                f.write(f"{protocol:<8} {conn.get('local_address', 'N/A'):<22} "
                       f"{conn.get('remote_address', 'N/A'):<22} {conn.get('status', 'N/A'):<12} "
                       f"{conn.get('pid', 'N/A'):<8} {conn.get('process_name', 'N/A')}\n")
        
        self.add_evidence_entry("Network Connections", "network/connections.json", f"{len(connections)} connections captured")
        return connections
    
    def collect_system_logs(self):
        """Collect relevant system logs"""
        self.logger.info("Collecting system logs...")
        
        log_files = []
        
        # Common log locations by OS
        if platform.system() == "Linux":
            potential_logs = [
                "/var/log/syslog",
                "/var/log/auth.log",
                "/var/log/secure",
                "/var/log/messages",
                "/var/log/kern.log"
            ]
        elif platform.system() == "Windows":
            # On Windows, we would typically use Windows Event Log API
            # This is a simplified approach
            potential_logs = [
                "C:\\Windows\\System32\\winevt\\Logs\\Security.evtx",
                "C:\\Windows\\System32\\winevt\\Logs\\System.evtx",
                "C:\\Windows\\System32\\winevt\\Logs\\Application.evtx"
            ]
        else:
            potential_logs = []
        
        for log_path in potential_logs:
            if os.path.exists(log_path) and os.access(log_path, os.R_OK):
                try:
                    # Copy log file with timestamp
                    log_name = os.path.basename(log_path)
                    dest_path = f"{self.output_dir}/logs/{log_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
                    
                    # For large log files, copy recent entries only
                    if os.path.getsize(log_path) > 100 * 1024 * 1024:  # 100MB
                        self.logger.info(f"Large log file detected: {log_path}, copying recent entries only")
                        self.copy_recent_log_entries(log_path, dest_path, lines=10000)
                    else:
                        subprocess.run(['cp', log_path, dest_path], check=True)
                    
                    log_files.append(dest_path)
                    self.add_evidence_entry(f"System Log: {log_name}", dest_path, f"System log file from {log_path}")
                    
                except Exception as e:
                    self.logger.error(f"Failed to copy log {log_path}: {e}")
        
        return log_files
    
    def copy_recent_log_entries(self, source_path, dest_path, lines=10000):
        """Copy recent entries from large log files"""
        try:
            # Use tail command to get recent entries
            with open(dest_path, 'w') as dest_file:
                subprocess.run(['tail', '-n', str(lines), source_path], stdout=dest_file, check=True)
        except Exception as e:
            self.logger.error(f"Failed to copy recent log entries: {e}")
    
    def collect_file_hashes(self, target_directories=None):
        """Collect file hashes for integrity verification"""
        self.logger.info("Collecting file hashes...")
        
        if target_directories is None:
            # Default directories to hash
            if platform.system() == "Linux":
                target_directories = ["/bin", "/sbin", "/usr/bin", "/usr/sbin"]
            elif platform.system() == "Windows":
                target_directories = ["C:\\Windows\\System32"]
            else:
                target_directories = []
        
        file_hashes = {}
        
        for directory in target_directories:
            if os.path.exists(directory):
                self.logger.info(f"Hashing files in {directory}...")
                for root, dirs, files in os.walk(directory):
                    for file in files[:100]:  # Limit to first 100 files per directory
                        file_path = os.path.join(root, file)
                        try:
                            if os.path.isfile(file_path) and os.access(file_path, os.R_OK):
                                file_hash = self.calculate_file_hash(file_path)
                                file_hashes[file_path] = {
                                    'sha256': file_hash,
                                    'size': os.path.getsize(file_path),
                                    'modified': datetime.fromtimestamp(os.path.getmtime(file_path)).isoformat()
                                }
                        except Exception as e:
                            self.logger.debug(f"Could not hash {file_path}: {e}")
        
        # Save file hashes
        with open(f"{self.output_dir}/system/file_hashes.json", 'w') as f:
            json.dump(file_hashes, f, indent=2)
        
        self.add_evidence_entry("File Hashes", "system/file_hashes.json", f"{len(file_hashes)} files hashed")
        return file_hashes
    
    def calculate_file_hash(self, file_path):
        """Calculate SHA256 hash of a file"""
        sha256_hash = hashlib.sha256()
        try:
            with open(file_path, "rb") as f:
                for chunk in iter(lambda: f.read(4096), b""):
                    sha256_hash.update(chunk)
            return sha256_hash.hexdigest()
        except Exception as e:
            self.logger.debug(f"Hash calculation failed for {file_path}: {e}")
            return None
    
    def add_evidence_entry(self, name, file_path, description):
        """Add entry to evidence log"""
        evidence_entry = {
            'timestamp': datetime.now().isoformat(),
            'name': name,
            'file_path': file_path,
            'description': description,
            'file_hash': self.calculate_file_hash(os.path.join(self.output_dir, file_path)) if os.path.exists(os.path.join(self.output_dir, file_path)) else None,
            'collected_by': os.getenv('USER', 'unknown')
        }
        
        self.evidence_log.append(evidence_entry)
    
    def generate_timeline(self):
        """Generate timeline of collected evidence"""
        self.logger.info("Generating evidence timeline...")
        
        timeline_entries = []
        
        # Add evidence collection events
        for evidence in self.evidence_log:
            timeline_entries.append({
                'timestamp': evidence['timestamp'],
                'event_type': 'Evidence Collection',
                'description': f"Collected: {evidence['name']}",
                'source': 'IR Toolkit'
            })
        
        # Sort by timestamp
        timeline_entries.sort(key=lambda x: x['timestamp'])
        
        # Save timeline
        with open(f"{self.output_dir}/reports/timeline.json", 'w') as f:
            json.dump(timeline_entries, f, indent=2)
        
        # Create human-readable timeline
        with open(f"{self.output_dir}/reports/timeline.txt", 'w') as f:
            f.write("INCIDENT RESPONSE TIMELINE\n")
            f.write("=" * 50 + "\n")
            f.write(f"Case ID: {self.case_id}\n")
            f.write(f"Generated: {datetime.now().isoformat()}\n\n")
            
            for entry in timeline_entries:
                f.write(f"{entry['timestamp']} - {entry['event_type']}\n")
                f.write(f"  {entry['description']}\n")
                f.write(f"  Source: {entry['source']}\n\n")
        
        return timeline_entries
    
    def generate_incident_report(self):
        """Generate comprehensive incident response report"""
        self.logger.info("Generating incident response report...")
        
        report = {
            'case_id': self.case_id,
            'report_generated': datetime.now().isoformat(),
            'investigator': os.getenv('USER', 'unknown'),
            'evidence_collected': len(self.evidence_log),
            'evidence_log': self.evidence_log,
            'system_summary': {
                'hostname': platform.node(),
                'os': f"{platform.system()} {platform.version()}",
                'architecture': platform.architecture()[0]
            }
        }
        
        # Save detailed report
        with open(f"{self.output_dir}/reports/incident_report.json", 'w') as f:
            json.dump(report, f, indent=2)
        
        # Create executive summary
        with open(f"{self.output_dir}/reports/executive_summary.txt", 'w') as f:
            f.write("INCIDENT RESPONSE EXECUTIVE SUMMARY\n")
            f.write("=" * 50 + "\n")
            f.write(f"Case ID: {self.case_id}\n")
            f.write(f"Investigation Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
            f.write(f"Investigator: {report['investigator']}\n")
            f.write(f"Target System: {report['system_summary']['hostname']}\n")
            f.write(f"Operating System: {report['system_summary']['os']}\n")
            f.write(f"Evidence Items Collected: {report['evidence_collected']}\n\n")
            
            f.write("EVIDENCE SUMMARY:\n")
            f.write("-" * 20 + "\n")
            for evidence in self.evidence_log:
                f.write(f"• {evidence['name']}: {evidence['description']}\n")
            
            f.write(f"\nAll evidence has been collected and stored in: {self.output_dir}\n")
            f.write("Chain of custody has been maintained throughout the investigation.\n")
        
        self.logger.info(f"Incident response investigation completed for case: {self.case_id}")
        self.logger.info(f"All evidence stored in: {self.output_dir}")
        
        return report
    
    def run_full_collection(self):
        """Run complete evidence collection process"""
        self.logger.info("Starting full evidence collection...")
        
        try:
            # Collect all available evidence
            self.collect_system_info()
            self.collect_running_processes()
            self.collect_network_connections()
            self.collect_system_logs()
            self.collect_file_hashes()
            
            # Generate reports
            self.generate_timeline()
            self.generate_incident_report()
            
            print(f"\n{'='*50}")
            print("INCIDENT RESPONSE COLLECTION COMPLETED")
            print(f"{'='*50}")
            print(f"Case ID: {self.case_id}")
            print(f"Evidence Directory: {self.output_dir}")
            print(f"Evidence Items: {len(self.evidence_log)}")
            print("Generated Reports:")
            print("- incident_report.json (detailed findings)")
            print("- executive_summary.txt (summary report)")
            print("- timeline.txt (evidence timeline)")
            
        except Exception as e:
            self.logger.error(f"Evidence collection failed: {e}")
            raise

def main():
    if len(sys.argv) != 2:
        print("Usage: python3 incident_response.py ")
        print("Example: python3 incident_response.py CASE-2024-001")
        sys.exit(1)
    
    case_id = sys.argv[1]
    
    print("Incident Response Toolkit")
    print(f"Case ID: {case_id}")
    print("=" * 40)
    
    # Initialize IR toolkit
    ir_toolkit = IncidentResponseToolkit(case_id)
    
    try:
        # Run full evidence collection
        ir_toolkit.run_full_collection()
        
    except KeyboardInterrupt:
        print("\nCollection interrupted by user")
    except Exception as e:
        print(f"Collection failed: {e}")

if __name__ == "__main__":
    import sys
    main()

Professional Reporting Standards

Report Structure Best Practices

Executive Summary

High-level incident overview
Business impact assessment
Key findings and recommendations
Resource requirements

Technical Details

Detailed timeline of events
Technical analysis and evidence
Attack vectors and methods
Remediation steps taken

Communication and Stakeholder Management

Stakeholder	Information Needed	Communication Method	Frequency
Executive Leadership	Business impact, costs, timeline	Executive briefings, dashboards	Initial + daily updates
IT Operations	Technical details, system status	Technical calls, email updates	Continuous during response
Legal/Compliance	Regulatory implications, evidence	Secure communications	As needed
Public Relations	Public disclosure requirements	Prepared statements	If public disclosure needed

Post-Incident Activities

Lessons Learned Review

Conduct post-incident review meeting
Document what worked well
Identify areas for improvement
Update incident response procedures
Enhance detection capabilities
Conduct additional training if needed

Security Improvements

Implement additional security controls
Update security policies
Enhance monitoring and alerting
Conduct vulnerability assessments
Update incident response plan
Share threat intelligence

Exercise

Incident Response Simulation

Objective: Practice incident response procedures using a simulated security incident.

Scenario Setup:
- Create a simulated security incident scenario
- Define roles and responsibilities
- Set up communication channels
Evidence Collection:
- Run the Python IR toolkit on a test system
- Practice proper evidence handling procedures
- Document chain of custody
Analysis and Reporting:
- Analyze collected evidence
- Create incident timeline
- Write executive summary and technical report
- Present findings to stakeholders
Lessons Learned:
- Conduct post-incident review
- Identify process improvements
- Update procedures based on findings

Use only test systems and simulated scenarios for this exercise.

Legal and Regulatory Considerations

Important Legal Aspects

Notification Requirements: Many jurisdictions require breach notification within specific timeframes
Evidence Preservation: Legal hold requirements may apply to prevent evidence destruction
Privacy Laws: GDPR, CCPA, and other privacy regulations may impact response procedures
Industry Regulations: HIPAA, PCI DSS, SOX, and other sector-specific requirements
Law Enforcement: When and how to engage law enforcement agencies
Insurance Claims: Cyber insurance may have specific documentation requirements

Lesson 9 of 10

Lessons