Remove Log Files Older Than 30 Days: Advanced File Management with Find -exec

In production environments, log files can accumulate rapidly, consuming valuable disk space and potentially impacting system performance. While log files older than 7 days might be candidates for compression, files older than 30 days are often safe to remove entirely. The find command with the -exec option provides powerful and flexible file management capabilities that go beyond simple deletion.

Understanding Find with -exec

What is -exec?

The -exec option in the find command allows you to execute a command on each file that matches your search criteria. This provides fine-grained control over file operations and enables complex workflows that simple deletion cannot achieve.

The `-exec` option is more powerful than using `-delete` because it allows you to perform any command on matched files, not just deletion.

Basic Command Structure

sudo find /path/to/folder -type f -name "*.log" -mtime +30 -exec rm -f {} \;

Command breakdown:

sudo - Required for system directories like /var/log
find - The base command to search for files
/path/to/folder - Target directory (e.g., /var/log)
-type f - Match only files (not directories)
-name "*.log" - Filter files ending with .log
-mtime +30 - Files modified more than 30 days ago
-exec rm -f {} \; - Execute rm -f on each matched file

Understanding the -exec Syntax

The -exec syntax has specific components:

-exec command {} \;

command - The command to execute
{} - Placeholder for the current file path
\; - Terminates the command (escaped semicolon)

The `{}` placeholder and `\;` terminator are essential. Missing either will cause the command to fail.

Advanced File Operations with -exec

Safe Deletion with Logging

# Delete with logging
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'echo "Deleting: $1" && rm -f "$1"' _ {} \;

# Delete with detailed logging
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'echo "[$(date)] Deleting: $1 ($(du -h "$1" | cut -f1))" >> /var/log/cleanup.log && rm -f "$1"' _ {} \;

Conditional Operations

# Only delete files larger than 1MB
sudo find /var/log -type f -name "*.log" -mtime +30 -size +1M -exec rm -f {} \;

# Delete only if file is not currently open
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'if ! lsof "$1" >/dev/null 2>&1; then rm -f "$1"; fi' _ {} \;

Archive Before Delete

# Move to archive before deleting
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'mkdir -p /var/log/archive && mv "$1" /var/log/archive/' _ {} \;

# Compress before archiving
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'gzip "$1" && mv "$1.gz" /var/log/archive/' _ {} \;

Practical Examples

Example 1: Basic Log Cleanup

# Simple deletion of old log files
sudo find /var/log -type f -name "*.log" -mtime +30 -exec rm -f {} \;

# With confirmation
sudo find /var/log -type f -name "*.log" -mtime +30 -exec rm -i {} \;

Example 2: Comprehensive Log Management

#!/bin/bash
# comprehensive_log_cleanup.sh

LOG_DIR="/var/log"
ARCHIVE_DIR="/var/log/archive"
DAYS_OLD=30

# Create archive directory
mkdir -p "$ARCHIVE_DIR"

# Function to log actions
log_action() {
    echo "[$(date)] $1" >> /var/log/cleanup.log
}

# Find and process old log files
find "$LOG_DIR" -type f -name "*.log" -mtime +$DAYS_OLD -exec sh -c '
    file="$1"
    size=$(du -h "$file" | cut -f1)
    log_action "Processing: $file ($size)"
    
    # Compress the file
    gzip "$file"
    log_action "Compressed: $file"
    
    # Move to archive
    mv "$file.gz" "$ARCHIVE_DIR/"
    log_action "Archived: $file.gz"
' _ {} \;

log_action "Log cleanup completed"

Example 3: Selective Cleanup by Log Type

# Different retention policies for different log types
# System logs: 30 days
sudo find /var/log -name "syslog*" -mtime +30 -exec rm -f {} \;

# Application logs: 60 days
sudo find /var/log -name "app*.log" -mtime +60 -exec rm -f {} \;

# Security logs: 90 days
sudo find /var/log -name "auth*" -mtime +90 -exec rm -f {} \;

# Web server logs: 30 days
sudo find /var/log -name "access.log*" -mtime +30 -exec rm -f {} \;
sudo find /var/log -name "error.log*" -mtime +30 -exec rm -f {} \;

Safety and Best Practices

Always Preview Before Deleting

# Dry run - see what would be deleted
find /var/log -type f -name "*.log" -mtime +30 -exec ls -lh {} \;

# Count files that would be deleted
find /var/log -type f -name "*.log" -mtime +30 | wc -l

# Show total size that would be freed
find /var/log -type f -name "*.log" -mtime +30 -exec du -ch {} + | tail -1

Implement Safety Checks

#!/bin/bash
# safe_log_cleanup.sh

LOG_DIR="/var/log"
DAYS_OLD=30
DRY_RUN=false

# Parse command line arguments
if [ "$1" = "--dry-run" ]; then
    DRY_RUN=true
fi

# Function to safely delete files
safe_delete() {
    local file="$1"
    
    # Check if file exists
    if [ ! -f "$file" ]; then
        echo "File not found: $file"
        return 1
    fi
    
    # Check if file is currently open
    if lsof "$file" >/dev/null 2>&1; then
        echo "Skipping open file: $file"
        return 1
    fi
    
    # Check file size
    local size=$(du -h "$file" | cut -f1)
    echo "Would delete: $file ($size)"
    
    if [ "$DRY_RUN" = false ]; then
        rm -f "$file"
        echo "Deleted: $file"
    fi
}

# Find and process files
find "$LOG_DIR" -type f -name "*.log" -mtime +$DAYS_OLD -exec sh -c 'safe_delete "$1"' _ {} \;

Log All Operations

# Create a log of all deletions
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'echo "[$(date)] Deleting: $1 ($(du -h "$1" | cut -f1))" >> /var/log/cleanup.log && rm -f "$1"' _ {} \;

# View cleanup log
tail -f /var/log/cleanup.log

Advanced Techniques

Using -exec with Multiple Commands

# Execute multiple commands on each file
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c '
    echo "Processing: $1"
    size=$(du -h "$1" | cut -f1)
    echo "Size: $size"
    gzip "$1"
    echo "Compressed: $1"
    mv "$1.gz" /var/log/archive/
    echo "Archived: $1.gz"
' _ {} \;

Conditional Processing

# Only process files larger than 10MB
sudo find /var/log -type f -name "*.log" -mtime +30 -size +10M -exec rm -f {} \;

# Skip files that are currently being written to
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'if [ ! -w "$1" ]; then rm -f "$1"; fi' _ {} \;

Batch Processing

# Process files in batches to avoid command line length limits
find /var/log -type f -name "*.log" -mtime +30 -print0 | xargs -0 -n 100 rm -f

# Or use -exec with xargs
find /var/log -type f -name "*.log" -mtime +30 -exec rm -f {} +

Automation and Scheduling

Cron Job Setup

# Add to crontab for daily cleanup
sudo crontab -e

# Run daily at 2 AM
0 2 * * * /usr/local/bin/log_cleanup.sh >> /var/log/cleanup.log 2>&1

# Run weekly with dry-run first
0 2 * * 0 /usr/local/bin/log_cleanup.sh --dry-run >> /var/log/cleanup.log 2>&1
0 3 * * 0 /usr/local/bin/log_cleanup.sh >> /var/log/cleanup.log 2>&1

Systemd Timer (Alternative to Cron)

# Create systemd service
sudo nano /etc/systemd/system/log-cleanup.service

[Unit]
Description=Log Cleanup Service
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/log_cleanup.sh
User=root

# Create systemd timer
sudo nano /etc/systemd/system/log-cleanup.timer

[Unit]
Description=Run log cleanup daily
Requires=log-cleanup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

# Enable and start timer
sudo systemctl enable log-cleanup.timer
sudo systemctl start log-cleanup.timer

Monitoring and Alerting

Disk Space Monitoring

# Check disk usage before and after cleanup
df -h /var/log

# Set up alerts for high disk usage
#!/bin/bash
# disk_alert.sh

THRESHOLD=80
USAGE=$(df /var/log | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$USAGE" -gt "$THRESHOLD" ]; then
    echo "Warning: /var/log is ${USAGE}% full" | mail -s "Disk Space Alert" admin@example.com
fi

Cleanup Reporting

#!/bin/bash
# cleanup_report.sh

LOG_DIR="/var/log"
DAYS_OLD=30

# Generate cleanup report
echo "Log Cleanup Report - $(date)" > /tmp/cleanup_report.txt
echo "=================================" >> /tmp/cleanup_report.txt
echo "" >> /tmp/cleanup_report.txt

# Count files to be deleted
COUNT=$(find "$LOG_DIR" -type f -name "*.log" -mtime +$DAYS_OLD | wc -l)
echo "Files older than $DAYS_OLD days: $COUNT" >> /tmp/cleanup_report.txt

# Calculate total size
SIZE=$(find "$LOG_DIR" -type f -name "*.log" -mtime +$DAYS_OLD -exec du -ch {} + 2>/dev/null | tail -1)
echo "Total size: $SIZE" >> /tmp/cleanup_report.txt

# Send report
mail -s "Log Cleanup Report" admin@example.com < /tmp/cleanup_report.txt

Common Pitfalls and Solutions

Pitfall 1: Command Line Length Limits

Problem: Too many files causing command line to exceed limits Solution: Use xargs or -exec ... + for batch processing

# Use xargs for large numbers of files
find /var/log -type f -name "*.log" -mtime +30 -print0 | xargs -0 rm -f

# Use -exec ... + for batch processing
find /var/log -type f -name "*.log" -mtime +30 -exec rm -f {} +

Pitfall 2: Deleting Active Log Files

Problem: Accidentally deleting files that applications are writing to Solution: Check if files are open before deleting

# Check if file is open before deleting
sudo find /var/log -type f -name "*.log" -mtime +30 -exec sh -c 'if ! lsof "$1" >/dev/null 2>&1; then rm -f "$1"; fi' _ {} \;

Pitfall 3: Insufficient Permissions

Problem: Permission denied errors when deleting files Solution: Use sudo and check file ownership

# Check file ownership and permissions
find /var/log -type f -name "*.log" -mtime +30 -exec ls -la {} \;

# Use sudo for system directories
sudo find /var/log -type f -name "*.log" -mtime +30 -exec rm -f {} \;

Conclusion

Using find with -exec provides powerful and flexible file management capabilities for log cleanup operations. The key to successful log management is implementing proper safety checks, logging all operations, and automating the process while maintaining oversight.

Key takeaways:

Always preview before deleting - Use dry-run mode to see what will be affected
Implement safety checks - Verify files aren't in use before deletion
Log all operations - Keep records of what was deleted and when
Use appropriate retention policies - Different log types need different retention periods
Automate with caution - Test thoroughly before implementing automated cleanup

Remember: The goal is not just to free disk space, but to maintain a clean, organized, and compliant log management system that supports your operational needs.

Table of Contents