Effective management of telemetry data within high-concurrency cloud environments necessitates a robust log rotation strategy. In massive-scale deployments, such as global content delivery networks or industrial energy monitoring systems, the continuous ingestion of request headers and access data generates substantial write overhead. Without a systematic rotation strategy, disk saturation becomes an inevitability; this leads to catastrophic packet-loss, service degradation, and potential hardware failures due to excessive storage controller thermal-inertia. Nginx Log Rotation serves as the primary mechanism for maintaining system health by atomizing log files into manageable segments without disrupting the active worker processes or increasing application latency. This manual defines the operational standard for implementing the logrotate utility alongside Nginx, ensuring that the transition between log segments remains idempotent and transparent to the end-user. By leveraging the USR1 signal, administrators can force the Nginx master process to reopen its file handles: a critical step in preserving the integrity of the data payload while preventing the accumulation of massive, unmanageable flat files that can cripple backup throughput.
Technical Specifications
| Requirement | Default Operating Range | Protocol/Standard | Impact Level (1-10) | Recommended Resources |
| :— | :— | :— | :— | :— |
| Nginx Core | 1.18.x to 1.25.x | POSIX Signals | 10 | 2+ CPU Cores |
| Logrotate Binary | 3.8.0+ | Cron / Systemd Timer | 8 | 512MB RAM Min |
| Filesystem | ext4 / XFS / ZFS | IEEE 1003.1 | 9 | High IOPS SSD |
| Disk Space | 2x Log Peak | Inode Management | 7 | 20% Headroom |
| User Permissions | UID 0 or Sudo | Linux ACL | 9 | Root/Nginx User |
Configuration Protocol
Environment Prerequisites:
Before execution, verify the system environment meets the following baseline requirements:
1. Nginx version 1.10 or higher must be compiled with standard modules.
2. The logrotate package must be installed and linked to cron.daily or a systemd timer.
3. Access to root or a user within the sudo group is mandatory to modify /etc/logrotate.d/.
4. Storage volumes must have at least 15 percent free space to allow for the overhead of temporary compression tasks.
5. All file paths must comply with standard Linux directory structures, specifically /var/log/nginx/.
Section A: Implementation Logic:
The logic of Nginx log rotation centers on the Linux kernel’s handling of file descriptors. When a process opens a file, it tracks the file via an inode rather than a filename. If a log file is renamed using the mv command, Nginx continues to write to that same file under its new name because the file descriptor remains unchanged. This behavior is essential for zero-downtime operations. The engineering goal is to move the old log file, create a new empty file, and then instruct Nginx to switch its file descriptors to the new path. This is achieved through the USR1 signal. Unlike a full restart or reload, the USR1 signal tells the Nginx worker processes to reopen all currently open log files. This process is highly efficient and does not drop active connections or increase request latency. It ensures that the transition is atomic, preventing any loss of the log payload during high-concurrency traffic spikes.
Step-By-Step Execution
Step 1: Evaluate Current Logging Footprint
Command: du -sh /var/log/nginx/.
System Note: This command allows the architect to calculate the current storage throughput and determine the necessary rotation frequency. It checks the cumulative size of the log directory to ensure that the partition is not nearing a critical saturation point.
Step 2: Create the Nginx Rotation Definition
Command: nano /etc/logrotate.d/nginx.
System Note: By creating a specific configuration file within the logrotate.d directory, we ensure the rotation logic is encapsulated and managed independently of other system services. This prevents configuration drift and allows for service-specific tuning.
Step 3: Define Rotation Parameters and Directives
In the configuration file, insert the following logic block:
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
endscript
}
System Note: The daily directive sets the frequency; while rotate 14 ensures two weeks of history. The delaycompress flag is vital; it keeps the most recently rotated file uncompressed to prevent the CPU from being taxed while log handles are still being resolved.
Step 4: Validate File Permissions and Ownership
Command: chown -R www-data:adm /var/log/nginx/ && chmod 755 /var/log/nginx/.
System Note: Correct ACLs are required for the logrotate utility to modify files without triggering security blocks. Improper permissions often lead to signal-attenuation where the rotation script fails to notify the Nginx master process of the change.
Step 5: Execute a Non-Destructive Dry Run
Command: logrotate -d /etc/logrotate.d/nginx.
System Note: The -d flag enters debug mode. It simulates the rotation process without actually moving files or sending signals. This is a critical audit step to identify syntax errors or pathing conflicts before the process is live in production.
Step 6: Force Initial Rotation Test
Command: logrotate -f /etc/logrotate.d/nginx.
System Note: The -f (force) flag overrides the internal timestamp check in /var/lib/logrotate/status. This forces the kernel to execute the postrotate script and verifies that Nginx successfully reopens its log handles in the new path.
Section B: Dependency Fault-Lines:
Software conflicts frequently arise when SELinux or AppArmor profiles are active. If logrotate lacks the security context to send signals to the Nginx process, the USR1 command will be silently blocked; resulting in Nginx continuing to write to the old, renamed log file until the disk space is exhausted. Another common bottleneck is I/O contention. During the compression of massive log files, the disk controller may experience high latency, which can delay the execution of the postrotate script. Furthermore, if the nginx.pid file path is incorrectly mapped in the configuration, the kill command will fail. This causes the log entries to be lost or misdirected to a deleted file handle, leading to a total loss of telemetry data for that cycle.
THE TROUBLESHOOTING MATRIX
Section C: Logs & Debugging:
When rotation fails, the first point of inspection is the logrotate status file located at /var/lib/logrotate/status. This file records the last execution time for every managed log. If the logs are not rotating, check for the error string “error: skipping /var/log/nginx/access.log because parent directory has insecure permissions.” This indicates that the directory is world-writable, triggering a safety abort in logrotate.
If Nginx is still writing to the old file (e.g., access.log.1), execute lsof | grep deleted. If the output shows Nginx worker processes holding a handle on a “deleted” file, the USR1 signal was not delivered. Verify the PID path using nginx -T | grep pid. If the PID is stored in /run/nginx.pid instead of /var/run/nginx.pid, update the rotation script immediately. For physical infrastructure monitoring, check the storage array for spikes in write-latency during the rotation window; this often points to a need for throttled compression using the nice or ionice commands.
OPTIMIZATION & HARDENING
Performance Tuning:
To minimize the impact on system throughput, avoid high-ratio compression (like xz or bzip2) during peak traffic hours. Stick to gzip for its low CPU overhead. For systems with extreme concurrency, use the size directive instead of daily. This triggers rotation once a file reaches a specific threshold (e.g., size 1G), preventing individual files from becoming so large that they impact the performance of log parsers or security information and event management (SIEM) collectors.
Security Hardening:
Ensure that log files are created with 0640 permissions. This allows the www-data user to write while restricting read access to the adm group and the root user. This prevents sensitive request payloads, which might contain session tokens or PII, from being accessible to unprivileged local users. Additionally, implement immutable attributes on historical logs using chattr +i if your auditors require a strict non-repudiation chain of custody.
Scaling Logic:
In a distributed architecture, local log rotation is often the first step in a telemetry pipeline. As traffic grows, local storage will become a bottleneck regardless of rotation frequency. In such cases, use logrotate to move files to a dedicated mount point, such as an NFS share or an S3 bucket-backed filesystem. Implementing a centralized logging agent like Fluentd or Logstash to “tail” the rotated files ensures that logs are offloaded from the edge node, maintaining low local disk overhead and high service availability.
THE ADMIN DESK
How do I prevent log loss during rotation?
Use the USR1 signal method rather than copytruncate. The copytruncate method has a microsecond gap between copying and truncating where data is lost; whereas USR1 allows Nginx to switch handles with zero payload loss.
What happens if the rotate script fails?
If the postrotate script fails, Nginx continues writing to the renamed file (e.g., access.log.1). The system remains operational, but the logs will not be correctly archived, and disk space will not be reclaimed until the signal is sent.
Can I rotate logs every hour?
Yes. Move the nginx configuration from /etc/logrotate.d/ to /etc/logrotate.hourly/ and ensure that the system crontab or systemd-timer is configured to execute the hourly rotation bin. This is recommended for high-throughput ingress controllers.
Why is my log file empty after rotation?
This usually occurs if the create directive is missing or if the Nginx user lacks write permissions to the directory. Nginx cannot create its own log files; it requires the file to exist or the parent directory to be writable.
Is it safe to compress logs immediately?
It is safer to use delaycompress. This ensures that the log file from the previous cycle is not compressed until the next cycle begins; allowing Nginx ample time to finish writing any buffered data in its internal pipes.
