Race condition in size-triggered log rotation in Native logger
Description
Environment
Activity
Scott CantorApril 20, 2018 at 9:48 PM
Per the original plan, the default logging for mod_shib was switched to local syslog and should work out of the box on all non-Windows platforms assuming syslog is available. Plus there's the echoing of much of it to Apache's error log.
Scott CantorNovember 15, 2017 at 11:00 PM
It's fine when specific package maintainers handle it for their platforms, but at the upstream level it just becomes unmanageable unfortunately. It always seems simple but then all the details creep in.
I'm fine with whatever Ferenc wants to do for Debian, and SUSE and so on, but for us I think our best choice is to make it as little used as we can manage.
Vlad MenclNovember 15, 2017 at 9:59 PM
Agree.
I just found I had to make another tweak on the Debian script - "service" lives in /usr/sbin and not /sbin.
Agree this would not be manageable cross-platform (though I'm still trying to do that within Tuakiri). But not from a package maintainer's point of view, agree with that.
Cheers,
Vlad
Scott CantorNovember 15, 2017 at 7:47 PM
That's why syslog is the answer.
Ideally I'd like to get native.log down to the point that it is essentially silent and never used at all outside of very rare instances, mostly debugging the RequestMap (which is not an Apache need, really).
Vlad MenclNovember 15, 2017 at 7:40 PM
Hi,
Just FYI: when I was testing the same (moving log rotation to logrotate) on Ubuntu, I found that while the logrotate.d file nicely slips in, having pos-rotate action
service httpd reloadwould not work - this has to be
service apache2 reloadSo I've ended up with two separate configuration snippets, using httpd on RedHat and apache2 on Debian+Ubuntu - my snippets are at https://github.com/REANNZ/Tuakiri-public/tree/master/shibboleth-sp
Just letting you know - in case this gets incorporated in, it would need a different logrotate.d file based on how Apache is called on the target system...
Cheers,
Vlad
Hi,
There is a race condition in the log rotation in the NativeSP module linked into Apache.
When Apache is reloading the configuration (which is also triggered as post-rotate action from /etc/logrotate.d/httpd on CentOS 6 and 7), all httpd forked children are active at the same time.
And all of them do some logging from mod_shib.
And if the native.log is just below the threshold for sized-based log rotation, the children try to rotate it each independently, stepping on each other's toes.
I've even seen:
ls -lA /var/log/shibboleth-www/ total 2240 -rw-r--r--. 1 root root 990963 Oct 26 15:31 native.log -rw-r--r--. 1 root root 135521 Oct 26 15:31 native_warn.logservice httpd reload
ls -lA /var/log/shibboleth-www/ total 2268 -rw-r--r--. 1 apache apache 4411 Oct 26 15:32 native.log -rw-r--r--. 1 apache apache 0 Oct 26 15:32 native.log.1 -rw-r--r--. 1 apache apache 4411 Oct 26 15:32 native.log.2 -rw-r--r--. 1 apache apache 4510 Oct 26 15:32 native.log.3 -rw-r--r--. 1 apache apache 2632 Oct 26 15:32 native.log.4 -rw-r--r--. 1 root root 1000462 Oct 26 15:32 native.log.5 -rw-r--r--. 1 root root 136656 Oct 26 15:32 native_warn.log(the host runs in NZ timezone; hence the future timestamps).
IMHO, the only way to get around this is to move the rotation out to external tools - eg, logrotate, as documented eg at https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPLogRotation
PS: and the log rotation also doesn't go well with SELinux: Apache is only allowed to append to logs, not move/rename/delete them. Another reason for not doing it in Apache.
PS2: this is different from https://shibboleth.atlassian.net/browse/SSPCPP-646#icft=SSPCPP-646