August 2nd, 2013

New Security Feature in Fedora 20 Part 1: Setroubleshoot - Journald integration

A little early to start this series, but I am pretty excited about all the new features coming in Fedora 20.

A while ago a fellow worker Máirín Duffy was very proud of herself for figuring out an SELinux issue.  Basically she had a problem that is fairly common struggle people have with SELinux.  She had created some content for her web server in her home directory, and then mv'd it into a /var/www subdir and tried to look at it in here web browser and got a screen like the following.

Screenshot from 2013-08-02 08:02:04

When she went to look at the /var/log/httpd/error_log, she saw something like:

# tail /var/log/httpd/error_log
[Fri Aug 02 08:05:43.347080 2013] [core:error] [pid 10556] (13)Permission denied: [client ::1:38045] AH00132: file permissions deny server access: /var/www/html/index.html

Instead of thinking there was a problem with Ownership or Permission flags, she thought of SELinux and  figured their might be  problem with the labels,  so she ran:

# restorecon -R -v /var/www
restorecon reset /var/www/html/index.html context unconfined_u:object_r:user_home_t:s0->unconfined_u:object_r:httpd_sys_content_t:s0

Well if she had continued to investigate she would have seen the following in other logs.

In the /var/log/audit/audit.log the kernel had reported an AVC message like the following.

# ausearch -m avc -ts recent -i
type=PATH msg=audit(08/02/2013 08:05:43.346:1197) : item=0 name=/var/www/html/index.html inode=3145858 dev=08:03 mode=file,644 ouid=root ogid=root rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0
type=CWD msg=audit(08/02/2013 08:05:43.346:1197) :  cwd=/
type=SYSCALL msg=audit(08/02/2013 08:05:43.346:1197) : arch=x86_64 syscall=open success=no exit=-13(Permission denied) a0=0x7f476595da40 a1=O_RDONLY|O_CLOEXEC a2=0x0 a3=0x7fffe27e11b0 items=1 ppid=10552
pid=10556 auid=unset uid=apache gid=apache euid=apache suid=apache fsuid=apache egid=apache sgid=apache fsgid=apache ses=unset tty=(none) comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(08/02/2013 08:05:43.346:1197) : avc:  denied { read } for  pid=10556 comm=httpd name=index.html dev="sda3" ino=3145858 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

SETroubleshoot received this message and reported in /var/log/messages.

# grep setroubleshoot /var/log/messages
Aug  2 08:01:46 redsox setroubleshoot: SELinux is preventing /usr/sbin/httpd from read access on the file /var/www/html/index.html. For complete SELinux messages. run sealert -l fd6b9022-1ced-4065-905a-8f0e884f9915

If she finally ran this sealert command specified in the log file she would have found setroubleshoot had written its analysis in /var/lib/setroubleshoot/setroubleshoot_database.xml

# sealert -l fd6b9022-1ced-4065-905a-8f0e884f9915

SELinux is preventing /usr/sbin/httpd from read access on the file /var/www/html/index.html.

*****  Plugin restorecon (92.2 confidence) suggests  *************************

If you want to fix the label.
/var/www/html/index.html default label should be httpd_sys_content_t.
Then you can run restorecon.
# /sbin/restorecon -v /var/www/html/index.html

Nice that Máirín figured out her problem so quickly, but as you see the system had scattered little clues all over the system to help her.  And who says SELinux is not easy.

Ok Dan get to the point?

If you have been following the Fedora-devel list their has been a large discussion going on about whether or not to enable syslog (rsyslog) in the default install.  The idea is to consolidate logging into journald which I talked about in a previous blog.  And get log analysis tools to start relying on its data rather then scanning /var/log/messages.

SETroubleshot integration with journald.

Back in Fedora 18 or 19 we started dumping setroubleshoot data into the journal.  The problem with this data is that it was setroubleshoot reporting on the httpd process, but there was no way for journald/systemd to know this.  I asked the journald team to add a mechanism for a privileged application to specify a Process ID (PID) to which the log message pertained.  Well systemd/journald has added this in systemd-206-1.  I added the functionality to setroubleshoot-3.2.11-1.fc20.x86_64

Now if Máirín had simply checked the status of the httpd service, she would see a message like:

# systemctl status httpd
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
   Active: active (running) since Fri 2013-08-02 08:01:35 EDT; 30min ago
 Main PID: 10552 (httpd)
   Status: "Total requests: 4; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─10552 /usr/sbin/httpd -DFOREGROUND
           ├─10553 /usr/libexec/nss_pcache 196611 off /etc/httpd/alias
           ├─10554 /usr/sbin/httpd -DFOREGROUND
           ├─10555 /usr/sbin/httpd -DFOREGROUND
           ├─10556 /usr/sbin/httpd -DFOREGROUND
           ├─10557 /usr/sbin/httpd -DFOREGROUND
           ├─10558 /usr/sbin/httpd -DFOREGROUND
           └─10569 /usr/sbin/httpd -DFOREGROUND

Aug 02 08:01:35 systemd[1]: Started The Apache HTTP Server.
Aug 02 08:01:46 python[10564]: SELinux is preventing /usr/sbin/httpd from read access on the file /va...html.
                                                              *****  Plugin restorecon (

Then if she used the journalctl command she could see the full analysis.

# journalctl  -r -o verbose -u httpd.service

-- Logs begin at Tue 2013-04-09 15:19:05 EDT, end at Fri 2013-08-02 08:32:35 EDT. --
Fri 2013-08-02 08:05:45 EDT [s=3087834a63d74580811c9e1088ac7fdf;i=1195d;b=71667fa0d7074a3f8bdf1c0da22fe234;m=22b877728;t=4e2f5c7552cb6;x=908
    _CMDLINE=/usr/bin/python -Es /usr/sbin/setroubleshootd -f
    MESSAGE=SELinux is preventing /usr/sbin/httpd from read access on the file /var/www/html/index.html.
            *****  Plugin restorecon (92.2 confidence) suggests  *************************
            If you want to fix the label.
            /var/www/html/index.html default label should be httpd_sys_content_t.
            Then you can run restorecon.
            # /sbin/restorecon -v /var/www/html/index.html
            *****  Plugin catchall_boolean (7.83 confidence) suggests  *******************
            If you want to allow httpd to read user content
            Then you must tell SELinux about this by enabling the 'httpd_read_user_content' boolean.
            You can read 'user_selinux' man page for more details.
            setsebool -P httpd_read_user_content 1
            *****  Plugin catchall (1.41 confidence) suggests  ***************************
            If you believe that httpd should be allowed read access on the index.html file by default.
            Then you should report this as a bug.
            You can generate a local policy module to allow this access.
            allow this access for now by executing:
            # grep httpd /var/log/audit/audit.log | audit2allow -M mypol
            # semodule -i mypol.pp


Pretty cool!

All of the information is now consolidated into a single tool chain, and if administrators start using systemct status foobar, they will get important SELinux information in the main log, making SELinux easier to use and thus getting more people to keep it enabled. 

BTW, this feature will be in Red Hat Enterprise 7.


I hope to work with the systemd/journald team to allow systemctl status to provide extended information, so the entire message could have been shown in that screen,  eliminating the need the next step, but this is a huge step forward in usability.

In the coming fedora's I hope to eliminate the setroubleshoot database totally and just use journald as its information database.

And I still want to push the kernel team forward on the Friendly EPERM.