Tired of ads? Upgrade to paid account and never see ads again!


Dan Walsh's Blog

Got SELinux?

Setroubleshoot does a nice job nowadays but do people read it?
Getting a bugzillas with an setroubleshoot alert mesage that tells the user user exactly 
what the problem and solution is, and yet the user goes through the GUI looking for the 
Report the Bug button.  

I call this the Bug Dan Walsh Button, although I guess I could call it the Bug Miroslav Grepl 
Button now. 

Then the user waits for a response from a human saying to say 

"Did you read the alert?  It told you what to do." Bugzilla Closed NotABug

All the time the user sits with a broken tool or in permissive mode, when the user could 
have fixed the problem in seconds.  Lots of bugzillas say, XYZ is mislabeled just run restorecon XYZ  :^(


Setroubleshoot did a great job of diagnosing this problem.  

It gave the user two good solutions, and one fall back.

Bottom Line, Please read the alert information before reporting a bugzilla.

Of course the tooling in NetworkManager should have done this automatically, but we have a 
bugzilla open for that.

Description of problem:
Upon trying to activate the VPN interface, I received the pop-up advising me that it was blocked. 
SELinux is preventing /usr/sbin/openvpn from 'open' accesses on the file /home/dwalsh/personalVPN/CN00318823.crt.

*****  Plugin openvpn (47.5 confidence) suggests  ****************************

If you want to mv CN00318823.crt to standard location so that openvpn can have open access
Then you must move the cert file to the ~/.cert directory
# mv /home/dwalsh/personalVPN/CN00318823.crt ~/.cert
# restorecon -R -v ~/.cert

*****  Plugin openvpn (47.5 confidence) suggests  ****************************

If you want to modify the label on CN00318823.crt so that openvpn can have open access on it
Then you must fix the labels.
# semanage fcontext -a -t home_cert_t /home/dwalsh/personalVPN/CN00318823.crt
# restorecon -R -v /home/dwalsh/personalVPN/CN00318823.crt

*****  Plugin catchall (6.38 confidence) suggests  ***************************

If you believe that openvpn should be allowed open access on the CN00318823.crt file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
allow this access for now by executing:
# grep openvpn /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

Additional Information:
Source Context                system_u:system_r:openvpn_t:s0
Target Context                unconfined_u:object_r:user_home_t:s0
Target Objects                /home/dwalsh/personalVPN/CN00318823.crt [ file ]
Source                        openvpn
Source Path                   /usr/sbin/openvpn
Port                          <Unknown>
Host                          (removed)
Source RPM Packages           openvpn-2.3.2-1.fc19.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.12.1-73.fc19.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     (removed)
Platform                      Linux (removed) 3.10.10-200.fc19.x86_64 #1 SMP Thu
                              Aug 29 19:05:45 UTC 2013 x86_64 x86_64
Alert Count                   2
First Seen                    2013-09-03 16:46:59 EDT
Last Seen                     2013-09-05 17:57:28 EDT
Local ID                      f008846c-ad32-4676-925c-4a86a1b87a2b

Raw Audit Messages
type=AVC msg=audit(1378418248.924:702): avc:  denied  { open } for  pid=1996 comm="openvpn" path="/home/dwalsh/personalVPN/CN00318823.crt" dev="dm-2" ino=11141302 scontext=system_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

type=SYSCALL msg=audit(1378418248.924:702): arch=x86_64 syscall=open success=no exit=EACCES a0=7fffcf992f0e a1=0 a2=1b6 a3=7fffcf990410 items=0 ppid=1992 pid=1996 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=openvpn exe=/usr/sbin/openvpn subj=system_u:system_r:openvpn_t:s0 key=(null)

Hash: openvpn,openvpn_t,user_home_t,file,open

New MLS videos at access.redhat.com

We no how much you like SELinux!!!

Well if you really want to turn up the enjoyment, why not try Multi Level Security (MLS) policy?

Well you probably only want to do this if you need to store data on a machine that is truly multi level. 

MLS Policy (selinux-policy-mls)

We have been shipping MLS policy since RHEL5, and Fedora 6.

RHEL5 and RHEL6 have achieved a government certification of EAL4+/LSPP.  This certification basically says that these Operating Systems are able to store and handle data that  has different Security levels, like Top Secret/Secret/Unclassified.  It is the same certification that Trusted Solaris achieved.  Bottom line is the Defense Organisations are using SELinux MLS on RHEL5 and RHEL6  in the most sensitive places.   If you are an administrator of these types of machines, you need to see some new videos on access.redhat.com.

Dave Egts is at it again.

I absolutely loved Dave's videos on confined users.  Whenever I teach SELinux Roles Based Access Control (RBAC), I show the videos.

They are Must See TV

Dave does a great job of explaining complex technology  in short easy to consume videos.

Now Dave and the folks who build the content on access.redhat.com have put together 8 MLS Videos.

Multilevel Security with Red Hat Enterprise Linux and SELinux

Check it out.

What would happen if an OpenShift gear became root?
Here is a video I put together for the Red Hat Summit.

The video shows how SELinux would control an openshift gear if it somehow it got a privilege escalation and actually became root.

New Security Feature in Fedora 20 Part 1: Setroubleshoot - Journald integration
A little early to start this series, but I am pretty excited about all the new features coming in Fedora 20.

A while ago a fellow worker Máirín Duffy was very proud of herself for figuring out an SELinux issue.  Basically she had a problem that is fairly common struggle people have with SELinux.  She had created some content for her web server in her home directory, and then mv'd it into a /var/www subdir and tried to look at it in here web browser and got a screen like the following.

Screenshot from 2013-08-02 08:02:04

When she went to look at the /var/log/httpd/error_log, she saw something like:

# tail /var/log/httpd/error_log
[Fri Aug 02 08:05:43.347080 2013] [core:error] [pid 10556] (13)Permission denied: [client ::1:38045] AH00132: file permissions deny server access: /var/www/html/index.html

Instead of thinking there was a problem with Ownership or Permission flags, she thought of SELinux and  figured their might be  problem with the labels,  so she ran:

# restorecon -R -v /var/www
restorecon reset /var/www/html/index.html context unconfined_u:object_r:user_home_t:s0->unconfined_u:object_r:httpd_sys_content_t:s0

Well if she had continued to investigate she would have seen the following in other logs.

In the /var/log/audit/audit.log the kernel had reported an AVC message like the following.

# ausearch -m avc -ts recent -i
type=PATH msg=audit(08/02/2013 08:05:43.346:1197) : item=0 name=/var/www/html/index.html inode=3145858 dev=08:03 mode=file,644 ouid=root ogid=root rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0
type=CWD msg=audit(08/02/2013 08:05:43.346:1197) :  cwd=/
type=SYSCALL msg=audit(08/02/2013 08:05:43.346:1197) : arch=x86_64 syscall=open success=no exit=-13(Permission denied) a0=0x7f476595da40 a1=O_RDONLY|O_CLOEXEC a2=0x0 a3=0x7fffe27e11b0 items=1 ppid=10552
pid=10556 auid=unset uid=apache gid=apache euid=apache suid=apache fsuid=apache egid=apache sgid=apache fsgid=apache ses=unset tty=(none) comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(08/02/2013 08:05:43.346:1197) : avc:  denied { read } for  pid=10556 comm=httpd name=index.html dev="sda3" ino=3145858 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

SETroubleshoot received this message and reported in /var/log/messages.

# grep setroubleshoot /var/log/messages
Aug  2 08:01:46 redsox setroubleshoot: SELinux is preventing /usr/sbin/httpd from read access on the file /var/www/html/index.html. For complete SELinux messages. run sealert -l fd6b9022-1ced-4065-905a-8f0e884f9915

If she finally ran this sealert command specified in the log file she would have found setroubleshoot had written its analysis in /var/lib/setroubleshoot/setroubleshoot_database.xml

# sealert -l fd6b9022-1ced-4065-905a-8f0e884f9915

SELinux is preventing /usr/sbin/httpd from read access on the file /var/www/html/index.html.

*****  Plugin restorecon (92.2 confidence) suggests  *************************

If you want to fix the label.
/var/www/html/index.html default label should be httpd_sys_content_t.
Then you can run restorecon.
# /sbin/restorecon -v /var/www/html/index.html

Nice that Máirín figured out her problem so quickly, but as you see the system had scattered little clues all over the system to help her.  And who says SELinux is not easy.

Ok Dan get to the point?

If you have been following the Fedora-devel list their has been a large discussion going on about whether or not to enable syslog (rsyslog) in the default install.  The idea is to consolidate logging into journald which I talked about in a previous blog.  And get log analysis tools to start relying on its data rather then scanning /var/log/messages.

SETroubleshot integration with journald.

Back in Fedora 18 or 19 we started dumping setroubleshoot data into the journal.  The problem with this data is that it was setroubleshoot reporting on the httpd process, but there was no way for journald/systemd to know this.  I asked the journald team to add a mechanism for a privileged application to specify a Process ID (PID) to which the log message pertained.  Well systemd/journald has added this in systemd-206-1.  I added the functionality to setroubleshoot-3.2.11-1.fc20.x86_64

Now if Máirín had simply checked the status of the httpd service, she would see a message like:

# systemctl status httpd
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
   Active: active (running) since Fri 2013-08-02 08:01:35 EDT; 30min ago
 Main PID: 10552 (httpd)
   Status: "Total requests: 4; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─10552 /usr/sbin/httpd -DFOREGROUND
           ├─10553 /usr/libexec/nss_pcache 196611 off /etc/httpd/alias
           ├─10554 /usr/sbin/httpd -DFOREGROUND
           ├─10555 /usr/sbin/httpd -DFOREGROUND
           ├─10556 /usr/sbin/httpd -DFOREGROUND
           ├─10557 /usr/sbin/httpd -DFOREGROUND
           ├─10558 /usr/sbin/httpd -DFOREGROUND
           └─10569 /usr/sbin/httpd -DFOREGROUND

Aug 02 08:01:35 redsox.boston.devel.redhat.com systemd[1]: Started The Apache HTTP Server.
Aug 02 08:01:46 redsox.boston.devel.redhat.com python[10564]: SELinux is preventing /usr/sbin/httpd from read access on the file /va...html.
                                                              *****  Plugin restorecon (

Then if she used the journalctl command she could see the full analysis.

# journalctl  -r -o verbose -u httpd.service

-- Logs begin at Tue 2013-04-09 15:19:05 EDT, end at Fri 2013-08-02 08:32:35 EDT. --
Fri 2013-08-02 08:05:45 EDT [s=3087834a63d74580811c9e1088ac7fdf;i=1195d;b=71667fa0d7074a3f8bdf1c0da22fe234;m=22b877728;t=4e2f5c7552cb6;x=908
    _CMDLINE=/usr/bin/python -Es /usr/sbin/setroubleshootd -f
    MESSAGE=SELinux is preventing /usr/sbin/httpd from read access on the file /var/www/html/index.html.
            *****  Plugin restorecon (92.2 confidence) suggests  *************************
            If you want to fix the label.
            /var/www/html/index.html default label should be httpd_sys_content_t.
            Then you can run restorecon.
            # /sbin/restorecon -v /var/www/html/index.html
            *****  Plugin catchall_boolean (7.83 confidence) suggests  *******************
            If you want to allow httpd to read user content
            Then you must tell SELinux about this by enabling the 'httpd_read_user_content' boolean.
            You can read 'user_selinux' man page for more details.
            setsebool -P httpd_read_user_content 1
            *****  Plugin catchall (1.41 confidence) suggests  ***************************
            If you believe that httpd should be allowed read access on the index.html file by default.
            Then you should report this as a bug.
            You can generate a local policy module to allow this access.
            allow this access for now by executing:
            # grep httpd /var/log/audit/audit.log | audit2allow -M mypol
            # semodule -i mypol.pp


Pretty cool!

All of the information is now consolidated into a single tool chain, and if administrators start using systemct status foobar, they will get important SELinux information in the main log, making SELinux easier to use and thus getting more people to keep it enabled. 

BTW, this feature will be in Red Hat Enterprise 7.


I hope to work with the systemd/journald team to allow systemctl status to provide extended information, so the entire message could have been shown in that screen,  eliminating the need the next step, but this is a huge step forward in usability.

In the coming fedora's I hope to eliminate the setroubleshoot database totally and just use journald as its information database.

And I still want to push the kernel team forward on the Friendly EPERM.

New Security Feature in Fedora 19 Part 5: gssproxy

On Friday, I was asked to quickly write policy for the new gssproxy daemon which was added to Fedora 19.  Investigating it, I found I had a new security feature blog, I had to write.

Fedora 19 is adding the gssproxy daemon.  gssproxy will be replacing rpc.svcgssd.

gssproxy is a a proxy daemon for the GSS-API, which is a higher level API used mainly in Kerberized applications.

File systems services like CIFS, (samba), NFS and AFS make lots of kerberos encryption calls within the kernel, using keying material handed into the kernel.  But the initialization and creation, renewing and cleaning up of credential caches, really should be handled in user space.  gssproxy is the daemon to handle this.

A useful feature of gssproxy will eliminate the need to daemons that use kerberos keytabs from needing to read them directly.  Currently if you set up an Apache server with kerberos, it needs full read access to the keytab file, allowing it to see the keying material.  If Apache gets hacked, the hacker would get full access to the secrets.  With gssproxy developers could setup Apache to talk to gssproxy and allow it to handle the keytab file, eliminating this need.  This will allow my team to alter the SELinux policy and prevent the Apache daemon from being able to read keying information directly and force it to talk to the daemon.

Another cool feature, explained to me by Simo Sorce,  is to take an application that needs kerberos access to a nfs share, but does not know anything about kerberos or the gssapi.  Currently the way you handle this is to wrap an init scripts with things like k5start so the application has a keytab and a credential cache regularly refreshed.  Then rpc.gssd would have to look in /tmp for a ccache to use for the application user and allow the application authenticated access to the secure nfs share.

With gssproxy you will not longer need to do this.  All you need to do is drop a keytab in /var/lib/gssproxy/clients/<id>.keytab and the gssproxy will automatically initiate a connection when nfs client needs it. Advantages of this method:
1. No more wrapping of init scripts
2. Privilege separation, the app has no access to the keytab
3. Initiate on request, if the app never needs nfs we never initiate or renew tickets unnecessarily

gssproxy can be setup to renew Kerberos Tickets from keytabs for long running jobs, eliminating a lot of previous hacks.

New Security Feature in Fedora 19 Part 4: openssh 6.2 better support for multi-factor authentication
We are beginning to see the end of passwords as the only means of authenticating yourself to a system, hopefully. 

Fedora 19 will be our first release of Openssh 6.2, which has introduced the AuthenticationMethods setting.
This feature allows you to require multiple different types of authorization to get into a system.  For example it is very easy to require  both an ssh public key and a password to login.   If you don't have the public key, you will never get to the password prompt.

In previous Fedora releases, there were some tricky  ways to do multi-factor using pam but this allows for more  combinations, and easier setup.
I found this blog that does a great job of describing the feature.

Bottom line if you have a critical server, you want a user to prove multiple ways that he is worthy to get on the system.

How do I tell what would be allowed by a boolean?

I received and Email today that asked the following question:

I still fail to understand the difference between  httpd_can_network_connect_db and httpd_can_network_connect. Some people say the former allows connections to known database ports. My question are:

What are these ports? Where are the corresponding policy defined? I found many  .pp files deeply under /etc/selinux, and I feel sorry that they are binary which are almost impossible to interpret, so where can I find the the source files for the compiled policy, and what is the language to define policies?

You could use the semanage command for how the booleans are described.

> semanage boolean -l | grep httpd_can_network_connect
httpd_can_network_connect_db   (off  ,  off)  Allow HTTPD scripts and modules to connect to databases over the network.
httpd_can_network_connect      (off  ,  off)  Allow HTTPD scripts and modules to connect to the network using TCP.

The best answer to this is to look at the sesearch and seinfo tools and on newer (Fedora/RHEL7) systems sepolicy command.  Also look at the man pages that have been generated.

man httpd_selinux

sesearch and seinfo are available in the setools-cmdline package.  sepolicy is in policycoreutils-python package.


sesearch -A -s httpd_t -b httpd_can_network_connect_db -p name_connect
   allow httpd_t postgresql_port_t : tcp_socket { recv_msg send_msg name_connect } ;
   allow httpd_t mssql_port_t : tcp_socket name_connect ;
   allow httpd_t oracle_port_t : tcp_socket name_connect ;
   allow httpd_t mysqld_port_t : tcp_socket { recv_msg send_msg name_connect } ;
   allow httpd_t gds_db_port_t : tcp_socket name_connect ;

The command above reads in the policy and prints out what happens when you enable the httpd_can_network_connect_db boolean.  We further restrict the search to see how it affects the httpd_t, apache, process type with the name_connect access.     sesearch tells us that turning on httpd_can_network_connect_db would allow the httpd_t domain to connect to tcp ports labeled postgresql_port_t, mssql_port_t, oracle_port_t, mysqld_port_t, gds_db_port_t.  You can use seinfo to turn these port types into port definitions. semanage port -l would also work.

> seinfo  --port | grep -e postgresql_port_t -e mysqld_port_t -e oracle_port_t -e gds_db_port_t | grep tcp
    portcon tcp 3050 system_u:object_r:gds_db_port_t:s0
    portcon tcp 1186 system_u:object_r:mysqld_port_t:s0
    portcon tcp 3306 system_u:object_r:mysqld_port_t:s0
    portcon tcp 63132-63164 system_u:object_r:mysqld_port_t:s0
    portcon tcp 1521 system_u:object_r:oracle_port_t:s0
    portcon tcp 2483 system_u:object_r:oracle_port_t:s0
    portcon tcp 2484 system_u:object_r:oracle_port_t:s0
    portcon tcp 5432 system_u:object_r:postgresql_port_t:s0

> sepolicy network -t postgresql_port_t
postgresql_port_t: tcp: 5432


> sesearch -A -s httpd_t -b httpd_can_network_connect -p name_connect
Found 1 semantic av rules:
   allow httpd_t port_type : tcp_socket name_connect ;

The above command shows that httpd_can_network_connect allows httpd_t to connect to all tcp socket types that have the port_type attribute.

> seinfo -aport_type -x | wc -l

Using seinfo above would show you that port_type is the attribute of all port types, meaning that turning on the httpd_can_network_connect boolean, allows the httpd_t domain to connect to ALL tcp network ports.

Bottom Line httpd_can_network_connect_db allows httpd_t to connect to an additional 10 ports while httpd_can_network_connect adds thousands.

sandbox -X in Fedora 19 is resizeable, finally!
Thomas Liu worked as an intern for me a few years ago and wrote a patch for Xephyr to allow a Xepher window to be resizeable.

We shipped this patch in RHEL6 and waited for it to get upstream so we could pull it in Fedora.  Well as of today a Xephr
 xorg-x11-server-1.14.1-2.fc19 has been pushed into Fedora 19, and we have support for resizing sandbox in Fedora.

Now rumors have it Labelled NFS may finally reach upstream, that would truly be remarkable...

New Security Feature in Fedora 19 Part 3: Hard Link/Soft Link Protection
It is surprising to most people who understand Linux and Unix that you are allowed to Hard Link to any file on the OS as long as it is on the same file system.
Hard linking means that you can put the same Inode number into two different directories.  Prior to Fedora 19, a normal user would be allowed to do something like:

> ln /etc/shadow ~/shadow

Even though the /etc/shadow is owned by root.  Then if the user tricked an administrator into doing something like

# chown -R  dwalsh:dwalsh ~

This would cause the /etc/shadow file to be owned by dwalsh.

Similarly in the past hackers have tricked privileged applications to follow hard and soft links, created in a user account to compromise the system.

Kees Cook wrote some patches for the linux kernel that allows distributions to disable this behaviour.

Here is Kees Description of the benefits of the patch:

This patch adds symlink and hardlink restrictions to the Linux VFS.


A long-standing class of security issues is the symlink-based time-of-check-time-of-use race, most commonly seen in world-writable
directories like /tmp. The common method of exploitation of this flaw is to cross privilege boundaries when following a given symlink (i.e. a
root process follows a symlink belonging to another user). For a likely incomplete list of hundreds of examples across the years, please see:

The solution is to permit symlinks to only be followed when outside a sticky world-writable directory, or when the uid of the symlink and
follower match, or when the directory owner matches the symlink's owner.



On systems that have user-writable directories on the same partition as system files, a long-standing class of security issues is the
hardlink-based time-of-check-time-of-use race, most commonly seen in world-writable directories like /tmp. The common method of exploitation
of this flaw is to cross privilege boundaries when following a given hardlink (i.e. a root process follows a hardlink created by another
user). Additionally, an issue exists where users can "pin" a potentially vulnerable setuid/setgid file so that an administrator will not actually
upgrade a system fully.

The solution is to permit hardlinks to only be created when the user is already the existing file's owner, or if they already have read/write
access to the existing file.

Many Linux users are surprised when they learn they can link to files they have no access to, so this change appears to follow the doctrine
of "least surprise". Additionally, this change does not violate POSIX, which states "the implementation may require that the calling process
has permission to access the existing file"[1].

This change is known to break some implementations of the "at" daemon, though the version used by Fedora and Ubuntu has been fixed[2] for
a while. Otherwise, the change has been undisruptive while in use in Ubuntu for the last 1.5 years.


This feature is turned on by default in /usr/lib/sysctl.d/50-default.conf in F19.

grep fs /usr/lib/sysctl.d/50-default.conf 
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

There was some concern that this could cause problems in applications that expect this behaviour, so it can be disabled.
But overall, I think it is a positive feature for Fedora.

SELinux is a labeling system. First thought should be "Is there a label that would make this work?"
On the SELinux mail list today, someone asked:

I want to store the logs from openswan into a different file ( /var/log/ipsec ) than the default. For this purpose I added


to ipsec.conf.
    As long as I keep the server in permissive mode, openswan starts OK. If, however, I switch to enforcing, the daemon refuses to start with the following error message displayed in the console:

ipsec_setup: Starting Openswan IPsec U2.6.32/K3.0.78-1.el6.elrepo.x86_64...
ipsec_setup: Cannot write to "/var/log/ipsec".

   The audit log does not record anything useful so I tried to switch dontaudit to off and see if anything useful comes out. After running audit2allow and a bit of trial and error I came out with the following custom policy :

module myipsec 1.0;
require {
        type ipsec_t;
        type var_log_t;
        class file { write ioctl getattr append };
#============= ipsec_mgmt_t ==============
allow ipsec_mgmt_t var_log_t:file write;

   The above policy worked for me but I am wondering if it is OK

The problem is the administrator decided to add policy that allows ipsec_mgmt_t to write any file labeled var_log_t.  A hacked ipsec_mgmt could now overwrite any log file on the system labeled var_log_t, including /var/log/messages.  var_log_t is the default label for ANY file in /var/log directory that does not  have SELinux policy controlling it.  Also remember "write" access is always more dangerous then "append" access, since "write" allows you to truncate a file, destroying evidence, versus append to the end of a file.

In the paper I wrote a few years ago,

What is SELinux trying to tell me?
The 4 key causes of SELinux errors.

I explain that adding policy should be your third option, not your first.  In this case Dominic Grift pointed out the admin, that changing the label of the target would fix the problem and not involve adding custom policy.

semanage fcontext -a -t ipsec_log_t "/var/log/ipsec.*"
restorecon -v /var/log/ipsec

By telling SELinux that the content in the /var/log/ipsec log file was ipsec_log_t, you solve your problem and end up with the same security you had before the change.

Think Labels First...

You are viewing danwalsh