Containers and MLS

I have just updated the container-selinux policy to support MLS (Multi Level Security).  

SELinux and Container technology have a long history together.  Some people imagine that containers started just a few years ago with the introduction of Docker, but the technology goes back a lot further then that. 

SELinux originally supported two type of Mandatory Access Control,  Type Enforcement and RBAC (Roles Based Access Control).  

Type Enforcement

Type Enforcement is SELinux main security measure.  Basically every process on the system gets a assigned a type (httpd_t, unconfined_t, container_t ...)  And every object (file, directory, socket, tcp port ...) gets assigned a type. (httpd_sys_content_t, user_home_t, httpd_port_t, container_file_t).  Then we write rules that define the access between process types and object types. Note: The type field is always the third field of the SELinus label.

allow container_t container_file_t:file { open read write }

Anything that is not allowed is denied.

RBAC (Roles Based Access Control)

Read more...Collapse )

Attributes make writing SELinux policy easier

Yesterday I received an email from someone who was attempting to write SELinux policy for a daemon process, "abcd", that he was being required to run on his systems.



 Here's the problem:  the ****** agent runs as root and has the ability to
make changes to system configuration files.  So deploying this agent
means handing over root-level control of all of my RHEL servers to
people I don't know and who have no accountability to my system owner or
authorizing official.  ..... really, really
uncomfortable about this and want to protect our system from
unauthorized changes inflicted by some panicking yabbo ....

This seems like a job for SELinux.  Our RHEL7 baseline has SELinux
enforcing (
AWESOME) already, so I figured we just need to write a custom policy
that defines a type for the ******* agent process (abcd_t) and confines
that so that it can only access the agent's own data directories.

We did that, and it's not working out.  Once we start the confined
agent, it spews such a massive volume of AVC denials that it threatens
to fill the log volume and the agent never actually starts.  There are
way too many errors for the usual 'read log/allow exception' workflow to
be viable.  My guess is the agent is trying to read a bunch of sensitive
system files and choking because SELinux won't let it.

Have you got any advice or ideas here?


Here is my response:

Read more...Collapse )

SELinux blocks loading kernel modules

The kernel has a feature where it will load certain kernel modules for a process, when certain syscalls are made.  For example, loading a kernel module when a process attempts to create a different network socket.  

I wrote a blog on explaining how this is probably a bad idea from a containers perspective.  I don't want to allow container processes to trigger modifications of the kernel.  And potentially causing the kernel to load risky modules that could have vulnerabilities in them.  I say, let the Administrator or packagers decide what kernel modules need to be loaded and then make the containers live with what is provided for them.  Here is a link to the blog.

Why all the DAC_READ_SEARCH AVC messages?

If you followed SELinux policy bugs being reported in bugzilla you might have noticed a spike in messages about random domains being denied DAC_READ_SEARCH.

Let's quickly look at what the DAC_READ_SEARCH capability is.  In Linux the power of "root" was broken down into 64 distinct capabilities.  Things like being able to load kernel modules or bind to ports less then 1024.  Well DAC_READ_SEARCH is one of these.

DAC stands for Discretionary Access Control, which is what most people understand as standard Linux permissions, Every process has owner/group.  All file system objects are  assigned owner, group and permission flags.  DAC_READ_SEARCH allows a privilege process to ignore parts of DAC for read and search.

man capabilities



              * Bypass file read permission checks and directory read and execute permission checks;

There is another CAPABILITY called DAC_OVERRIDE


              Bypass file read, write, and execute permission checks.

As you can see DAC_OVERRIDE is more powerful then DAC_READ_SEARCH, in that it can write and execute content ignoring DAC rules, as opposed to just reading the content.

Well why did we suddenly see a spike in confined domains needing DAC_READ_SEARCH?

Read more...Collapse )

New version of buildah 0.2 released to Fedora.
New features and bugfixes in this release

Updated Commands
buildah run
     Add support for -- ending options parsing
     Add a way to disable PTY allocation
     Handle run without an explicit command correctly
Buildah build-using-dockerfile (bud)
    Ensure volume points get created, and with perms
buildah containers
     Add a -a/--all option - Lists containers not created by buildah.
buildah Add/Copy
     Support for glob syntax
buildah commit
     Add flag to remove containers on commit
buildah push
     Improve man page and help information
buildah export:
    Allows you to export a container image
buildah images:
    update commands
    Add JSON output option
buildah rmi
    update commands
buildah containers
     Add JSON output option

New Commands
buildah version
     Identify version information about the buildah command
buildah export
     Allows you to export a containers image

Buildah docs: clarify --runtime-flag of run command
Update to match newer storage and image-spec APIs
Update containers/storage and containers/image versions

Quick Blog on Buildah.
Buildah is a new tool that we released last week for building containers without requiring a container runtime daemon running. --nodockerneeded

Here is a blog that talks about some of its features.

Our main goal was to make this simple.  I was asked by a fellow engineer about a feature that docker has for copying a file out of a container onto the host.  "docker cp".  In docker this ends up being a client server operation, and required someone to code it up.  We don't have this feature in buildah.  :^(

BUT, buildah gives you the primitives you need to do simpler functionality and allows you to use the full power of bash.  If I want to copy a file out of a container, I can simply mount the container and copy it out.

# mnt=$(buildah mount CONTAINER_ID)
# buildah umount CONTAINER_ID

The beauty of this is we could use lots of tools, I could scp if I wanted to copy to another machine, or rsync, or ftp...

Once your have the container mounted up, you can use any bash command on it, to move files in or out.

buildah == simplicity

What capabilities do I really need in my container?

I have written previous blogs discussing using linux capabilities in containers.

Recently I gave a talk in New York and someone in the audience asked me about how do they figure out what capabilities their containers require?

This person was dealing with a company that was shipping their software as a container image, but they had instructed the buyer, that you would have to run their container ‘fully privileged”.  He wanted to know what privileges the container actually needed.  I told him about a project we worked on a few years ago, we called Friendly Eperm.

Permission Denied!  WHY?

A few years ago the SELinux team realized that more and more applications were getting EPERM returns when a syscall requested some access.  Most operators understood EPERM (Permission Denied) inside of a log file to mean something was wrong with the Ownership of a process of the contents it was trying to access or the permission flags on the object were wrong.  This type of Access Control is called DAC (Discretionary Access Control) and under certain conditions SELinux also caused the kernel to return EPERM.  This caused Operators to get confused and is one of the reasons that Operators did not like SELinux. They would ask, why didn’t httpd report that Permission denied because of SELinux?  We realized that there was a growing list of other tools besides regular DAC and SELinux which could cause EPERM.  Things like SECCOMP, Dropped Capabilities, other LSM …   The problem was that the processes getting the EPERM had no way to know why they got EPERM.  The only one that knew was the kernel and in a lot of cases the kernel was not even logging the fact that it denied access.  At least SELinux denials usually show up in the audit log (AVCs).   The goal of Friendly EPERM was to allow the processes to figure out why they got EPERM and make it easier for admin to diagnose.

Here is the request that talks about the proposal.

The basic idea was to have something in the /proc file system which would identify why the previous EPERM happened.  You are running a process, say httpd, and it gets permission denied. Now somehow the process can get information on why it got permission denied.  One suggestion was that we enhanced the libc/kernel to provide this information. The logical place for the kernel to reveal it would be in /proc/self.  But the act of httpd attempting to read the information out of /proc/self itself could give you a permission denied.  Basically we did not succeed because it would be a race condition, and the information could be wrong.

Here is a link to the discussion!msg/fa.linux.kernel/WQyHPUdvodE/ZGTnxBQw4ioJ

Bottom line, no one has figured a way to get this information out of the kernel.


Later I received an email discussing the Friendly EPERM product and asking if there was a way to at least figure out what capabilities the application needed.

I wondered if the audit subsystem would give us anything here.  But I contacted the Audit guys at Red Hat, Steve Grubb and Paul Moore,  and they informed me that there is no Audit messages generated when DAC Capabilities are blocked.

An interesting discussion occurred in the email chain:

DWALSH: Well I would argue most developers have no idea what capabilities their application requires.

SGRUBB: I don't think people are that naive. If you are writing a program that runs as root and then you get the idea to run as a normal user, you will immediately see your program crash. You would immediately look at where it’s having problems. Its pretty normal to lookup the errno on the syscall man page to see what it says about it. They almost always list necessary capabilities for that syscall. If you are an admin restricting software you didn't write, then it’s kind of a  puzzle. But the reason there's no infrastructure is because historically it’s never been a problem because the software developer had to choose to use capabilities and it’s incumbent on the developer to know what they are doing.  With new management tools offering to do this for you, I guess it’s new territory.

But here we had a vendor telling a customer that it needed full root, ALL Capabilities,  to run his application,

DWALSH:  This is exactly what containers are doing.  Which is why the emailer is asking.  A vendor comes to him telling him it needs all Capabilities.  The emailer does not believe them and wants to diagnose what they actually need.

DWALSH: With containers and SELinux their is a great big "TURN OFF SECURITY" button, which is too easy for software packagers to do, and then they don't have to figure out exactly what their app needs.

Paul Moore - Red Hat SELinux Kernel Engineer suggested

That while audit can not record the DAC Failures, SELinux also enforces the capability checks.  If we could put the processes into a SELinux type that had no capabilities by default, then ran the process with full capabilities and SELinux in permissive mode, we could gather the SELinux AVC messages indicating which capabilities the application required to run.

“ (Ab)using security to learn through denial messages. What could possibly go wrong?! :)

After investigating further, turns out the basic type used to run containers, `container_t`, can be setup to have no capabilities by turning off an SELinux boolean.

To turn off the capabilities via a boolean, and put the machine into permissive mode.

setsebool virt_sandbox_use_all_caps=0

setenforce 0

Now execute the application via docker with all capabilities allowed.

docker run --cap-add all IMAGE ...

Run and test the application. This should cause SELinux to generate AVC messages about capabilities used.

grep capability /var/log/audit/audit.log

type=AVC msg=audit(1495655327.756:44343): avc:  denied  { syslog } for  pid=5246 comm="rsyslogd" capability=34  scontext=system_u:system_r:container_t:s0:c795,c887 tcontext=system_u:system_r:container_t:s0:c795,c887 tclass=capability2   


Now you know your list.

Turns out the application the emailer was trying to containerize was a tool which was allowed to manipulate the syslog system, and the only capability it needed was CAP_SYSLOG.  The emailer should be able to run the container by simply adding the CAP_SYSLOG capability and everything else about the container should be locked down.

docker run --cap-add syslog IMAGE ...


After writing this blog, I was pointed to

Find what capabilities an application requires to successful run in a container

Which is similar in that it finds out the capabilities needed for a container/process by using SystemTap.

SELinux and --no-new-privs and the setpriv command.

SELinux transitions are in some ways similar to a setuid executable in that when a transition happens the new process has different security properties then the calling process.  When you execute setuid executable, your parent process has one UID, but the child process has a different UID.

The kernel has a way for block these transitions, called --no-new-privs.  If I turn on --no-new-privs, then setuid applications like sudo, su and others will no longer work.   Ie you can not get more privileges then the parent process.

SELinux does something similar and in most cases the transition is just blocked.

For example.

We have a rule that states a httpd_t process executing a script labeled httpd_sys_script_exec_t (cgi label) will transition to httpd_sys_script_t.  But if you tell the kernel that your process tree has --no-new-privs, depending on how you wrote the policy , when the process running as httpd_t executes the httpd_sys_script_exec_t it will no longer transition, it will attempt to continue to run the script as httpd_d.

SELinux enforces that the new transition type must have a subset of the allow rules of its parent process.  But it can have no other allow rules.  IE the transitioned process can not get any NEW Privs.

This feature has not been used much in SELinux Policy, so we have found and fixed a few issues.

Container Runtimes like docker and runc have now added the --no-new-privs flag.  We have been working to make container-selinux follow the rules.

The container runtimes running container_runtime_t can start a container_t process only if container_runtime_t has all of the privileges of its parent process.

In SELinux policy you write a rule like:

typebounds container_runtime_t container_t;

This tells the compiler of policy to make sure that the container_t is a subset. If you are running the compiler in strict mode, the compiler will fail, if it is not a subsection.
If you are not running in strict mode, the compiler will silently remove any allow rules that are not in the parent, which can cause some surprises.

setpriv command

I recently heard about the setpriv command which is pretty cool for playing with kernel security features like dropping capabilities, and SELinux. One of the things you can do is execute

$ setpriv --no-new-privs sudo sh
sudo: effective uid is not 0, is sudo installed setuid root?

But if you want to try out SELinux you could combine the setpriv command and runcon together

$ setpriv --no-new-privs runcon -t container_t id -Z
$ setpriv --no-new-priv runcon -t container_t id -Z
runcon: id: Operation not permitted

This happens because container_t is not type bounds by staff_t.

Be careful relabeling volumes with Container run times. Sometimes things can go very wrong?
I recently revieved an email from someone who made the mistake of volume mounting /root into his container with the :Z option.

docker run -ti -v /root:/root:Z fedora sh

The container ran fine, and everything was well on his server machine until the next time he tried to ssh into the server.

The sshd refused to allow him in?  What went wrong?

I wrote about using volumes and SELinux on the project atomic blog.  I explain their that in order to use a volume within a non privileged container, you need to relabel the content on the volume.  You can either use the :z or the :Z option.

:z will relabel with a shared label so other containers ran read and write the volume.
:Z will relabel with a private label so that only this specific container can read and write the volume.

I probably did not emphasize enough is that as Peter Parker (Spider Man) says: With great power comes great responsibility.

Meaning you have to be careful what you relabel.  Using one of the :Z and :z options to recursively change the labels of the source content to container_file_t. When doing this you must be sure that this content is truly private to the container.  The content is not needed by other confined domains.  For example doing a volume mount of -v /var/lib/mariadb:/var/lib/mariadb:Z for a mariadb container is probably a good idea. But while doing -v /var/lib:/var/lib:Z will work, it is probably a bad idea.

Back to the email, the user relabeled all of the content under /root with a label similar to
Later when he attempted to ssh in, the sshd daemon,  running as the sshd_t type, attempts to read content in /root/.ssh it gets permission denied, since sshd_t is not allowed to read container_file_t.

The emailer realized what happened and tried to fix this situation by running restorecon -R -v /root, but this failed to change them?

Why did the labels not change when he ran restorecon?

There is a little known feature of restorecon called customizable_types, that I talked about 10 years ago.

By default, restorecon does not change types defined in the customizable_types file.  These types can be randomly scattered around the file system, and we don't want a global relabel to change them.  This is meant to make it easier to the admin, but sometimes causes confusion.  The
-F option tells restorecon to force the relabel and ignore customizable_types.

restorecon -F -R /root

This comand will reset the labels under /root to the system default and allow the emailer to login to the system via sshd again.

We have safeguards built into the SELinux go bindings which prevent container runtimes relabeling of
/, /etc, and /usr.

I need to open a pull request to add a few more directories to help prevent users from making serious mistakes in labeling, starting with

Understanding SELinux Roles
I received a container bugzilla today for someone who was attempting to assign a container process to the object_r role.  Hopefully this blog will help explain how roles work with SELinux.

When we describe SELinux we often concentrate on Type Enforcement, which is the most important and most used feature of SELinux.  This is what describe in the SELinux Coloring book as Dogs and Cats. We also describe MLS/MCS Separation in the coloring book.

Lets look at the SELinux labels
The SELinux labels consist of four parts, User, Role, Type and Level.  Often look something like


One area I do not cover is Roles and SELinux Users.

The analogy I like to use for the Users and Roles is around Russian dolls.  In that the User controls the reachable roles and the roles control the reachable types.

When we create an SELinux User, we have to specify which roles are reachable within the user.  (We also specify which levels are are available to the user.

semanage user -l

         Labeling   MLS/       MLS/                       
SELinux User    Prefix     MCS Level  MCS Range       SELinux Roles

guest_u             user       s0         s0                             guest_r
root                    user       s0         s0-s0:c0.c1023        staff_r sysadm_r system_r unconfined_r

staff_u               user       s0         s0-s0:c0.c1023        staff_r sysadm_r system_r unconfined_r
sysadm_u         user       s0         s0-s0:c0.c1023        sysadm_r
system_u          user       s0         s0-s0:c0.c1023        system_r unconfined_r
unconfined_u    user       s0         s0-s0:c0.c1023        system_r unconfined_r
user_u               user       s0         s0                             user_r
xguest_u           user       s0         s0                             xguest_r

In the example above you see the Staff_u user is able to reach the staff_r sysadm_r system_r unconfined_r roles, and is able to have any level in the MCS Range s0-s0:c0.c1023.

Notice also the system_u user, which can reach system_r unconfined_r roles as well as the complete MCS Range.  System_u is the default user for all processes started at boot or started by systemd.  SELinux users are inherited by children processes by default.

You can not assign an SELinux user a role that is not listed,  The kernel will reject it with a permission denied.

# runcon staff_u:user_r:user_t:s0 sh
runcon: invalid context: ‘staff_u:user_r:user_t:s0’: Invalid argument

The next part of the Russian dolls is roles.

Here is the roles available on my Fedora Rawhide machine.

# seinfo -r

Roles: 14


Roles control which types are available to a role.

# seinfo -rwebadm_r -x
      Dominated Roles:

In the example above the only valid type available to the webadm_r is the webadm_t.   So if you attempted to transition from webadm_t to a different type the SELinux kernel would block the access.

system_r role is the default role for all processes started at boot, most of the other roles are "user" roles.

seinfo -rsystem_r -x | wc -l

As you can see there are over 950 types that can be used with the system_r.

Other roles of note:

object_r is not really a role, but more of a place holder.  Roles only make sense for processes, not for files
on the file system.  But the SELinux label requires a role for all labels.  object_r is the role that we use to fill the objects on disks role.  Changing a process to run as object_r or trying to assign a different role to a file will always be denied by the kernel.

RBAC - Roles Based Access Control

SELinux Users and Roles are mainly used to implement RBAC.  The idea is to control what an adminsitrator can do on a system when he is logged in.  For certain activities he has to switch his roles.  For example, he might log in to a system as a normal user role, but needs to switch to an admin role when he needs to administrate the system.

Most of the other roles that are assigned above are user roles.  I have written about switching roles in the past when dealing with confined administrators and users.  I usually run as the staff_r but switch to the unconfined_r (through sudo) when I become the administrator of my system.

Most people who use SELinux do not deal with RBAC all that often, but hopefully this blog helps remove some of the confusion on the use of Roles.


Log in

No account? Create an account