Log in

No account? Create an account

SELinux and Containers

Next week at the Red Hat summit, I have a short session to talk about SELinux and Containers.  I am constantly reminded in bugzilla about how great the combination is.  

It truly is like Peanut Butter and Jelly.  

Sadly, people are still surprised when it blocks access.  For example I got a bugzilla recently that talked about containers not working on Fedora.  The avc was

type=AVC msg=audit(1524873307.948:1814): avc:  denied  { connectto } for  pid=28746 comm="boinc" path=002F746D702F2E5831312D756E69782F5831 scontext=system_u:system_r:container_t:s0:c420,c759 tcontext=unconfined_u:unconfined_r:xserver_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0

This AVC shows SELinux blocking the container process from connecting to the Xserver. We definitely do not want to allow containers to connect to the Xserver.  SELinux is doing precisely what it is designed to do.

Allowing a process to connect to the XServer would allow it to screen scrape all of you data on the desktop, it would also allow it to fool humans into typing passwords.  It would also allow it to grab all data in the cut and paste buffer. Especially things like passwords.

I can imagine that this works fine on other platforms with SELinux disabled. 

Read more...Collapse )

SELinux should and does BLOCK access to Docker socket

I get lots of bugs from people complaining about SELinux blocking access to the Docker socket.  For example https://bugzilla.redhat.com/show_bug.cgi?id=1557893

The aggravating thing is, this is exactly what we want SELinux to prevent.  If a container process got to the point of talking to the /var/run/docker.sock, you know this is a serious security issue.  Giving a container access to the Docker socket, means you are giving it full root on  your system.  

A couple of years ago, I wrote why giving docker.sock to non privileged users is a bad idea.

Now I am getting bug reports about allowing containers access to this socket.

Access to the docker.sock is the equivalent of sudo with NOPASSWD, without any logging. You are giving the process that talks to the socket, the ability to launch a process on the system as full root.

Usually people are doing this because they want the container to do benign operations, like list which containers are on the system, or look a the container logs.  But Docker does not have a nice RBAC system, you basically get full access or no access.  I choose to default to NO ACCESS.

If you need to give container full access to the system then run it as a --privileged container or disable SELinux separation for the container.

podman run --privileged ...
docker run --privileged ...

podman run --security-opt label:disable ...
docker run --security-opt label:disable ...

Run it privileged

Read more...Collapse )

Teaching an old dog new tricks

I have been working on SELinux for over 15 years.  I switched my primary job to working on containers several years ago, but one of the first things I did with containers was to add SELinux support.  Now all of the container projects I work on including CRI-O, Podman, Buildah as well as Docker, Moby, Rocket, runc, systemd-nspawn, lxc ... all have SELinux support.  I also maintain the container-selinux policy package which all of these container runtimes rely on.

Any ways container runtimes started adding the no-new-privileges capabilities a couple of years ago. 


The no_new_privs kernel feature works as follows:

  • Processes set no_new_privs bit in kernel that persists across fork, clone, & exec.
  • no_new_privs bit ensures process/children processes do not gain any additional privileges.
  • Process aren't allowed to unset no_new_privs bit once set.
  • no_new_privs processes are not allowed to change uid/gid or gain any other capabilities, even if the process executes setuid binaries or executables with file capability bits set.
  • no_new_privs prevents Linux Security Modules (LSMs) like SELinux from transitioning to process labels that have access not allowed to the current process. This means an SELinux process is only allowed to transition to a process type with less privileges.

Oops that last flag is a problem for containers and SELinux.  If I am running a command like

# podman run -ti --security-opt no-new-privileges fedora sh

Read more...Collapse )

Containers and MLS

I have just updated the container-selinux policy to support MLS (Multi Level Security).  

SELinux and Container technology have a long history together.  Some people imagine that containers started just a few years ago with the introduction of Docker, but the technology goes back a lot further then that. 

SELinux originally supported two type of Mandatory Access Control,  Type Enforcement and RBAC (Roles Based Access Control).  

Type Enforcement

Type Enforcement is SELinux main security measure.  Basically every process on the system gets a assigned a type (httpd_t, unconfined_t, container_t ...)  And every object (file, directory, socket, tcp port ...) gets assigned a type. (httpd_sys_content_t, user_home_t, httpd_port_t, container_file_t).  Then we write rules that define the access between process types and object types. Note: The type field is always the third field of the SELinus label.

allow container_t container_file_t:file { open read write }

Anything that is not allowed is denied.

RBAC (Roles Based Access Control)

Read more...Collapse )

Attributes make writing SELinux policy easier

Yesterday I received an email from someone who was attempting to write SELinux policy for a daemon process, "abcd", that he was being required to run on his systems.



 Here's the problem:  the ****** agent runs as root and has the ability to
make changes to system configuration files.  So deploying this agent
means handing over root-level control of all of my RHEL servers to
people I don't know and who have no accountability to my system owner or
authorizing official.  ..... really, really
uncomfortable about this and want to protect our system from
unauthorized changes inflicted by some panicking yabbo ....

This seems like a job for SELinux.  Our RHEL7 baseline has SELinux
enforcing (
AWESOME) already, so I figured we just need to write a custom policy
that defines a type for the ******* agent process (abcd_t) and confines
that so that it can only access the agent's own data directories.

We did that, and it's not working out.  Once we start the confined
agent, it spews such a massive volume of AVC denials that it threatens
to fill the log volume and the agent never actually starts.  There are
way too many errors for the usual 'read log/allow exception' workflow to
be viable.  My guess is the agent is trying to read a bunch of sensitive
system files and choking because SELinux won't let it.

Have you got any advice or ideas here?


Here is my response:

Read more...Collapse )

SELinux blocks loading kernel modules

The kernel has a feature where it will load certain kernel modules for a process, when certain syscalls are made.  For example, loading a kernel module when a process attempts to create a different network socket.  

I wrote a blog on https://medium.com/cri-o explaining how this is probably a bad idea from a containers perspective.  I don't want to allow container processes to trigger modifications of the kernel.  And potentially causing the kernel to load risky modules that could have vulnerabilities in them.  I say, let the Administrator or packagers decide what kernel modules need to be loaded and then make the containers live with what is provided for them.  Here is a link to the blog.


Why all the DAC_READ_SEARCH AVC messages?

If you followed SELinux policy bugs being reported in bugzilla you might have noticed a spike in messages about random domains being denied DAC_READ_SEARCH.

Let's quickly look at what the DAC_READ_SEARCH capability is.  In Linux the power of "root" was broken down into 64 distinct capabilities.  Things like being able to load kernel modules or bind to ports less then 1024.  Well DAC_READ_SEARCH is one of these.

DAC stands for Discretionary Access Control, which is what most people understand as standard Linux permissions, Every process has owner/group.  All file system objects are  assigned owner, group and permission flags.  DAC_READ_SEARCH allows a privilege process to ignore parts of DAC for read and search.

man capabilities



              * Bypass file read permission checks and directory read and execute permission checks;

There is another CAPABILITY called DAC_OVERRIDE


              Bypass file read, write, and execute permission checks.

As you can see DAC_OVERRIDE is more powerful then DAC_READ_SEARCH, in that it can write and execute content ignoring DAC rules, as opposed to just reading the content.

Well why did we suddenly see a spike in confined domains needing DAC_READ_SEARCH?

Read more...Collapse )

New version of buildah 0.2 released to Fedora.
New features and bugfixes in this release

Updated Commands
buildah run
     Add support for -- ending options parsing
     Add a way to disable PTY allocation
     Handle run without an explicit command correctly
Buildah build-using-dockerfile (bud)
    Ensure volume points get created, and with perms
buildah containers
     Add a -a/--all option - Lists containers not created by buildah.
buildah Add/Copy
     Support for glob syntax
buildah commit
     Add flag to remove containers on commit
buildah push
     Improve man page and help information
buildah export:
    Allows you to export a container image
buildah images:
    update commands
    Add JSON output option
buildah rmi
    update commands
buildah containers
     Add JSON output option

New Commands
buildah version
     Identify version information about the buildah command
buildah export
     Allows you to export a containers image

Buildah docs: clarify --runtime-flag of run command
Update to match newer storage and image-spec APIs
Update containers/storage and containers/image versions

Quick Blog on Buildah.
Buildah is a new tool that we released last week for building containers without requiring a container runtime daemon running. --nodockerneeded

Here is a blog that talks about some of its features.


Our main goal was to make this simple.  I was asked by a fellow engineer about a feature that docker has for copying a file out of a container onto the host.  "docker cp".  In docker this ends up being a client server operation, and required someone to code it up.  We don't have this feature in buildah.  :^(

BUT, buildah gives you the primitives you need to do simpler functionality and allows you to use the full power of bash.  If I want to copy a file out of a container, I can simply mount the container and copy it out.

# mnt=$(buildah mount CONTAINER_ID)
# buildah umount CONTAINER_ID

The beauty of this is we could use lots of tools, I could scp if I wanted to copy to another machine, or rsync, or ftp...

Once your have the container mounted up, you can use any bash command on it, to move files in or out.

buildah == simplicity

What capabilities do I really need in my container?

I have written previous blogs discussing using linux capabilities in containers.

Recently I gave a talk in New York and someone in the audience asked me about how do they figure out what capabilities their containers require?

This person was dealing with a company that was shipping their software as a container image, but they had instructed the buyer, that you would have to run their container ‘fully privileged”.  He wanted to know what privileges the container actually needed.  I told him about a project we worked on a few years ago, we called Friendly Eperm.

Permission Denied!  WHY?

A few years ago the SELinux team realized that more and more applications were getting EPERM returns when a syscall requested some access.  Most operators understood EPERM (Permission Denied) inside of a log file to mean something was wrong with the Ownership of a process of the contents it was trying to access or the permission flags on the object were wrong.  This type of Access Control is called DAC (Discretionary Access Control) and under certain conditions SELinux also caused the kernel to return EPERM.  This caused Operators to get confused and is one of the reasons that Operators did not like SELinux. They would ask, why didn’t httpd report that Permission denied because of SELinux?  We realized that there was a growing list of other tools besides regular DAC and SELinux which could cause EPERM.  Things like SECCOMP, Dropped Capabilities, other LSM …   The problem was that the processes getting the EPERM had no way to know why they got EPERM.  The only one that knew was the kernel and in a lot of cases the kernel was not even logging the fact that it denied access.  At least SELinux denials usually show up in the audit log (AVCs).   The goal of Friendly EPERM was to allow the processes to figure out why they got EPERM and make it easier for admin to diagnose.

Here is the request that talks about the proposal.


The basic idea was to have something in the /proc file system which would identify why the previous EPERM happened.  You are running a process, say httpd, and it gets permission denied. Now somehow the process can get information on why it got permission denied.  One suggestion was that we enhanced the libc/kernel to provide this information. The logical place for the kernel to reveal it would be in /proc/self.  But the act of httpd attempting to read the information out of /proc/self itself could give you a permission denied.  Basically we did not succeed because it would be a race condition, and the information could be wrong.

Here is a link to the discussion https://groups.google.com/forum/#!msg/fa.linux.kernel/WQyHPUdvodE/ZGTnxBQw4ioJ

Bottom line, no one has figured a way to get this information out of the kernel.


Later I received an email discussing the Friendly EPERM product and asking if there was a way to at least figure out what capabilities the application needed.

I wondered if the audit subsystem would give us anything here.  But I contacted the Audit guys at Red Hat, Steve Grubb and Paul Moore,  and they informed me that there is no Audit messages generated when DAC Capabilities are blocked.

An interesting discussion occurred in the email chain:

DWALSH: Well I would argue most developers have no idea what capabilities their application requires.

SGRUBB: I don't think people are that naive. If you are writing a program that runs as root and then you get the idea to run as a normal user, you will immediately see your program crash. You would immediately look at where it’s having problems. Its pretty normal to lookup the errno on the syscall man page to see what it says about it. They almost always list necessary capabilities for that syscall. If you are an admin restricting software you didn't write, then it’s kind of a  puzzle. But the reason there's no infrastructure is because historically it’s never been a problem because the software developer had to choose to use capabilities and it’s incumbent on the developer to know what they are doing.  With new management tools offering to do this for you, I guess it’s new territory.

But here we had a vendor telling a customer that it needed full root, ALL Capabilities,  to run his application,

DWALSH:  This is exactly what containers are doing.  Which is why the emailer is asking.  A vendor comes to him telling him it needs all Capabilities.  The emailer does not believe them and wants to diagnose what they actually need.

DWALSH: With containers and SELinux their is a great big "TURN OFF SECURITY" button, which is too easy for software packagers to do, and then they don't have to figure out exactly what their app needs.

Paul Moore - Red Hat SELinux Kernel Engineer suggested

That while audit can not record the DAC Failures, SELinux also enforces the capability checks.  If we could put the processes into a SELinux type that had no capabilities by default, then ran the process with full capabilities and SELinux in permissive mode, we could gather the SELinux AVC messages indicating which capabilities the application required to run.

“ (Ab)using security to learn through denial messages. What could possibly go wrong?! :)

After investigating further, turns out the basic type used to run containers, `container_t`, can be setup to have no capabilities by turning off an SELinux boolean.

To turn off the capabilities via a boolean, and put the machine into permissive mode.

setsebool virt_sandbox_use_all_caps=0

setenforce 0

Now execute the application via docker with all capabilities allowed.

docker run --cap-add all IMAGE ...

Run and test the application. This should cause SELinux to generate AVC messages about capabilities used.

grep capability /var/log/audit/audit.log

type=AVC msg=audit(1495655327.756:44343): avc:  denied  { syslog } for  pid=5246 comm="rsyslogd" capability=34  scontext=system_u:system_r:container_t:s0:c795,c887 tcontext=system_u:system_r:container_t:s0:c795,c887 tclass=capability2   


Now you know your list.

Turns out the application the emailer was trying to containerize was a tool which was allowed to manipulate the syslog system, and the only capability it needed was CAP_SYSLOG.  The emailer should be able to run the container by simply adding the CAP_SYSLOG capability and everything else about the container should be locked down.

docker run --cap-add syslog IMAGE ...


After writing this blog, I was pointed to

Find what capabilities an application requires to successful run in a container

Which is similar in that it finds out the capabilities needed for a container/process by using SystemTap.