Repository navigation
Description
I've been flailing away at the idea to run a pool of rootless containers as children of a docker container. My intent is to have the docker container run a web server that will spin up a pool of child, rootless containers to which requests can be proxied. These children would be designed to be isolated from each other and the host system from the side-effects of running untrusted code.
I need to pass additional file descriptors to these children which precludes running children as siblings using the host docker daemon. So here I am and I hope I'm not overstepping my bounds by asking for guidance via an issue.
Set up
Create a root filesystem tgz:
$ docker export $(docker create alpine) > rootfs.tgzDockerfile with runc, libseccomp2 and the rootfs:
FROM buildpack-deps
RUN apt-get update && apt-get install -y --no-install-recommends \
libseccomp2 \
&& rm -rf /var/lib/apt/lists/*
ADD rootfs.tgz /child/rootfs
ADD runc /usr/local/sbin/runc
WORKDIR /child/rootfs
RUN runc spec --rootless
CMD ["runc", "run", "child"]False starts:
Build and run the container, adding CAP_SYS_ADMIN:
$ docker run --rm -it --cap-add SYS_ADMIN $(docker build -q .)
container_linux.go:265: starting container process caused "process_linux.go:261: applying cgroup configuration
for process caused \"mkdir /sys/fs/cgroup/cpuset/child: read-only file system\""Same, but mount /sys/fs/cgroup as rw:
$ docker run --rm -it --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:rw $(do
cker build -q .)
container_linux.go:265: starting container process caused "process_linux.go:339: container init caused \"could
not create session key: operation not permitted\""Same, but invoke runc with --no-new-keyring:
$ docker run --rm -it --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:rw $(do
cker build -q .) runc run --no-new-keyring child
container_linux.go:265: starting container process caused "process_linux.go:339: container init caused \"rootfs
_linux.go:104: jailing process inside rootfs caused \\\"pivot_root operation not permitted\\\"\""Finally 'working':
Same, but also add --no-pivot:
$ docker run --rm -it --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:rw $(do
cker build -q .) runc run --no-new-keyring --no-pivot child
/ #Disclaimer: I'm still wrapping my head around all of the complexity and nuances of all the technologies we call 'containers' so please correct me if I'm wrong.
Removing pivot_root seems like a bad idea given my objectives so I created a copy of the default seccomp profile and added the pivot_root syscall to the big list of SCMP_ACT_ALLOW calls. This let me drop --no-pivot.
What kind of exposure am I creating by opening up by whitelisting the pivot_root syscall?
Also, I'm past my abilities in trying to figure out how I might avoid --no-new-keyring
What kind of exposure am I creating by using the --no-new-keyring flag?
Activity
cyphar commentedon May 24, 2017
You are already running privileged containers at this point. You'd need to do
--cap-drop alland a few other flags to entire drop privileges inside Docker. Basically you're running as mostly-root if you add that capability (and Docker has a bunch enabled by default).None really.
pivot_rootis a more securechroot. Docker disables it because it involves messing with mount namespaces (which normal containers shouldn't be doing) but it's not a security issue in principle (maybe @jessfraz might remember why it was added).Processes inside the container can access the host's kernel keyring directly (which contains various crypto stuff that some system components use) if a process is running with privileges. Though processes inside the inner container (if you're using user namespaces) might be blocked (I haven't read that kernel code in a while though).
In general I would discourage it, but to be honest if you don't specify
--no-new-keyringI would wager that there's a kernel-level hole that just hasn't been solved yet from the kernel side.cyphar commentedon May 24, 2017
Oh sorry, this too
Rootless containers most definitely cannot do this. The reason that cgroups actually work in the inner container is because your container has
CAP_SYS_ADMIN, which is basically the majority ofrootfunctionality.ggoodman commentedon May 24, 2017
@cyphar thank you for taking the time. I'm super excited about the whole rootless container story and all the work you've been doing.
I've now been able to get a working prototype of the concept described above. The key differences are that I have been unable to get the networking setup without adding
CAP_NET_ADMINto the parent, docker container and running the container's process as root. Despite running as root, I still seem to need to mount the hostcgroupfs and run the child sandboxes with the--no-new-keyringoption.Would it be fair to say that the machinery is not yet in place to be able to run rootless containers with networking as children of unprivileged containers themselves?
To provide network access, I've picked up @jessfraz' netns to great, frictionless effect and run this as a prestart hook. This also meant mounting the host's
/lib/modules(from the alpine-based xhyve vm used by docker-for-mac) and installingiptablesandkmod(formodprobe) in the debian-based parent docker container.Given the lack of granularity in
CAP_SYS_ADMIN, are we forced to play withseccompfilters as a mechanism to limit access to the components of this flag's capabilities?Please feel free to close at any time given that this isn't so much of an issue as it is a discussion. I hope that it helps someone else in their wacky experimentation down the road.
jessfraz commentedon May 25, 2017
This was merely not included to prevent people from shooting themselves in the foot, it should be a privileged operation, also if people were using pivot_root they were probably using other things that were blocked as well like mount or chroot so it kinda just went hand in hand, there wasn't an exact reason
frezbo commentedon Dec 27, 2017
@ggoodman @cyphar Good to see the awesome discussion, I'm also trying to run runc inside a container as a normal user (rootless), where I'm facing much issues, and I do not want to run the base container will all the privileges, I was able to get it working with udocker (https://github.com/indigo-dc/udocker), but I was hoping I could use runc and avoid all the other python dependency, the problem with udocker being it does not support OCI complaint images, so I would like to get your opinions on whats the best approach and the hurdles came across while running them. Thanks.
cyphar commentedon Dec 27, 2017
@frezbo You can use skopeo to convert a Docker image into an OCI image (I've added support to
skopeoto allow you to convert a local file that you would generate fromdocker save). If you then want to unpack the OCI image into an OCI runtime configuration that can be used byrunc, you can use umoci, which is a tool I wrote.umocihas a--rootlessflag which generates a rootless OCI runtime configuration and root filesystem without needing root (please read the relevant section in umoci's documentation for caveats).frezbo commentedon Dec 27, 2017
@cyphar I have used both the awesome tools by you, and the relevant issues is here: indigo-dc/udocker#111, udcoker can only understand v1 spec and skopeo copies from oci to v2, so I'm planning to use runc inside a container, with the least possible privilege escalation
kolyshkin commentedon Oct 8, 2026
Closing, as this is more of a discussion than an issue (as noted by the author above), and a lot has changed since 2017 (e.g. cgroup v2 with delegation, rootless networking via slirp4netns or pasta). If you have a specific problem running rootless runc inside a container with a recent runc version, please open a new issue.