Three large-scale campaigns have targeted Docker Hub users, planting millions of repositories designed to push malware and phishing sites since early 2021.
The more interesting question is, how many downloads do they get?
So without those numbers, I see this as largely FUD. It doesn’t seem like trusted repos are getting infiltrated, so this sounds like a nothing burger. It’s good that Docker removed them, but without actual evidence of harm, it just seems cosmetic.
As of this writing, I saw one comment:
I get all kinds flack for not using docker containers and this is why I don’t. 20% is 1 in 5 containers, that cannot be trusted. I have no desire to to build my own Docker containers. I would much rather spin up a VM. CPU, RAM and disk space are cheap these days.
- Throwdown
This entirely misses the point of Docker. Yeah, 1 in 5 containers have malware, but how likely are you to be mislead into using one of those containers? I only pull containers based on some official project.
Then again, I’m a developer, but I don’t see anything here about Docker itself being a problem, and the vast majority of users will probably just follow links from some blog to accomplish some task, they won’t be downloading random images from the Docker hub…
I tried installing a docker image on my server at one point, Synology brand, but it didn’t work out for me very well, so I gave up relatively quick as it was more just a curiosity than a need or want.
Are docker images not easily installed on Synologys? I did read the guide for it, and did the commands. It just never worked for me.
I haven’t used Synology, so I’m not sure. So I’ll just explain a bit how Docker works, which will hopefully give some context.
Terminology:
image - the stuff hosted on https://hub.docker.com - basically the application and any dependencies it has
container - a disposable runtime for an image; stores new files separately from the image
By default, the container doesn’t have access to anything that’s not explicitly passed to it, kind of like a VM, but without the expensive overhead a VM has.
So the issues could be:
the image is incompatible with the host - pretty rare, but it happens
the container isn’t started
the container doesn’t have what it needs - maybe you need to expose a network port, or pass it a file
the Synology application isn’t configured to use the container
Synology has two types of Docker, “Docker” and “Container Manager”. If you’re using DSM 7.2 and later, please follow the guide for DSM 7.2, or follow the DSM 6 guide if you’re on older system.
Synology DSM 7.2
Synology DSM 6
If you are using Synology with Portainer, please check this tutorial.
If I had a Synology system to test with, I’d look into it for you, but that’s the best I’ve got. I run my own DIY NAS (old PC running openSUSE Leap w/ BTRFS, Samba, and minidlna configured).
The other user on this thread pointed out, as you have, that they’ve updated the guide to include DSM 7s Container Manager, which is what I was originally trying it on to begin with. So, thanks to both of you, I now have a project to do this weekend to test out the updated guides! :)
That’s the guide I was following, and the commands “went through” without any error message, but it just never fully worked for me. This was at least sometime around this time last year, so maybe if I give it another go, it’ll work?
Read logs, usually they can help a lot. To see the live log of a container (e.g… hbbs) use the docker logs -f hbbshttps://docs.docker.com/config/containers/logging/ Without any usable info it’s not possible to help you.
If I remember correctly, from a friend who was trying to do something on a Synology, there was something screwy about those devices.
Images that worked fine on a PC or a Pi just didn’t on the Synology. It’s been years so I don’t remember what, but don’t take those systems as a good example of Docker.
I don’t think that’s possible to know definitively, but I will say that this research doesn’t indicate that any images or repositories were compromised, only that a bunch of malware-ridden images were pushed to new repositories. In other words, it’s like they’re implying Google Drive is compromised because someone uploaded malware to their own account and are sending links to it. AFAIK, there wasn’t an attack on Docker as a platform or any existing repositories, these are just new repositories set up to host malware.
The proper mitigation here is to inform users how to find “official” images and repositories. Pulling a Docker image is like downloading any form of software, make sure you trust the source before pulling it down.
The problem is that rootless docker is a pain and no one does it
Nobody does it because it’s generally not necessary.
The reason to run system services as non-root is because there’s no sandbox to keep you safe. Docker provides that sandbox. Yeah, rootless containers are better security-wise since it provides another level of protection, but just using any sandbox follows the Pareto principle (80% benefit for 20% effort). Yeah, you probably should use rootless containers, but the focus should be on making sure the configuration for the container doesn’t expose more than necessary.
The following needs to happen for a user to be impacted:
download untrusted docker image - similar to downloading random executables from the internet
configure untrusted docker image with way more access than is necessary - as in, essentially eliminate the sandbox
run the container on a system with valuable data on it w/o backups - production systems should separate data from applications (i.e. mount points, S3-like buckets, etc), and development systems should have backups
There’s no Docker vulnerability here, so the title is unnecessarily alarmist. Docker hub is still absolutely safe to use, just make sure you trust the repositories and images you use.
What you are saying is not new but you don’t seem to grasp the difference in risk when you run someone else’s configured environment on your system vs. manually setting them up yourself. You save a lot of time by using docker images but it comes with a price.
There’s no docker vulnerability
No need to. Like sudo doesn’t need a vulnerability when you let contributors of some repository use it on your box.
Things like snyk exist for a reason but it’s not mitigation, just monitoring.
You should stop telling people that using docker is no security problem because that’s wrong, as it adds attack surface to even the most secure projects.
Sure, it saves time but things like OPs news will keep popping up in the future like it did in the past. It can’t be fixed other than just not using it in production. At least build your own containers.
I said it’s not a Docker problem in the same way that installing a malware-laden executable isn’t an OS problem. Yes, you can install malware through Docker if you opt-in, the trick is to not opt-in. Check where they’re coming from, look at multiple examples online and see if they’re all using the same image, etc. Do your due diligence, especially if you’re not a developer and thus looking at the Dockerfiles is impractical.
But unless there’s an actual vulnerability in Docker (i.e. exploit in the daemon, credential breach of Docker hub, etc), the finger should be pointed elsewhere. Maybe that’s the kernel (i.e. breaking out of the sandbox), but it’s likely blog posts and users that are at fault.
I looked through those links, and here are my thoughts:
Yes, Docker vulnerabilities are a problem, but an attacker would need to trigger the vulnerability and break out of the sandbox, so it’s not as serious imo; users should stay up to date though, especially if they’re using a popular base image
This comes down to user configuration; if you confine a volume properly, it shouldn’t be an issue.
This is really bad, and it’s heartening to see Docker take it seriously. A breach is by far my biggest concern when it comes to hosted services like this.
I’ll need to find a follow-up to see how Docker handles account deletions, because if names can be reused, that’s ripe for exploits.
Yes, there are plenty of opportunities for exploits in the Docker ecosystem, but I think the risk is pretty low given Docker has it’s own sandbox, which adds a layer of complexity to exploits. The focus shouldn’t necessarily be on something Docker should be doing, but teaching users how to configure containers properly. That means:
one process per container - nudges users to configure things properly, which reduces risk (limit access to data, executables, etc)
rebuild frequently - using more general names (e.g. python:3.10) instead of specific names (python:3.10.3) means you’re more likely to get security updates when you rebuild; this also means scans targeting big repos are more effective since that’s just one org to contact
do regular backups - ransomware attacks should be a complete non-issue, and other malware attacks should be reasonably easy to recover from
No platform is perfect, and practicing the above is probably easier than implementing non-root containers, and non-root itself isn’t a cure-all. Do the above first, and then consider non-root containers in production.
in the same way that installing a malware-laden executable isn’t an OS problem
except no one is doing that. Every major distro hast mechanisms for software supply chain security and reproducible builds.
Do your due diligence, especially if you’re not a developer and thus looking at the Dockerfiles is impractical.
You’re on to something here. If you automate that process, you end up with something we call a package manager.
it’s likely blog posts and users that are at fault.
Exactly. And sincer reviewing Dockerfiles is impractical, there’s no way docker prevents you from shooting your own foot. Distros learned that long ago: Insecure default configs or injected dependencies are a thing of the past there. With docker, those get reintroduced.
Given the number of posts I’ve seen online and adding PPAs or RPM repos, or installing things from source, I’d say that number is a lot higher than 0.
Docker contains that nonsense in a way that’s easy to update. And it allows developers to test production dependencies on a system without those production libs.
something we call a package manager.
Package managers don’t provide a sandbox. You can get one with SELinux, but that requires proper configuration, and then we’re back to the original issue.
Docker provides a decent default configuration. So I think the average user who doesn’t run updates consistently, may add sketchy dependencies, and doesn’t audit things would be better off with Docker.
So yeah, you may have more vulnerabilities with Docker, but they’re less likely to cause widespread issues since each is in its own sandbox. And due to the way its designed, it’s often easier to do things “the right way” (separate concerns) than the wrong way (relax security until it works).
adding PPAs or RPM repos, or installing things from source, I’d say that number is a lot higher than 0.
Nothing wrong with that. Unlike docker that’s cryptographically protected toolchain/buildchain/depchain. Thus, a PPA owner is much less likely to get compromised.
Installing things from source in a secure environment is about as safe as you can get, when obtaining the source securely.
Docker contains that nonsense in a way that’s easy to update.
Really? Ist there already a builtin way to update all installed docker containers?
What’s uneasy about apt full-upgrade?
Package managers don’t provide a sandbox.
I didn’t say that.
average user who doesn’t run updates consistently, may add sketchy dependencies, and doesn’t audit things would be better off with Docker.
That’s false.
but they’re less likely to cause widespread issues since each is in its own sandbox.
Also false. Sandbox evasion is very easy and the next local PE kernel vulnerability only weeks away. Also VM evasion is a thing.
Basically one compromised container giving local execution is enough to pwn your complete host.
Thus, a PPA owner is much less likely to get compromised.
Again, I have yet to see evidence of a Docker repo being compromised.
Repos are almost never compromised. Most issues are from users making things insecure because it’s easier that way. As in:
opening too many ports
enabling passwordless sudo
removing SELinux rules instead of just the parts they need
And so on. Docker makes it really easy to throw away the entire VPS and redeploy, which means those bad configs are less likely to persist in production.
Installing things from source in a secure environment is about as safe as you can get, when obtaining the source securely.
Sure, but that’s only true from the moment you release.
Source-compiled packages rarely get updated, at least in my experience. However, putting source packages in a Docker container increases those chances because it’s really easy to roll back if something goes wrong, and if you upgrade the base, it rebuilds the source with those new libraries, even if statically compiled.
If you’re talking about repository packages built from source (I’m not, I’m talking about side-loading packages), that’s not really a thing anymore with reproducible builds.
Ist there already a builtin way to update all installed docker containers?
In our CI pipeline, yes, I just run a “rebuild all” script and it’s ready for the next deploy to production. Or the script run could be totally automated (I’m currently fighting our devOPs to enable it). We do “<thing>:<version>-<base>” pattern, so worse case, we update from “bullseye” to “bookworm” or whatever separately from the version and ship that. Total process is like 20s per repo (edit Dockerfile out docker-compose.yml, make PR, then ship), then whatever time it takes to build and deploy.
But I don’t necessarily want to upgrade everything. If I upgrade the OS version, I need to check that each process I use is compatible, so I’m more likely to delay doing it. With Docker, I can update part of my stack and it won’t impact anything else. At work, we have dozens of containers, all at various stages of updates (process A is blocked by dependency X, B is blocked by Y, etc), so we’re able to get most of them updated independently and let the others lag until we can fix whatever the issues are.
When installing to the host, I’d need to do all of it at once, which means I’m more likely to stay behind on updates.
That’s false.
Source? I’m coming from ~15 years of experience, and I can say that servers rarely see updates. Maybe it happens in larger firms, but not in smaller shops. But then, larger firms can also run security audits of docker images and whatnot.
Basically one compromised container giving local execution is enough to pwn your complete host.
Maybe. That depends on if the attacker is able to get out of the sandbox. If it was a vulnerability in a process on the host, there’s no sandbox to escape.
So while your Docker containers will probably lag a bit, they come with a second layer of protection. If your host lags a bit, there’s probably no second layer of protection.
The more interesting question is, how many downloads do they get?
So without those numbers, I see this as largely FUD. It doesn’t seem like trusted repos are getting infiltrated, so this sounds like a nothing burger. It’s good that Docker removed them, but without actual evidence of harm, it just seems cosmetic.
As of this writing, I saw one comment:
This entirely misses the point of Docker. Yeah, 1 in 5 containers have malware, but how likely are you to be mislead into using one of those containers? I only pull containers based on some official project.
For example, if I want to host nextcloud, there’s an “official” image for that. For CI, I generally use images from “official” sources (e.g. this one from Rust for Rust projects).
Then again, I’m a developer, but I don’t see anything here about Docker itself being a problem, and the vast majority of users will probably just follow links from some blog to accomplish some task, they won’t be downloading random images from the Docker hub…
Thank you for your in depth explanation.
I have never used docker, so I was thinking it was a bigger issue. You have given very good points, and I now know a little more about docker.
Thank you!
I’m happy to explain more if you’d like. :) I use it at work and home, so I know a thing or two.
I tried installing a docker image on my server at one point, Synology brand, but it didn’t work out for me very well, so I gave up relatively quick as it was more just a curiosity than a need or want.
Are docker images not easily installed on Synologys? I did read the guide for it, and did the commands. It just never worked for me.
To give more info, it was for RustDesk self host.
Also, thanks for being so nice and kind!
I haven’t used Synology, so I’m not sure. So I’ll just explain a bit how Docker works, which will hopefully give some context.
Terminology:
By default, the container doesn’t have access to anything that’s not explicitly passed to it, kind of like a VM, but without the expensive overhead a VM has.
So the issues could be:
For Synology and RustDesk specifically, I found this in the instructions:
If I had a Synology system to test with, I’d look into it for you, but that’s the best I’ve got. I run my own DIY NAS (old PC running openSUSE Leap w/ BTRFS, Samba, and minidlna configured).
The other user on this thread pointed out, as you have, that they’ve updated the guide to include DSM 7s Container Manager, which is what I was originally trying it on to begin with. So, thanks to both of you, I now have a project to do this weekend to test out the updated guides! :)
Rustdesk has a section specifically for Synologys: https://rustdesk.com/docs/en/self-host/rustdesk-server-oss/synology/ Have you read this?
Yep!
That’s the guide I was following, and the commands “went through” without any error message, but it just never fully worked for me. This was at least sometime around this time last year, so maybe if I give it another go, it’ll work?
There are some issues about this guides in the docs repo maybe worth to read them as well: https://github.com/rustdesk/doc.rustdesk.com/issues?q=is%3Aissue+synology
Read logs, usually they can help a lot. To see the live log of a container (e.g… hbbs) use the
docker logs -f hbbshttps://docs.docker.com/config/containers/logging/ Without any usable info it’s not possible to help you.WOAH!
They didn’t have a separation of DSM 6 and 7 before, so I just kind of winged it.
Thank you SO much for pointing me in the right direction! It seems I have a project to do this weekend! :)
If I remember correctly, from a friend who was trying to do something on a Synology, there was something screwy about those devices.
Images that worked fine on a PC or a Pi just didn’t on the Synology. It’s been years so I don’t remember what, but don’t take those systems as a good example of Docker.
It’s just pointing out the risk of letting someone you don’t know with no legal obligations setup your complete environment.
Probably as likely as someone cracking your really secure ssh password. Still, any sane expert will recommend disabling password auth.
How do you know they weren’t compromised?
The problem is that rootless docker is a pain and no one does it. Privileged software sideloading other software is a huge risk.
That risk now became an incident. Even if you’re not affected, the risk still remains.
This is why I use my own Images if I can afford it.
I don’t think that’s possible to know definitively, but I will say that this research doesn’t indicate that any images or repositories were compromised, only that a bunch of malware-ridden images were pushed to new repositories. In other words, it’s like they’re implying Google Drive is compromised because someone uploaded malware to their own account and are sending links to it. AFAIK, there wasn’t an attack on Docker as a platform or any existing repositories, these are just new repositories set up to host malware.
The proper mitigation here is to inform users how to find “official” images and repositories. Pulling a Docker image is like downloading any form of software, make sure you trust the source before pulling it down.
Nobody does it because it’s generally not necessary.
The reason to run system services as non-root is because there’s no sandbox to keep you safe. Docker provides that sandbox. Yeah, rootless containers are better security-wise since it provides another level of protection, but just using any sandbox follows the Pareto principle (80% benefit for 20% effort). Yeah, you probably should use rootless containers, but the focus should be on making sure the configuration for the container doesn’t expose more than necessary.
The following needs to happen for a user to be impacted:
There’s no Docker vulnerability here, so the title is unnecessarily alarmist. Docker hub is still absolutely safe to use, just make sure you trust the repositories and images you use.
What you are saying is not new but you don’t seem to grasp the difference in risk when you run someone else’s configured environment on your system vs. manually setting them up yourself. You save a lot of time by using docker images but it comes with a price.
No need to. Like sudo doesn’t need a vulnerability when you let contributors of some repository use it on your box.
Things like snyk exist for a reason but it’s not mitigation, just monitoring.
You should stop telling people that using docker is no security problem because that’s wrong, as it adds attack surface to even the most secure projects. Sure, it saves time but things like OPs news will keep popping up in the future like it did in the past. It can’t be fixed other than just not using it in production. At least build your own containers.
Don’t forget various past issues:
I said it’s not a Docker problem in the same way that installing a malware-laden executable isn’t an OS problem. Yes, you can install malware through Docker if you opt-in, the trick is to not opt-in. Check where they’re coming from, look at multiple examples online and see if they’re all using the same image, etc. Do your due diligence, especially if you’re not a developer and thus looking at the Dockerfiles is impractical.
But unless there’s an actual vulnerability in Docker (i.e. exploit in the daemon, credential breach of Docker hub, etc), the finger should be pointed elsewhere. Maybe that’s the kernel (i.e. breaking out of the sandbox), but it’s likely blog posts and users that are at fault.
I looked through those links, and here are my thoughts:
Yes, there are plenty of opportunities for exploits in the Docker ecosystem, but I think the risk is pretty low given Docker has it’s own sandbox, which adds a layer of complexity to exploits. The focus shouldn’t necessarily be on something Docker should be doing, but teaching users how to configure containers properly. That means:
No platform is perfect, and practicing the above is probably easier than implementing non-root containers, and non-root itself isn’t a cure-all. Do the above first, and then consider non-root containers in production.
except no one is doing that. Every major distro hast mechanisms for software supply chain security and reproducible builds.
You’re on to something here. If you automate that process, you end up with something we call a package manager.
Exactly. And sincer reviewing Dockerfiles is impractical, there’s no way docker prevents you from shooting your own foot. Distros learned that long ago: Insecure default configs or injected dependencies are a thing of the past there. With docker, those get reintroduced.
Given the number of posts I’ve seen online and adding PPAs or RPM repos, or installing things from source, I’d say that number is a lot higher than 0.
Docker contains that nonsense in a way that’s easy to update. And it allows developers to test production dependencies on a system without those production libs.
Package managers don’t provide a sandbox. You can get one with SELinux, but that requires proper configuration, and then we’re back to the original issue.
Docker provides a decent default configuration. So I think the average user who doesn’t run updates consistently, may add sketchy dependencies, and doesn’t audit things would be better off with Docker.
So yeah, you may have more vulnerabilities with Docker, but they’re less likely to cause widespread issues since each is in its own sandbox. And due to the way its designed, it’s often easier to do things “the right way” (separate concerns) than the wrong way (relax security until it works).
Nothing wrong with that. Unlike docker that’s cryptographically protected toolchain/buildchain/depchain. Thus, a PPA owner is much less likely to get compromised.
Installing things from source in a secure environment is about as safe as you can get, when obtaining the source securely.
Really? Ist there already a builtin way to update all installed docker containers?
What’s uneasy about
apt full-upgrade?I didn’t say that.
That’s false.
Also false. Sandbox evasion is very easy and the next local PE kernel vulnerability only weeks away. Also VM evasion is a thing.
Basically one compromised container giving local execution is enough to pwn your complete host.
Again, I have yet to see evidence of a Docker repo being compromised.
Repos are almost never compromised. Most issues are from users making things insecure because it’s easier that way. As in:
And so on. Docker makes it really easy to throw away the entire VPS and redeploy, which means those bad configs are less likely to persist in production.
Sure, but that’s only true from the moment you release.
Source-compiled packages rarely get updated, at least in my experience. However, putting source packages in a Docker container increases those chances because it’s really easy to roll back if something goes wrong, and if you upgrade the base, it rebuilds the source with those new libraries, even if statically compiled.
If you’re talking about repository packages built from source (I’m not, I’m talking about side-loading packages), that’s not really a thing anymore with reproducible builds.
In our CI pipeline, yes, I just run a “rebuild all” script and it’s ready for the next deploy to production. Or the script run could be totally automated (I’m currently fighting our devOPs to enable it). We do “<thing>:<version>-<base>” pattern, so worse case, we update from “bullseye” to “bookworm” or whatever separately from the version and ship that. Total process is like 20s per repo (edit Dockerfile out docker-compose.yml, make PR, then ship), then whatever time it takes to build and deploy.
But I don’t necessarily want to upgrade everything. If I upgrade the OS version, I need to check that each process I use is compatible, so I’m more likely to delay doing it. With Docker, I can update part of my stack and it won’t impact anything else. At work, we have dozens of containers, all at various stages of updates (process A is blocked by dependency X, B is blocked by Y, etc), so we’re able to get most of them updated independently and let the others lag until we can fix whatever the issues are.
When installing to the host, I’d need to do all of it at once, which means I’m more likely to stay behind on updates.
Source? I’m coming from ~15 years of experience, and I can say that servers rarely see updates. Maybe it happens in larger firms, but not in smaller shops. But then, larger firms can also run security audits of docker images and whatnot.
Maybe. That depends on if the attacker is able to get out of the sandbox. If it was a vulnerability in a process on the host, there’s no sandbox to escape.
So while your Docker containers will probably lag a bit, they come with a second layer of protection. If your host lags a bit, there’s probably no second layer of protection.