Don’t pin base image exact version tags
Published Aug 7, 2023
Google for “secure container image best practices” and you will see dozen of articles from big and reputable companies giving you the same advice:
This means that the base image for your Docker file might change between builds. This could result in inconsistent behavior because of changes made to the base image. There are multiple ways to mitigate this issue and improve your Docker security posture:
- Prefer the most specific tag available. If the image has multiple tags, such as :8 and :8.0.1 or even :8.0.1-alpine, prefer the latter, as it is the most specific image reference. Avoid using the most generic tags, such as latest. Keep in mind that when pinning a specific tag, it might be deleted eventually (*)
Sounds reasonable? Oh yes, it is. And also it falls into my favorite category of advice: if you are a small team, most “best practices” from technology giants have zero relevance to you. “These stunts are performed by trained professionals, don’t try this at home”
Every single small team I’ve seen following this advice ended up running MANY vulnerable components in the production environment. IT HURTS.
Without a dedicated security operations team monitoring SBOMs, life cycles, vulnerabilities, and EOLs, this guidance will put you at risk. You might overlook vulnerabilities and create a flawed image or end up in a never-ending manual vulnerability management cycle with your engineering team, while the advantage of reproducible builds has marginal practical value. You can just pull an older image, remember?
So I would advise you to do exactly the opposite thing. Use the “latest” tag whenever possible.
It may be extreme, depending on the software nature; if it is “too dynamic” and tends to break the backward compatibility, you can pin major or even minor versions but definitely not the exact build or worse, the image hash.
Additional tips:
Scan rigorously, don’t allow security scans to fail, and conduct periodic scans outside pipelines.
Maintain image-specific guidance, to share knowledge of exactly how volatile a specific piece of software is. Like, “golang and nginx are safe for latest, for node.js you should pin the major version, and for Python the minor one”. Semantic versioning is supposed to handle that, but YMMV.
Increase test coverage, as things WILL break this way. Fixing occasional issues on pipeline failure still demands fewer resources than manual vulnerability management.
Ditch Debian, it’s a mess due to its bloat and its complex vulnerability management process inevitably leads to false alarms. Use Alpine or distroless instead.
Update source dependencies. Unfortunately, the same approach falls short — dependencies are too many, developers may be careless, there are supply chain attacks etc. — but you need a simple way to keep up to date, and if there is a security bug, do it right away. Consider something like “ go vulncheck ./… || go get -u ./… “ or “npm audit fix”, and do a casual version bump once a while. If nothing goes awry, stick with it; if issues arise, add them to the backlog.
Start collecting full SBOM, it is handy.
(*) https://snyk.io/blog/10-docker-image-security-best-practices/