> The main advantage of 1. is that it’s a bit simpler. My main concern with it is that multiple builds could occur in parallel that might cause issues. We had seen instance when running multiple instances of pip simultaneously caused the cache to become corrupted.
I believe that is what the ‘sharing’ option for cache mounts solve? Per docker docs[0]:
> One of ‘shared’, ‘private’, or ‘locked’. Defaults to ‘shared’. A shared cache mount can be used concurrently by multiple writers. private creates a new mount if there are multiple writers. locked pauses the second writer until the first one releases the mount.
So, for pip, can’t you just do:
RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
pip install -r requirements.txt
Although I’m not sure how that works with different python versions… Does docker link the cache to the specific FROM image? Probably not. I suggest using the 'id' argument with the python version, maybe:
RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-cache-py3-12 \
pip install -r requirements.txt
Although, the proxy solution is certainly a valid alternative.
In the old world a simple global http proxy would suffice to catch all such traffic, but I guess pip is also protecting my package installation behaviour against eavesdropping of foreign intelligence.
> The main advantage of 1. is that it’s a bit simpler. My main concern with it is that multiple builds could occur in parallel that might cause issues. We had seen instance when running multiple instances of pip simultaneously caused the cache to become corrupted.
I believe that is what the ‘sharing’ option for cache mounts solve? Per docker docs[0]:
> One of ‘shared’, ‘private’, or ‘locked’. Defaults to ‘shared’. A shared cache mount can be used concurrently by multiple writers. private creates a new mount if there are multiple writers. locked pauses the second writer until the first one releases the mount.
So, for pip, can’t you just do:
Although I’m not sure how that works with different python versions… Does docker link the cache to the specific FROM image? Probably not. I suggest using the 'id' argument with the python version, maybe: Although, the proxy solution is certainly a valid alternative.[0]: https://docs.docker.com/reference/dockerfile/#run---mounttyp...
Had no idea this feature existed, thanks for sharing!
I only discovered it a couple weeks ago, I’m glad I can help spread the word.
In the old world a simple global http proxy would suffice to catch all such traffic, but I guess pip is also protecting my package installation behaviour against eavesdropping of foreign intelligence.
No you can totally still do this. May want to set the requests CA if you're using self-provisioned cert.