As ever for Python, software environments are important when using Coiled. Users should refer to our docs (here) for complete information. In this article, we'll cover some additional tips and tricks that can be handy for users to know. The key point to remember here is that you need to have a suite of packages that are all consistent; not only with each other, but with Dask, Coiled (and Coiled Cloud).
Don't forget Dask. Something that is easy to forget is that when you create a Coiled software environment (and your local client software environment), is that you need to include
dask. We recommend this as as a
conda dependency, with the
conda-forge channel specified.
Use the conda-forge channel. For the best results, and latest Dask versions, use the
Include lz4 and python-blosc. Dask will pull in
lz4-c, but sometimes
lz4-c, is also useful. Neither
acoiledwill pull in
python-blosc, but sometimes issues will result if it is absent.
Example minimum recommended Coiled Software environments:
"python=3.9" # As of this writing, Dask does not support 3.10
For examples of the default software environments that Coiled uses, visit this page (click on the 'eye' icon. Note that explicitly specifying some of the packages shown on the Coiled software page is optional. Dask will pull in minimum required dependencies like pandas and numpy.
Example minimum requirements.yml file for a `conda env create -f requrements.yml` installation
Use care when relying upon packages to pull in dask. Sometimes, advanced users will create a package that relies upon Dask, and include
dask and other dependencies in the requirements.txt (or other specification) for that package. Then, they may create Coiled software environments like the following:
In principle, this should work fine. However, experience has shown that great care to check all the depedencies is required. In particular, we have encountered cases where
your-specific-package has included pins on dependencies that are not compatible with
coiled, or with Coiled Cloud (or even out of date versions of
coiled). Also, we have seen issues develop when unpinned packages update, and upon building a Coiled software environment you get a recent version that is no longer compatible with some other package that has been pinned to an older version.
Great care is needed in creating and managing Coiled software environments.