Tips & Tricks - Software Environments

As ever for Python, software environments are important when using Coiled.  Users should refer to our docs (here) for complete information.  In this article, we'll cover some additional tips and tricks that can be handy for users to know.  The key point to remember here is that you need to have a suite of packages that are all consistent; not only with each other, but with Dask, Coiled (and Coiled Cloud).  

Don't forget Dask. Something that is easy to forget is that when you create a Coiled software environment (and your local client software environment), is that you need to include dask.  We recommend this as as a conda dependency, with the conda-forge channel specified.

Use the conda-forge channel.  For the best results, and latest Dask versions, use the conda-forge channel.

Include lz4 and python-blosc.  Dask will pull in lz4-c, but sometimes lz4-c, is also useful.  Neither dask nor acoiledwill pull in python-blosc, but sometimes issues will result if it is absent.

Example minimum recommended Coiled Software environments:

import coiled
coiled.create_software_environment(
    conda={
        "channels": [
             "defaults",
             "conda-forge", 
         ],
        "dependencies": [
            "dask",
            "lz4",
            "python=3.9"              # As of this writing, Dask does not support 3.10
            "python-blosc",
        ],
    },
)

For examples of the default software environments that Coiled uses, visit this page (click on the 'eye' icon.  Note that explicitly specifying some of the packages shown on the Coiled software page is optional.  Dask will pull in minimum required dependencies like pandas and numpy.  

Example minimum requirements.yml file for a `conda env create -f requrements.yml` installation

name: coiled-env
channels:
  - conda-forge
  - defaults
dependencies:
  - dask
  - lz4
  - python=3.9
  - python-blosc

Use care when relying upon packages to pull in dask.  Sometimes, advanced users will create a package that relies upon Dask, and include dask and other dependencies in the requirements.txt (or other specification) for that package.  Then, they may create Coiled software environments like the following:

import coiled
coiled.create_software_environment(
    conda={
        "channels": [
            "defaults",
            "conda-forge", 
        ],
        "dependencies": [
            "your-specific-package",
        ],
    },
)

In principle, this should work fine.  However, experience has shown that great care to check all the depedencies is required.  In particular, we have encountered cases where your-specific-package has included pins on dependencies that are not compatible with coiled, or with Coiled Cloud (or even out of date versions of coiled).  Also, we have seen issues develop when unpinned packages update, and upon building a Coiled software environment you get a recent version that is no longer compatible with some other package that has been pinned to an older version. 

Great care is needed in creating and managing Coiled software environments.