Optional dependencies in conda? Discuss in guide?

Questions about Python Packaging & Code
1

Hi all :wave:

A question came up during a review about optional dependencies and conda

I tried to sum up what I know in a reply:

Basically:

@ocefpaf @chenghlee please chime in if I’m not explaining this well

I also suggested we might add some mention of this somewhere in the guide. It might be too much in a tutorial to try and explain how pip relates to conda, etc., but we do in the tutorials show the use of optional dependencies for adding e.g. tests.
Should we address this somewhere @lwasser?

edit: see issue on Guide here: Request discussion of distributing with conda if you have optional pip dependencies · Issue #173 · pyOpenSci/python-package-guide · GitHub

1 Like
2

Something I forgot to mention is that, sometimes you don’t want to manage optional dependencies in multiple outputs but you do want to control the versions the user will install later on. For that you can use run_constrained like in xarray-feedstock/recipe/meta.yaml at 6d0be5d030b7be2064cf871a2c4f418f650ff77b · conda-forge/xarray-feedstock · GitHub

IMO, that with an import error message telling the user to install the missing package, is the most friendly way to implement optional packages. The reason is b/c one must dig into the packages code to figure out the names for the optional packages. I find it more user friendly to try to use something and get the error “install x to make this work.”

2 Likes
3

thanks so much for this post @NickleDave and the discussion @ocefpaf :sparkles: i need to ask a basic question. When we are talking about optional dependencies, i suspect we are talking about (considering the way we break things down here)…

1280×1200 78 KB

… dependencies that are feature dependencies that you want to list in a conda recipe? is that the case? my apologies if i am missing something important, David. i just want to make sure i follow.

it looks like that nbconvert example has two install groups nbconvert and nbconvert-core each of which run against slightly different versions of deps?

i feel like i know how to get a pure python package to conda-forge but i’ve never worked on a pure conda package build. Regardless i’m happy to take some time to understand the issue so we can collectively add some content to our guide!! as it sounds like it would be useful and we can get help on process from awesome folks like @ocefpaf and @chenghlee .

4

Thank you for asking, that is really helpful context I did not take the time to add

:100: yes those are exactly the optional dependencies that I mean.

And I think the advice we give might depend on whether they are optional feature dependencies or optional development dependencies.
I think what can happen is that a scientist who starts learning with conda tries to do development and so they end up with e.g. three env.yaml files, one for test dependencies, one for build dependencies, one for docs.

re: optional dev deps, If it were up to me I would suggest “develop in pure Python if it all possible and use optional development dependencies as we show here, then after you publish a pure Python package, rely on conda-forge to build a conda binary for you” – for the simple reason that there’s one less moving part to worry about during development. To be clear: conda is great and exists for a reason, and people might need to use it; I’m just saying this rec is to make life easier for someone who is new to this and wants to take the path of least resistance

re: optional feature deps, my experience lines up with @ocefpaf where I rarely if ever use optional features? But I know there are def core scientific Python packages that rely on it, e.g. scikit-image for installing pooch to download bigger datasets, and dask

1 Like
5

Hi all a bit late to the party here, just joined the group. I’m the one that opened the issue at github, and I’m here to learn and listen and say hi :wave:

1 Like
6

Welcome @Eric! Very glad you could join

7

Typically I’ve see the pattern where “[PACKAGE]” comes with all optional feature dependencies and then “[PACKAGE]-base” (or -core, etc) comes without any of the optional dependencies. Matplotlib, is another good example of doing that:

and I’ve had to set it up for a few packages I maintain at this point. It definitely would be nice to provide direct optional dependency support in conda (though I’m not sure how well that tracks with other languages’ packaging conventions).

1 Like
8

And I don’t think I’d worry about developmental dependencies related to conda. I don’t think conda supports editable installs, and generally, on the development side, even when using conda environments, you’ll see a pattern like:

git clone [THING] && cd [THING]
conda env create -f environment.yml
conda activate [THING-ENV]
python -m pip install --no-deps -e .

with a bunch of variations in the pip command itself.

1 Like