API Reference building locally but not on RTH - missing path?

Hello everyone,

thanks for pointing me to this community!

I have a question regarding my GemGIS package. I am in the process of building the API Reference on RTD. Even though it builds locally, the API fails to build due to Import Errors and Module Not Found Errors.

For reference, please see python sphinx - API Reference building locally but not on RTD - Stack Overflow where Dave Nickle has already proposed a solution which did not work. By now, I have removed all but two methods to get a mini API Reference running but get the error that another module is missing which only appears in the __init__.py file. So maybe there is something missing there…here the link to one of the latest builds.

Maybe someone has another idea what I am missing here (probably something very small and easy to fix).

Cheers Alex

2 Likes

hey there @AlexanderJuestel !! welcome to pyOpenSci :pyos_animated: And also cool to see another software-underground community member here :sparkles:

We can get you to a solution where your documentation builds correctly in RTD. But i wondered if we could work together (we can support you here) to update the infrastructure of your package to make it a bit more build / community friendly with modern packaging approaches??

there are a handful of things that i think we can do to clean things up which i think will make your build generally a bit easier. are you open to that?

if so, we’d like to start by moving you away from using a setup.py file to a pyproject.toml file to declare metadata, dependencies, build tools, etc.

then we can start to look at your dev environment files together. there are a few other things as well. but i wonder if working on things one at a time is best in case we need to call in expertise from other folks on particular items.

let me know what you think.
Leah

1 Like

Dear @lwasser,
thank you for your message! I would gladly accept your help in making GemGIS even better! I do like the step-by-step approach!

How would we proceed from here?

1 Like

@AlexanderJuestel excellent!! :sparkles:

This is actually going to also help pyOpenSci and me greatly! Because i’m working on tutorials for packaging and it will help me call attention to any points of confusion, etc!
let’s start by migrating your setup.py file to a pyproject.toml file.

have a look at this page - as a point of guidance.

so this file - https://github.com/cgre-aachen/gemgis/blob/main/setup.py can eventually go away and we’ll replace it with a pyproject.toml file. i don’t yet have our example package setup (working on that!) but you can look at this example here to get started.

it looks like you’re using setuptools to build so you’re build section would look something like this

[build-system]
requires = ["setuptools>=45"] 
build-backend = "setuptools.build_meta"

let us know if you have any questions with this first step. There is some explanation of this step in the link in our guide above! the benefit of doing this is multifold but i think the biggest value is how readable the file is making all of your dependencies, metadata, etc super explicit collected in a single file. feel free to do this, submit a pr and if you have questions you can ping me there in the pr.

@AlexanderJuestel you might find this pyproject.toml easier to digest as i’ve left comments. but do ask any questions that come up as well. the main difference in that pyproject.toml is i’m using pdm rather than setuptools. but i tried to document it well.

Dear @lwasser,

I created a first project.toml file (https://github.com/cgre-aachen/gemgis/blob/dev_gemgis3/project.toml) and filled in information that I felt somewhat comfortable with adding. However, I find it quite difficult to know what to add, what options there are etc. your commented code already helps a lot but maybe an overview of what can be put in a project.toml file could be something for your website in the future?

Cheers
Alex

1 Like

@AlexanderJuestel what a great start!

a few notes on your pr
each section starts with a [text-here].
You can think of the items below that as list elements in some ways. so for example your build-system looks like this

[build-system]
requires = ["setuptools>=68.1.2",
            "wheel",
            "setuptools_scm[toml]>=6.2"]
            
build-backend = "setuptools.build_meta"

it should look like this instead. you can remove wheel now because it should install automagically with the newer versions of setuptools!

[build-system]
requires = ["setuptools>=68.1.2",
            "setuptools_scm[toml]>=6.2"]
build-backend = "setuptools.build_meta"

Below i’ve removed the commented out maintainers section (that is an optional section)

[project]
name = "GemGIS"
authors = [
    {name = "Alexander Jüstel", email = "alexander.juestel@rwth-aachen.de"},
]
description = "Spatial data processing for geomodeling"
keywords = ["visualization", "python", "processing", "jupyter", "modeling", "geospatial", "jupyter-notebook",
            "geographic-data", "spatial-data", "notebooks", "raster-data", "vector-data", "geographic", "geomodeling"]
readme = "README.md"
license = {file = LICENSE}
dynamic = ["Version"]
requires-python = ">=3.8"
classifiers = [
    'Development Status :: 5 - Production/Stable',
    'Intended Audience :: Science/Research',
    'Topic :: Scientific/Engineering :: Information Analysis',
    'Programming Language :: Python :: 3.8',
    'Programming Language :: Python :: 3.9',
    'Programming Language :: Python :: 3.10',
    'Programming Language :: Python :: 3.11',
    'License :: OSI Approved :: LGPL-3.0-only License', #
    'Operating System :: OS Independent',
]
dependencies = [
    "geopandas",
    "rasterio",
    "pyvista",
]
  1. i think your license will throw an error - it needs to be in the list of valid classifiers listed on pypi to be recognized in the metadata.

  2. And are you using dynamic versioning? if you are we can leave setuptools_scm there and set that up. what that means is that setuptools will derive the version from a git tag that you create rather than you having to bump it yourself

However, I find it quite difficult to know what to add, what options there are etc. your commented code already helps a lot but maybe an overview of what can be put in a project.toml file could be something for your website in the future?

Great timing! We are working on a pyproject.toml tutorial now. .

So what would you like to see? do you want to see

  • ALL elements that could go into the metadata for such a file
  • which are required
  • options for required elements that have format and word requirements such as the python version classifiers??

Feel free to add comments to that google doc above. we will be sprinting on creating these resources over the next months as a community with the first meeting starting today!

also @AlexanderJuestel here is a small tutorial we’re working on - on this specific file. feel free to leave any comments, edits as you see fit.

and here is an excellent resource from pypa

@lwasser sorry for the late reply. My PhD kept me busy.

I changed the toml file according to your suggestions. You can find the latest version here: https://github.com/cgre-aachen/gemgis/blob/dev_gemgis3/project.toml

What I would like to see in a project.toml tutorial would be what you have mentioned, so a minimum working file with explanations of what each required field means and which options you have and optional fields with explanations on how to add the metadata.

1 Like

and i’ve been offline for a few weeks so i’m just catching up myself now!!

ok excellent! i made a note in our tutorial to add a minimal example that we can then flesh out more with additional fields for users that want to use them!!

Remove your setup.py file

Now that your pyproject.toml file is good to go, you can remove setup.py altogether. at this point it’s only populating metadata and your more readable pyproject.toml file covers that. Before you remove the setup.py file, be sure that all of the metadata in there is in the pyproject.toml. then you can remove it and try to build locally. it should build properly without the setup.py!.

Metadata in your init.py file

i also see some metadata in your init.py file. I am fairly certain we can remove that entirely but i also note that there are some authors in there and info that may not be in your pyproject.toml file. Let’s for now make sure that ALL of the info in that section is also in your pyproject.toml file.

i’ve actually also asked folks in our slack about how / if that information would be used so someone may respond here to that as well.

1 Like

i didn’t want the above to get to long - do that first!!

but then let’s get versioning setup for you using setuptools_scm - steps are below. Take this one step at a time and ask questions if things don’t make sense! it’s a small set of changes but i know it can be a bit confusing when we first work on this part!!

Versioning

The next thing i want to do is talk about how you handle versioning. I see that you have setuptools_Scm listed in your pyprioject.toml file and you have an action that pushes to pypi when you make a release! both are excellent.

I also see however, that you have versions that appear to be manually entered throughout your package.
If you use setuptools_scm then you can upon build, use the tags that you are creating to version your package. This means that you’ll never have to manually add a version number again! yay to no more potential for human error in versioning :slight_smile: and a single point of truth for the version!

to do this you’ll want to add a few things to your setup. For reference you can use stravalib as i set the same thing up there!

Let’s get setuptools_Scm working for you here…

Step one - update your pyproject.toml file

we use a src/ layout so i’ll modify here given you are using a flat layout right now. but you can see how things work there if you wish.

add a section like this to your pyproject.toml file. Note that i adjusting the paths for you to NOT use src given your package uses a flat layout not a src layout.

[tool.setuptools_scm]
# Make sure setuptools uses version based on the last created tag
version_scheme = "post-release"
# Make sure scm doesn't use local scheme version for push to pypi
# (so there isn't a + in the version)
local_scheme = "no-local-version"
# This _version_generated.py file is a file that you'll never want to add to version control - so you'll want to add it to your gitignore file. but when you build your package, setuptools_Scm creates it. and it contains version information that you will pull into your package below. 
write_to = "stravalib/_version_generated.py"
write_to_template = '__version__ = "v{version}"'

then in the project section of your pyproject, you set the version to be dynamic - see here for an example

[project]
name = "stravalib"
description = "A Python package that makes it easy to access and download data from the Strava V3 REST API."
keywords = ["strava", "running", "cycling", "athletes"]
readme = "README.md"
dynamic = ["version"]

Step two - update your init file to import the version value from the file setuptools_scm creates

To make this work at the package level so you can call __version__ on your package, you can add a small conditional to your init.py like so (Example here):

try:
    from ._version_generated import __version__
except ImportError:
    __version__ = "unreleased"

What this does is in the built version of your package, it will import that version_generated file and access the version that way. but if your working locally it will assign version as unreleased. There are a few ways to do this but i’ve found this to work.

you should now be able to build your package and it will recognize the version in that build. when you make a release to pypi via that action, the new tag number will become the version that pypi uses because of that dynamic setting in your pyproject.toml file.

also i see your docs build works now?? :rocket: i was just going to play with that a bit… but it seems fine! i promise last post from me until you respond again :slight_smile:

copying my reply from Slack :slightly_smiling_face:@AlexanderJuestel I agree with @lwasser that you probably want to remove this metadata and have your pyproject.toml be the single source of truth. And I probably want to do this in my packages too!

Some more context from our discussion in Slack:

I am guilty of doing this.

I actually have no idea what tool would read / use this information from the init file.

I think it’s done so people have that info available from in the package itself, most importantly the __ version __ Like, I want to be able to do this in a REPL

>>> mpykg.__version__ '0.1.0'

And then the fact that some tools use it is kind of a happy accident, and/or terrible idea that could allow metadata to be out-of-sync as you point out
I cargo culted this from a PyPA tutorial, see #3 here where they point to warehouse as an example of a package that does this: Single-sourcing the package version - Python Packaging User Guide

so, yeah, it’s probably a case of “we use to read this in setup.py but we shouldn’t do that anymore and just use static metadata”

1 Like

Hey @lwasser thank you for your extensive replies. I think I included all your comments in my files now :slight_smile:

But it may be good to have a video chat or something to go through everything maybe?

Cheers
Alex

1 Like

I think I included all your comments in my files now :slight_smile:

woo hoo!!

@AlexanderJuestel that sounds good!! A few options, you could join our slack if you wish. We can also do a video chat! can you email me at leah at pyopensci.org and we can find a time to chat and go through everything? or you can dm me in the software underground slack if you prefer that route!

Leah