Good, Better, Best: required versus recommended changes by reviewers - your feedback is requested

Hi all, we have two packages in review for which similar discussions have arisen around “good, better, best” guidelines:

Should pyOpenSci aim to have all components be at least “good” and ask to reviewers to provide recommendations for better and best, but not let that limit approval?

More specifically, how should the review process distinguish between required and recommended changes, as identified by the reviewers?

As one example, the review template provides a requirement of Examples for all user-facing functions, and one reviewer noted that not all docstrings for user facing functions had examples. However, there were many examples in the provided tutorials.

As of right now, the submitting author has addressed this comment by adding examples (and links to the tutorials) to all user-facing functions, except those that could be considered to be self-explanatory (because they have either no parameters or have only simple parameters that are sufficiently described in the corresponding parameters section).

So in this example, is adding examples to ALL docstrings for user facing functions considered a required or recommended change?

We plan to discuss this further at our upcoming meeting on 2/28 11am MDT (meeting notes here) and also welcome comments and thoughts on this post.

Thanks and looking forward to hearing everyone’s thoughts!

Hi All!
We are exploring language like this to summarize the requirements for package documentation. We have found that for larger packages, it’s more difficult to enforce both doc string inputs and outputs AND example code for ALL user-facing functions. We do encourage it however. so we could have “good enough” be that all user facing functions are documented in the code / user facing (online) docs. And better being there are tutorials and vignettes in addition. We will would want an example of how to get started using the package. Should that always be in the readme? Please provide your input here.

ok (required): all user facing functions (inputs and outputs) are documented both in the code and in the user-facing documentation.
better: all user facing functions are documented with examples (in tutorials or doc strings)
best: all user facing functions are documented and there is are tutorials (vignettes) that demonstrate example user workflows

Thank you for your work on this!

Given the current draft requirements, I’m not sure what’s the key difference between better and best:

better: all user facing functions are documented with examples (in tutorials or doc strings)
best: all user facing functions are documented and there is are tutorials (vignettes) that demonstrate example user workflows

It sounds like

  • examples in doc strings => better
  • tutorial with example workflow => best

Is that what you had in mind?