Some experiences of doing a pyOpenSci review

pyOpenSci Review Process Questions
1

Some feedback from having done a review of a pyOpenSci package.

  • It wasn’t clear whether comments should be inline with each review topic or just at the end. I found it easier to note comments inline, and have any additional comments at the end.
  • There’s a slight formatting issue when I copied the raw Markdown from the template to the editor that I used (Simplenote) and then back again, where the check boxes changed formatting. This might be how my editor handles Markdown.
  • It might make sense to put the JOSS requirements after the functionality requirements
  • There was some repetition about packaging guidelines and other categories

In terms of accessing the reviewing template and instructions, I didn’t have any particular issues. However I think it would be good if links to the reviewing instructions and templates were included in the GitHub comment (and thus the email notification) which asks the reviewer to do the review.

2 Likes
2

Also, in the Guide for Reviewers it would be great to include (or link to) guidance on how to set up Python virtual environments to make it easier to test Python packages.

2 Likes
3

thank you @npch !! this is great feedback. i’ll try to compile all of this feedback for discussion in our next meeting. In the meantime @jlpalomino and chris <i don’t think chris is in the forum yet as i can’t find his username> … any thoughts to add to Neil’s comments given you also just did a review??

4

I second Neil’s comments and do not have any additional comments about the review process from the reviewer’s perspective. The criteria/questions for the reviewer were straight-forward and clear in terms of what the reviewer was asked to consider. Regarding the raw Markdown formatting, Chris and I used HackMD and it copied well into the editor.

1 Like
5

I also agree with all that Neil said. My experience reviewing https://github.com/pyOpenSci/software-review/issues/7 was fine and kind of smooth, I believe the problems were more related to my lack of experience reviewing a third party software than the review process per se.

I found very useful to do the review first by opening issues upstream and then summarizing the review the in the issue in the PyOpenSci submission. That way all the actionable items I expect from the submitter are in the issues upstream, as they are closed I can just check the boxes and approve the package.

2 Likes
6

ok - time to address some review issues.

from the list of 4 items above:

  1. @npch we updated the review submission template in this pr

Pull Request

to include a link to the review template at the bottom. However if what i am hearing is correct, you might also like it in the editor template which is where i pinged you (you were pinged twice but that is where the editor formally pings you.so that is here: https://github.com/pyOpenSci/dev_guide/blob/master/content/appendices/templates.md should we update this template as well?

  1. Hmmmm i’m open to suggestions here. did anyone else run into this format issue with their text editors? @choldgraf or @ocefpaf ??
  2. I just submitted a pr for this Move JOSS requirements does this look good to you @npch ?
  3. I am not super clear about what was redundant but we should definitely eliminate overlap where we can! any specific suggestions? Please feel free to submit a PR to address any of the above items as well @npch thank you for this feedback.
7

This is great feedback. And it’s something i struggled with when i first learned about testing and python environments! Here is a question for the group that could be a separate question.

Here is an issue to address this: Add tutorial on how to setup a Python testing envt · Issue #27 · pyOpenSci/software-peer-review · GitHub
i’ve also added it to an informal todo list in our meeting notes for next time!!

Welcome any feedback on this!! thank you again, Neil!

1 Like
8

Wow this is great @ocefpaf.

  1. thank you for the super speedy review!! and
  2. for being willing to pilot this new approach of adding upstream issues.

@marskar did this approach work for you? any feedback?
see htis issue where we initially discussed this approach

i think we’d like to eventually update our documentation to address how to

  1. create these issues and
  2. link back to them.

Filipe did this nicely but he is an github expert!! i know that most people i work with who have less experience may not be familiar with issue tracking / linking to PR’s etc!!! Thank you both for working through this with us all!!

1 Like
9

I definitely agree that @ocefpaf did an awesome job with the review. Thanks Filipe!

I think Filipe’s approach of separating everything out:

  • low-level: individual issues in the package repo
  • high-level: a summary in the pyOpenSci issue

is a great way to set up a review.

This way, everyone can have a high-level overview of the review in the pyOpenSci issue without having to look at each individual issue in the package repo.

I will get started on resolving the issues tomorrow. As I work through each issue in the package repo, I will report progress in the pyOpenSci issue.

1 Like
10

Regarding (2) I did not had any formatting issues but I did not use an external text editor.

About (4) I’m not sure if these are the redundancies that @npch saw, but here are a few:

  • Documentation/ installation instructions: for the development version of package and any non-standard dependencies in README
  • Functionality/ Installation: Installation succeeds as documented.

I believe those those could be merged into one.

  • Documentation/Vignette(s) demonstrating major functionality that runs successfully locally
  • Examples for all user-facing functions

I have to say that those are confusing to me b/c examples of user-facing functions can be a vignette (not the other way around though b/c of the literary description required to be a vignette)/

1 Like