Trouble GitHub action for publishing to PyPI

[jsdodge]

Hello,

I have set up a GitHub Action for publishing to TestPyPI and the official PyPI on my project, following the instructions here. Everything works fine except the step that publishes the distribution to PyPI (TestPyPI works fine), which is not triggered as I expected. I thought it would run when I complete a pull request of a tagged commit, but I’ve done several of these now and the action never runs. Could anyone give me a hand with this?

Thanks in advance!

Steve

[lwasser]

hey @jsdodge welcome to pyOpenSci!

a few questions

1. can you please clarify what your process is here:

I thought it would run when I complete a pull request of a tagged commit,

are you creating a tag locally and pushing it to github? OR are you creating a tag in your browser? i haven’t see a tag created via a pull request before so let’s start there!

this is an example of what a push to pypi on a (tagged & published) release looks like.

the key pieces to focus on in that example are the triggers including release at the top

on:
  release:
    types: [published]. # tell the action to push to real pypi when you make a release
  push:
    branches:
      - main # the branch where you want to push to test pypi

and then specifying the event type to push to real pypi vs test.

      - name: Publish package to PyPI
        # Only publish to real PyPI on release
        if: github.event_name == 'release'
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_TOKEN }}

in this specific workflow you’d perform the following steps to push to pypi

1. create a new release with an associated tag.
2. publish it
3. it will trigger the action and push to pypi

there are no pull requests involved.
i haven’t setup a build with just tags but it’s possible someone else here has if you are trying to avoid releases.

[jsdodge]

Hi @lwasser, thanks for your reply. To answer your first question, I am creating a tag on the dev branch of my local git repository, then pushing it to GitHub, then creating a pull request to merge it with the main branch. The instructions say, " These two steps use the pypa/gh-action-pypi-publish GitHub Action: the first one uploads contents of the dist/ folder into TestPyPI unconditionally and the second does that to PyPI, but only if the current commit is tagged." Have I misunderstood these instructions?

Regardless of how this went wrong, it sounds like the workflow in your example is more straightforward, so I may just copy it over to my repo and delete the other one. I just assumed that I should follow the PyPA recommendations. Is there a documented template for that workflow, so that I can easily customize it for my repo? I don’t understand some of the underlying logic.

[lwasser]

Ok i can break this down for you more later today (we are actually working on tutorials to explain this type of thing so doing this is inline with what i’m working on now already)

i also suggest that you do NOT copy my build action. let’s fix yours instead!! if you open a pr i can work with you but let’s break this down here first.
Lets look at the tutorial first

breakdown - when the entire action is triggered

at the top of your action is tells github to only trigger the action when there is a PUSH to the main branch. This means that your action will only be triggered when that happens.

on:
  push:
    branches: [ 'main' ]

in the pypa tutorial is has this

name: Publish Python 🐍 distributions 📦 to PyPI and TestPyPI

on: push

this means that the action will be triggered in all pushes (with some caveats if you have logic below which you do in your action)

but this is the first different and it may be why your build isn’t working.

Breakdown - what is running and when using conditionals

let’s break down the second part

pypa has this:

 - name: Publish distribution 📦 to Test PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
        repository-url: https://test.pypi.org/legacy/
    - name: Publish distribution 📦 to PyPI
      if: startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

and you have this

    - name: Publish distribution 📦 to Test PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        password: ${{ secrets.TEST_PYPI }}
        repository-url: https://test.pypi.org/legacy/
    - name: Publish distribution 📦 to PyPI
      if: startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        password: ${{ secrets.PYPI }}

in your chunk of code in the action - there is a condition here

  if: startsWith(github.ref, 'refs/tags')

this tells github to ONLY run that part of the action which is pushing to pypi (rather than test pypi) IF the push “item” starts with a github reference to tags (github.ref, refs/tags.) WHile a tag or release can be associated with a branch ( see image below when i create a release)

it does not entail a pull request. and again notice at the top of your action, you are telling github to only trigger that action on a PUSH to the MAIN branch (no other branches will be triggered).

---

OK so i hope that is clear! BUT i understand if it isn’t please ask more questions!

my action is a bit different but similar in how it behaves. let’s break down the pieces that would be relevant to you here.

on:
  release:
    types: [published]
  push:
    branches:
      - master

here i’m telling GitHub to trigger this action when one of two conditions is true:

1. on a release that is published OR
2. on a push to master branch (i know we need to change our branch to main)…

you can ignore the middle part of my action for now - i’m using a different build tool than you are - you are using hatch and hatchling (i’m curious how you think it is to use!). I am using setuptools and setuptools scm for versioning. so i’d stick with what you have in the middle of your action.

Below i have some logic that tells the action to ONLY push to test pypi when the event is a push event. Because i have the action trigger on push to master only at the TOP it will then push to test pypi when ever we push to the master (in your case main) branch.

      - name: Publish package on test PyPI on merge
        # THIS LOGIC IS A TRIGGER TO ONLY RUN THE PUBLISH PACKAGE ON TEST PYPI SECTION OF THE BUILD ON A PUSH TO MASTER (WITH THE BRANCH NAME BEING SPECIFIED AT THE TOP OF THE ACTION). 
        if: github.event_name == 'push'
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.TEST_PYPI_TOKEN }}
          repository_url: https://test.pypi.org/legacy/
          # THIS JUST TELLS IT TO NOT TRY TO OVERWRITE AN EXISTING TAGGED RELEASE ON TEST PYPI 
          skip_existing: true

      - name: Publish package to PyPI
        # THIS TELLS GITHUB TO ONLY RUN THIS SECTION OF THE ACTION WHEN THERE IS A RELEASE (A RELEASE AND TAG ARE NOT A PULL REQUEST BUT THEY CAN BE ASSOCIATED WITH A BRANCH AND HAVE A COMMIT REFERENCE THAT IS A TAG NUMBER)
        if: github.event_name == 'release'
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_TOKEN }}

Summary!

this combination of logic just allows you to control

1. when the action is triggered to begin with - in MY example it’s triggered on a release or on a push to master. You want the same but you want to change master to main (which is better for many reasons!).
2. You can add conditionls to sections below in your action that tell github to only run that part of the action given particular situations. In this case you want to tell it to push to test pypi when you push to main and you want it to push to the real pypi when you create a new release.

Follow up

Please let me know if this generally makes sense to you. you can actually do the same thing using several different approaches.

Next steps -

why don’t you submit a pr to your repo with a revised action and i’ll have a look.

[jsdodge]

Thanks, @lwasser ! I’ve made some changes and created a PR here. We can continue our conversation there. I created a test release and it does seem to have triggered the action—the action failed because I have already published this version to PyPI, but the important thing is that the release triggered the action. I still have a couple of questions about how it works, though, so I’d appreciate reviewing it with you on the PR.

Regarding hatch and hatchling, once I figured out how to use hatch I like it a lot—it simplifies my development by putting multiple tools under the same umbrella, saving me the trouble of remembering/looking up the appropriate tool-specific commands. The documentation is pretty spare, but I was able to set it up with a bit of effort. As for hatchling, all I can say is that it works and I’m fine with it. As with the GitHub workflow that we are discussing, I just did what PyPA suggested.

[lwasser]

Done!
thanks for the input on tools. i only ask because when we create tutorials i’m looking at various tools and plan to pick one. i was curious what your experience with hatch was so thank you for sharing it.

i think in reality most of us don’t really notice the difference in build back ends if they do their job and create the wheel and sdist! but also i know a lot of people like hatchling a lot. i hope this pr when merged gets you where you need to be with your dev work!

[jsdodge]

Hi @lwasser, I’m just coming back to let you know that the revised workflow works as expected:

1. I pushed a new tagged commit, which included an incremented version, to my dev branch, which did not trigger the workflow; then
2. I created a pull request to merge this version into the main branch, which built the distribution and published it to TestPyPI; then
3. I created a new release from the tag, which published the distribution to PyPI.

Thanks for all of your help!

[lwasser]

YAY!! so great to hear, @jsdodge !! very glad I could help out with this.
i took notes as well for when we create tutorials on this topic!!