Figshare package

NickleDave · September 27, 2019, 5:07pm

Hi all,

I’ve been working on a command-line tool for working with Figshare

and I’ve kind of hit a brick wall; I tweeted about it (https://twitter.com/nicholdav/status/1177593328780677126) and @lwasser suggested I bring the discussion here.

So, any all feedback welcome: do you think this would be useful, does it fall under the PyOpenSci umbrella, can meet the requirements for review?

To start with, there is an RFigshare tool that’s part of ROpenSci:

https://docs.ropensci.org/rfigshare/
and it (admirably and as expected since it’s part of ROpenSci) meets all the review criteria, tests, docs, CI, etc.

The goal for any figshare command-line tool, I think, should be to make interfacing with Figshare transparent to the user as much as possible.

The ROpenSci package achieves this, basically by pretending to be a web app, i.e. like when you sign in to Discourse using your GitHub credentials. In other words, to interface with the Figshare API, the RFigshare devs pretend that they are a web app, and use the OAuth web app workflow to get authorization from the user to work with their data.

I’ll come back to this below, but let me say there’s a couple existing Python packages that I found very helpful in getting me started

I think neither is actively maintained, based on this discussion

and neither has tests/non-docstring docs/continuous integration

I also remembered upon searching PyPI again (Search results · PyPI) that there’s also this package:
GitHub - makkus/pigshare: Python cli-client and wrapper library for v2 of the Figshare API
which looks like it might actually solve some of the issues that I’m having … but the README says it’s only been tested with institutional figshare and mostly for downloading data. (I want to upload data for sure, and not require the user to have an institutional account.)

And that brings me to issues I’m having:
Since these are command-line tools, not full-blown web apps, they require some hacky tricks to work with the OAuth web app workflow:

pop open a browser, and get the user to authorize the command-line tool
in OAuth 2.0, somehow get the authorization token returned by Figshare from a redirect url

This is what (the now deprecated) OAuth2Client does:

github.com

googleapis/oauth2client/blob/master/oauth2client/tools.py

# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Command-line tools for authenticating via OAuth 2.0

Do the OAuth 2.0 Web Server dance for a command line application. Stores the
generated credentials in a common file that is used by other example apps in
the same directory.
"""

This file has been truncated. show original

and people have used this approach in scripts

gist.github.com

https://gist.github.com/burnash/6771295

get_oauth2_token.py

'''
    This script will attempt to open your webbrowser,
    perform OAuth 2 authentication and print your access token.

    It depends on two libraries: oauth2client and gflags.

    To install dependencies from PyPI:

    $ pip install python-gflags oauth2client

This file has been truncated. show original

This hacky approach to having a command-line tool pass as a web app works because you can specify the redirect url as ‘http://localhost/8080’

The problem is that Figshare requires the redirect url to be https not http.

Notice this isn’t a problem for the figshare packagr from rmcgibbo and RFigshare, because they still use the OAuth 1.0 workflow, which I thought Figshare maintains for backwards compatability.

github.com

rmcgibbo/figshare/blob/master/figshare/oauth_dance.py

# This code was adapted from Python Twitter Tools (MIT license)
# https://github.com/sixohsix/twitter/blob/master/twitter/oauth_dance.py
# Authors: Mike Verdone, Robert T. McGibbon

from __future__ import print_function
import os
import time
import webbrowser
from requests_oauthlib import OAuth1Session

try:
    _input = raw_input
except NameError:
    _input = input


def write_token_file(filename, oauth_token, oauth_token_secret):
    """
    Write a token file to hold the oauth token and oauth token secret.
    """

This file has been truncated. show original

github.com

ropensci-archive/rfigshare/blob/master/R/fs_auth.R


#' Figshare authentication via OAuth1.0 using httr
#'
#' Figshare authentication via OAuth1.0 using httr
#' @param cKey optional argument for the consumer key.  See details.  
#' @param cSecret optional argument for the consumer secret key. See details. 
#' @param token optional argument for the user-specific token.  See details.  
#' @param token_secret Optional argument to provide a secret token assigned
#' to the user, rather than let fs_auth() automatically handle authentication. 
#' See details.  
#' @details Explicit calls to fs_auth() are usually not needed,
#' as the function is called automatically by all other functions that
#' need authentication.  As of version 0.3, no arguments are needed as
#' authentication is done online, and fs_auth() will not attempt to load
#' keys stored in options.  
#' 
#' By default, the function will use the application's consumer key and
#' consumer secret key, rather than expecting the user to create their own
#' application.  The user-specific tokens will then be generated and locally
#' cached for use between sessions, if indicated by the interactive options.

This file has been truncated. show original

I thought it would make sense to use an OAuth 2.0 workflow (not 1.0) since that is what the most current Figshare API uses, and it should be more convenient / secure, but it turns out to not be obvious how to get this to work for a command-line tool, esp. for someone like me that knows little about networking.

My (currently failing) approach is here:

github.com

NickleDave/pyfigshare/blob/master/src/pyfigshare/auth.py

"""Figshare authentication

based on ideas from:
https://github.com/ropensci/rfigshare/blob/master/R/fs_auth.R
https://github.com/NickleDave/figshare/blob/master/figshare/oauth_dance.py
https://developer.okta.com/blog/2018/07/16/oauth-2-command-line

some functions/classes modified from
https://github.com/googleapis/oauth2client/blob/master/oauth2client/tools.py
https://github.com/googleapis/oauth2client/blob/master/oauth2client/_helpers.py
under Apache license
https://github.com/googleapis/oauth2client/blob/master/LICENSE
"""
from configparser import ConfigParser
import http.server
import os
from pathlib import Path
import ssl
import sys
import urllib.parse

This file has been truncated. show original

Something is going wrong, I think, because the redirect url is https and not http, which breaks the hacky ways of getting tokens out of the actual redirect that Figshare performs. I can provide more detail about the errors I’m getting if that would clarify things.

An alternative would be for users to get their own tokens and work with the tool that way.
FigShare has added this ability for users to generate tokens without declaring an application.
The RFigshare package allows for this option–see in that fs_auth.R code I linked above.
I think it’s good to provide this option, but I think only having this option would limit the number of users, because it adds friction–requiring users to know what tokens are, how to get them, how to provide them to the tool–so I would prefer something like the web app authorization workflow.

Again, any feedback welcome.
Sorry for the overly detailed bits about what’s not working, but as Leah suggested I’m hoping to bring other (more experienced) devs into the conversation and get some feedback on that too.

–David

Topic		Replies	Views
Libraries to automate building datasets? like make + snakemake Questions about Python Packaging & Code	4	380	July 23, 2019
What about projects that are not hosted on GitHub or cannot be transferred to the PyOpenSci organization Meta	3	484	June 7, 2019
UPDATE: what pyopensci is working on pyOpenSci Community Chat	0	323	November 29, 2022
We are going for funding! pyOpenSci Community Chat	4	453	February 9, 2021
Request to Join Python Packaging Mentorship/Training pyOpenSci Community Chat	5	60	May 23, 2024

Related topics