pagination does not show total size or number of pages, impossible to navigate paginated responses

[stevekm]

I am using commands such as

tw  runs view --id 123456 --workspace 567890 tasks

To get the list of tasks from pipeline runs, among other things. It seems that in most cases, only a single page of items are returned, up to 100 records (as per the --max arg #440 ). However, there is no indicator that the result returned is even paginated in the first place. If you did not know that pagination was in effect, then you would have no idea that there are more records available.

Its possible to add an extra arg such as --page 5, etc., to the command, and you get some extra text at the bottom like this

$ tw runs view --id 12345 --workspace 567899 tasks --page 5

  Pipeline's run 12345 tasks:

     task_id | process | tag | status
    ---------+---------+-----+--------
             |         |     |

  Page 5, showing from 400 to 499.

However, this is broken in a couple ways

• it does not tell you how many records exist, total
• it does not tell you how many records your query returned
• it does not tell you how many pages exist, total
• the number of records it shows is wrong; in this case, 0 records are returned, in fact, it seems that any time <100 records are returned, this is not reflected in the text

Further, this text only shows up in the console stdout in the text representation. If you run a command like this;

tw --output json runs view --id 12345 --workspace 567890 tasks --page 4

you get JSON formatted output, and there is no indication at all that the result has any pagination, no indication of the total number of records, etc..

For context, when querying a JSON API that returns a paginated response, the JSON response usually looks like this;

{
"prev": "https://.......?p=1",
"next": "https://......?p=3",
"count": 100,
"result": [ ...]
}

this kind of response indicates that pagination is in effect by listing the previous page endpoint URL, next page, number of items returned in the current result, and the items themselves. I am guessing that Nextflow Tower API probably gives a result like this directly.

The problem is that, given the fact that the --max arg does not work correctly as per #440 , it seems like its functionally not possible to return the complete, unpaginated set of all results from a query with tw, given that tw is not handling the pagination itself, and it is not exposing any method by which to programmatically navigate the pagination of its own response (or even detect that its response is paginated). The only way I can see to use tw for paginated responses is likely one of the following;

• manually run the command, interactively, guess that there might be more pages, manually re-run the command with --page 2, manaully count the number of records returned to see if <100 records were given, then continue to manually re-run the command with every value of --page until you get an empty result
• try to automate the above with some sort of pipe to grep or jq and a for loop?

Ultimately, the console output of tw needs to show the correct pagination state, even for JSON results. And it requires that there is some arg for tw by which to return the full set of results without having to manually and interactively run the tw command with an iterative set of --page args.