generation usage should accept a list of dict for async generation

[dove-young]

So far usage in generation is a dictionary. There is a problem with it.

Some LLM provide async feature in their generation function. It accept a list of prompts, and process them in a batch. Then it gives a list of generated responses.

In this case there are a list of prompts has been processed, and a list of generation and token usage count are returned. And there is a single generate/completion call. So there would be a single generation object in the trace.

But now it is not possible to record these list of usage data into a usage record in the generation object

[maxdeichmann]

Hi @dove-young thanks a lot for the pointer. This is indeed a bit of a mismatch. Could you create a generation for each of the prompts in the list?
Which models are you talking about that take a list of prompts?