TIL there's a batch API.. This seems like something a lot of AFK coders should be using.
The pattern for those users is typically they would set some kind of token budget, but their agent would still try to burn through those tokens as quickly as possible, rather than a more sensible "do this at your own leisure over the next ~8 hours".
Looking forward to further commodification of LLM usage in the future to make it more affordable. Batch APIs and more freedom over scheduling/priorities/deadlines seems like the more sustainable approach to driving costs down.
TIL there's a batch API.. This seems like something a lot of AFK coders should be using.
The pattern for those users is typically they would set some kind of token budget, but their agent would still try to burn through those tokens as quickly as possible, rather than a more sensible "do this at your own leisure over the next ~8 hours".
Looking forward to further commodification of LLM usage in the future to make it more affordable. Batch APIs and more freedom over scheduling/priorities/deadlines seems like the more sustainable approach to driving costs down.