You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Some mismatches between localAI's and openAI's streaming messages:
when streaming, openAI does not send the role key with every data message, but instead only sends the role in an initial delta message that lacks any content, which all follow in lean data messages. localAI streams currently send the role key with every delta message. Aside from not being compatible, this is also inefficient over the wire.
when streaming, openAI terminates the stream not only with a ..., choices: [... finish_reason: stop] message, but also a separate message containing only "[DONE]". localAI streams currently lack this.
LocalAI version:
ed5df1e
Describe the bug
Some mismatches between localAI's and openAI's streaming messages:
role
key with every data message, but instead only sends therole
in an initial delta message that lacks anycontent
, which all follow in lean data messages. localAI streams currently send therole
key with every delta message. Aside from not being compatible, this is also inefficient over the wire...., choices: [... finish_reason: stop]
message, but also a separate message containing only "[DONE]". localAI streams currently lack this.These break integration with tools which trigger explicitly on these expected aspects of the openAI spec, such as org-ai. For 2., see https://platform.openai.com/docs/api-reference/chat/create#chat/create-stream, and for 1., see https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb (I couldn't find this in the spec itself, only the examples).
There are some other differences (lack of
id
&created
keys, and localAI superfluously sends consecutivedata
events as named events) that haven't lead to any practical issues in my testing.To Reproduce
An example response from localAI's streaming API:
Expected behavior
An example response from openAI's v1 streaming API:
The text was updated successfully, but these errors were encountered: