Get streaming LLM completions with SSE

The language model to use for the completion. Call /.api/modelconfig/supported-models.json to get the list of available models.

The maximum number of tokens to generate before stopping. Note that our models may stop before reaching this maximum. This parameter only specifies the absolute maximum number of tokens to generate.

Amount of randomness injected into the response.

Defaults to 1.0. Ranges from 0.0 to 1.0. Use temperature closer to 0.0 for analytical / multiple choice, and closer to 1.0 for creative and generative tasks.

Note that even with temperature of 0.0, the results will not be fully deterministic.

Only sample from the top K options for each subsequent token.

Used to remove "long tail" low probability responses. Learn more technical details here: https://towardsdatascience.com/how-to-sample-from-language-models-682bceb97277

Recommended for advanced use cases only. You usually only need to use temperature.

Use nucleus sampling.

In nucleus sampling, we compute the cumulative distribution over all the options for each subsequent token in decreasing probability order and cut it off once it reaches a particular probability specified by top_p. You should either alter temperature or top_p, but not both.

Recommended for advanced use cases only. You usually only need to use temperature.

Whether to stream the response.

• When true, the response body is an SSE stream of event: completion\ndata: JSON where each JSON blob is a CompletionResponse.
• When false, the response body is a single JSON-encoded CompletionResponse.

Include the log probabilities on the logprobs most likely tokens, as well the chosen tokens. For example, if logprobs is 5, the API will return a list of the 5 most likely tokens. The API will always return the logprob of the sampled token, so there may be up to logprobs+1 elements in the response. The maximum value for logprobs is 5. For more details, see Fireworks API docs https://docs.fireworks.ai/api-reference/post-completions