OpenAI LLM Connector

Options

OpenAI Model Selection

Model selection

Whether all available models are listed or only selected compatible ones.

Available options:

• Default models: Shows default models for this model type.
• All models: Shows all models available for the provided API key. This includes models that may not be compatible with this specific endpoint, so it is the responsibility of the user to select a model that is compatible with this node.

Model ID

Select an OpenAI completions model to be used.

Specific model ID

Select from a list of all available OpenAI models. The model chosen has to be compatible with OpenAI's Completions API. This configuration will overwrite the default model configurations when set.

Model Parameters

Maximum response length (token)

The maximum number of tokens to generate.

This value, plus the token count of your prompt, cannot exceed the model's context length.

Temperature

Sampling temperature to use, between 0.0 and 2.0.

Higher values will lead to less deterministic answers.

Try 0.9 for more creative applications, and 0 for ones with a well-defined answer. It is generally recommended altering this, or Top-p, but not both.

Completions generation

How many chat completion choices to generate for each input message.

Note: This parameter can quickly consume your token quota.

Seed

Set the seed parameter to any integer of your choice to have (mostly) deterministic outputs. The default value of 0 means that no seed is specified.

If the seed and other model parameters are the same for each request, then responses will be mostly identical. There is a chance that responses will differ, due to the inherent non-determinism of OpenAI models.

Please note that this feature is in beta and only currently supported for gpt-4-1106-preview and gpt-3.5-turbo-1106 [1].

[1] OpenAI Cookbook

Number of concurrent requests

Maximum number of concurrent requests to LLMs that can be made, whether through API calls or to an inference server. Exceeding this limit may result in temporary restrictions on your access.

It is important to plan your usage according to the model provider's rate limits, and keep in mind that both software and hardware constraints can impact performance.

For OpenAI, please refer to the Limits page for the rate limits available to you.

Top-p sampling

An alternative to sampling with temperature, where the model considers the results of the tokens (words) with top_p probability mass. Hence, 0.1 means only the tokens comprising the top 10% probability mass are considered.

Views

This node has no views

Workflows

• No workflows found

Links

• No links available

Developers