HF Hub LLM Connector

Options

Hugging Face Hub Settings

Repo ID

The model name to be used, in the format <organization_name>/<model_name>. For example, mistralai/Mistral-7B-Instruct-v0.3 for text generation, or sentence-transformers/all-MiniLM-L6-v2 for embedding model.

You can find available models at the Hugging Face Models repository.

Model Parameters

Top k

The number of top-k tokens to consider when generating text.

Typical p

The typical probability threshold for generating text.

Repetition penalty

The repetition penalty to use when generating text.

Max new tokens

The maximum number of tokens to generate in the completion.

The token count of your prompt plus max new tokens cannot exceed the model's context length.

Number of concurrent requests

Maximum number of concurrent requests to LLMs that can be made, whether through API calls or to an inference server. Exceeding this limit may result in temporary restrictions on your access.

It is important to plan your usage according to the model provider's rate limits, and keep in mind that both software and hardware constraints can impact performance.

For OpenAI, please refer to the Limits page for the rate limits available to you.

Temperature

Sampling temperature to use, between 0.0 and 100.0. Higher values will make the output more random, while lower values will make it more focused and deterministic.

Top-p sampling

An alternative to sampling with temperature, where the model considers the results of the tokens (words) with top_p probability mass. Hence, 0.1 means only the tokens comprising the top 10% probability mass are considered.

Views

This node has no views

Workflows

• No workflows found

Links

• No links available

Developers