Contact us if you'd like to request a custom solution or instance type.

Amazon Web Services

Microsoft Azure

Google Cloud Platform

CPU GPU INF2

East US us-east

Intel Sapphire Rapids

1 vCPU · 2 GB

$ 0.033/h

Intel Sapphire Rapids

2 vCPUs · 4 GB

$ 0.067/h

Intel Sapphire Rapids

4 vCPUs · 8 GB

$ 0.134/h

Intel Sapphire Rapids

8 vCPUs · 16 GB

$ 0.268/h

Intel Sapphire Rapids

16 vCPUs · 32 GB

$ 0.536/h

Protected Public HF Restricted AWS Private

The Endpoint is available from the Internet, and secured with TLS/SSL.
Only you can access it, using a Hugging Face Token generated from your personal account.

Automatic Scale-to-Zero

Endpoints scaled to 0 replicas are not billed. They may take some time to scale back up once they start receiving requests again.

Number of replicas

Automatically scale the number of replicas within Min and Max based on compute usage. Min is always 0 if Scale-To-Zero is active.

Min Max

More options

Autoscaling Strategy

Control what type of trigger will cause your Endpoint to scale up.

Hardware Usage Pending Requests

Hardware Utilization Threshold (%)

A scale up event will be triggered if the average hardware utilisation (%) exceeds this threshold for more than 20 seconds.

Container Type

The Default container is the easiest way to deploy endpoints, and is very flexible thanks to custom Inference Handlers. You can also select a container optimized for Text-Generation inference, or link your own Custom container.

Max Tokens (per Request)

Maximum number of tokens that can be sent in a request.

optional

Max Concurrent Requests

The maximum amount of concurrent requests for this deployment. Higher limit requires more memory allocation. For instance, if set to 4 requests with 1024 tokens maximum per request, the system needs memory capacity for 4096 tokens total.

optional

Embeddings pooling method

Pooling method for embeddings. Please refer to the model card to know which pooling method is used by the model.

Number of layers on GPU

The number of layers offloaded to the GPU. Layers not offloaded will be stored in the system RAM and processed by the CPU. If you leave this blank, all layers will be offloaded to the GPU. For CPU-only containers, this field is ignored.

optional

Multimodal Projector File

Select the multimodal projector (mmproj) file for multimodal models. This is required for models that support image input.

Not applicable for this model

Estimated total memory usage: 20GB

Default Env

Environment variables that will be provided to your container during deployment.

Key

Value

Secret Env

Same as Default, but people with access to this endpoint will not be able to read these values after creation.

Key

Value

Commit Revision

Specify a revision commit hash for the Hugging Face repository

optional

Task

Select a supported task with the right inputs and outputs for your model pipeline, or define a custom task.

Container Arguments

Arguments passed to the container entrypoint.

optional

Container Command

Command executed in the container.

optional

Download Pattern

Specify what file type(s) should be downloaded from the repository.

gte-Qwen2-7B-instruct-Q4_K_M-GGUF