Contact us if you'd like to request a custom solution or instance type.

Amazon Web Services

Microsoft Azure

Google Cloud Platform

CPU GPU INF2

Intel Sapphire Rapids

2x vCPUs · 4 GB

$0.067 / h

available Hardware should be compatible with the selected model.

Private Public Authenticated

Only you can access your endpoint, using a Hugging Face Token generated from your personal account.

Automatic Scale-to-Zero

Endpoints scaled to 0 replicas are not billed. They may take some time to scale back up once they start receiving requests again.

Number of replicas

Automatically scale the number of replicas within Min and Max based on compute usage. Min is always 0 if Scale-To-Zero is active.

Min Max

More options

Autoscaling Strategy

Control what type of trigger will cause your Endpoint to scale up.

Hardware Usage Pending Requests

Hardware Utilization Threshold (%)

A scale up event will be triggered if the average hardware utilisation (%) exceeds this threshold for more than 20 seconds.

Inference Engine

Max Tokens (per Batch)

Number of tokens that can be added to a batch before forcing queries to wait in the internal queue. A value of 1000 can fit 10 queries of 100 tokens or a single query of 1000 tokens.

optional

Max Concurrent Requests

The maximum amount of concurrent requests for this deployment. Having a low limit will refuse clients requests instead of having them wait for too long and is usually good to handle backpressure correctly.

optional

Pooling

Override the model pooling configuration.

Container Arguments

Arguments passed to the container entrypoint.

optional

Container Command

Command executed in the container.

optional

Default Env

Environment variables that will be provided to your container during deployment.

Secret Env

Same as Default, but people with access to this endpoint will not be able to read these values after creation.

VPC Config

Check to activate and configure AWS PrivateLink

Use AWS PrivateLink

Commit Revision

Specify a revision commit hash for the Hugging Face repository

optional

Task

Select a supported task with the right inputs and outputs for your model pipeline, or define a custom task.

Download Pattern

Specify what file type(s) should be downloaded from the repository.

gte-large