AWS Launches llama.cpp Integration for Amazon SageMaker AI
TL;DR: AWS has released a new Deep Learning Container that enables deployment of decision models using llama.cpp on Amazon SageMaker AI. The solution supports OpenAI-compatible chat completions and includes a SemIf extension that returns probabilities over declared options for routing, classification, and agent action selection workflows.
What's New
AWS has released a specialized Deep Learning Container (DLC) for deploying decision models with llama.cpp on Amazon SageMaker AI. The release provides two deployment paths: a standard text-generation path that serves OpenAI-compatible chat completions, and a structured-decision path that uses SemIf (formerly OpenJev) to return probabilities over declared options without decoding answer tokens.
The standard deployment path packages models as model.tar.gz files in Amazon S3 and references the published DLC image directly. The container automatically finds GGUF files under /opt/ml/model and maps POST /invocations to llama-server at /v1/chat/completions. Configuration happens through SM_LLAMA_CPP_* environment variables.
The structured-decision path uses AWS CodeBuild to build a child image stored in Amazon Elastic Container Registry. This approach replaces the default server with a SageMaker-compatible adapter while preserving the DLC's compiled CPU or CUDA libraries. SemIf accepts requests with caller-defined IDs, unstructured state, questions, and ordered lists of allowed options, returning option IDs, probabilities, and raw logits along with metadata.
According to AWS benchmarks using 231 JevBench tasks, a Qwen3.5-4B Q4_K_M model on ml.g6e.xlarge GPU instances achieved 0.352 second median latency and 77.9% accuracy. The same model on ml.c6i.8xlarge CPU instances with 32 threads produced 1.715 second median latency and 77.5% accuracy, making GPU deployment 4.9 times faster at median and 32.2 times faster at 95th percentile.
Pricing and Availability
The AWS Deep Learning Container for llama.cpp is available now on Amazon SageMaker AI. Pricing follows standard SageMaker AI endpoint pricing based on instance type and usage duration. The release includes support for CPU instances (such as ml.c6i.8xlarge) and GPU instances through SageMaker AI instance pools, which list compatible GPU types including g6e, g6, g5, and g4dn in priority order.
How to Try It
Developers can deploy the standard text-generation path by packaging a GGUF model file as model.tar.gz in Amazon S3 and creating a SageMaker AI endpoint that references the AWS DLC for llama.cpp image. The container accepts OpenAI Chat Completions schema with messages arrays containing role and content fields, plus generation controls like max_tokens and stream parameters.
For structured decisions, developers build a child image using AWS CodeBuild that replaces the default server with the SemIf adapter. Requests specify a decision ID, state description, question, and options array with IDs and descriptions. Responses return option_ids, probabilities, and option_logits in the same order as the input.
AWS provides accompanying code samples with both CPU and GPU deployment options. The solution supports both single-request decisions and batched decisions that share the same state.
Who It's For
This release targets applications that need scores over fixed choice sets rather than free-form text generation. Example use cases include routing customer support cases to appropriate queues, selecting agent actions in agentic workflows, extracting values from known sets, assigning incident priorities, checking policy outcomes, and making classification decisions based on unstructured context.
AWS recommends keeping authorization and deterministic business rules outside the model layer and validating model probabilities before treating them as operational confidence scores. The solution is designed for developers who need managed inference infrastructure for quantized models on CPU or GPU.
Alternatives
AWS mentions several related technologies in the decision model space. TypeSafe's decision model provides a /v1/systemone API through its Jev implementation. Kev is a model server designed for TypeSafe API compatibility. Strands Decider 2B is a purpose-built model for agent workflows with calibrated outputs. For general-purpose model serving, developers might consider vLLM or other inference frameworks depending on their API compatibility and performance requirements.
FAQ
What is the difference between the two deployment paths?
The standard text-generation path serves OpenAI-compatible chat completions through the llama-server API. The structured-decision path uses a SemIf adapter that returns probabilities over declared options without generating answer tokens, designed for routing, classification, and decision workflows.
What performance can I expect from CPU versus GPU deployment?
In AWS benchmarks using Qwen3.5-4B Q4_K_M on JevBench tasks, GPU deployment on ml.g6e.xlarge achieved 0.352 second median latency versus 1.715 seconds on ml.c6i.8xlarge CPU with 32 threads. GPU was 4.9 times faster at median and 32.2 times faster at 95th percentile.
What input formats does the solution accept?
The standard path accepts OpenAI Chat Completions schema with messages arrays containing role and content fields. The SemIf path uses a constrained decision schema with an ID, state description, question, and an ordered list of options with IDs and descriptions.
When should I use the SemIf extension versus standard chat completions?
Use SemIf when your application defines allowed choices at request time and needs probabilities over those options for routing, classification, or agent action selection. Use standard chat completions when you need OpenAI-compatible text generation through InvokeEndpoint.