inference

The process of running a trained model to generate output for new inputs. Generative AI inference is autoregressive — generating one token at a time.

Syntax

generative-ai

output = model.generate(input_ids, max_new_tokens=200)

Example

generative-ai

# Inference with transformers:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

inputs = tokenizer("The future of AI is", return_tensors="pt")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=50, temperature=0.7)
print(tokenizer.decode(output[0]))