Transformers

Hugging Face Transformers crawling on WSL2 with CUDA Ran a simple AutoModelForCausalLM generate on a T4 but got one token every five seconds. from transformers import AutoTokenizer, AutoModelForCausalLM import torch, time model = AutoModelForCausalLM.from_pretrained("gpt2") tokenizer = AutoTokenizer.from_pretrained("gpt2") t0 = time.time() model.generate(**tokenizer("hello", return_tensors="pt").to("cuda")) print(time.time() - t0) # ~5 seconds for a single token GPU showed zero utilisation.