Slow generation speed

Slow generation speed makes the product feel heavier than users expect. Even if the answer is accurate, long waiting times create the impression that the system is struggling.

Generation speed is often affected by model size, token output length, prompt size, and the amount of extra context fed into the request. Many of these factors can be improved without changing the core product idea.

Users usually care more about responsiveness than perfection. Faster generation often improves satisfaction more than one extra round of polishing the answer.