Sablefin Advisory runs a Gemini-based assistant over a 300-page policy handbook, resending the whole handbook as context on every request, and the monthly bill is dominated by input tokens. Two workloads exist: an interactive chat used by 200 advisers during office hours, and an overnight job producing 60,000 compliance summaries with no deadline before 07:00. Finance wants a large reduction in spend without degrading interactive answer quality. What should you do?
- A.
Put the handbook into Gemini Enterprise Agent Platform context caching for the interactive chat, and move the 60,000 overnight summaries to batch prediction.
- B.
Reduce the maximum output token limit on both workloads so that every response is shorter and fewer output tokens are billed each month.
- C.
Buy Provisioned Throughput for the interactive chat so that input tokens are no longer billed at all.
- D.
Increase the temperature on both workloads so the model spends fewer sampling steps per response.
Show answer
Answer: A
Context caching removes the cost of resending the same handbook on every interactive call, and batch prediction serves the deadline-free overnight job at a lower price than online requests.
- A. Caching bills the repeated handbook prefix at the lower cached-input rate without changing what the model sees, and batch prediction prices the deadline-free overnight generations below online requests.
- B. The bill is dominated by input tokens, so capping output length saves little, and shorter answers degrade the interactive quality Finance wants preserved.
- C. Provisioned Throughput is a capacity reservation aimed at availability; it changes how you are billed but does not make input tokens free.
- D. Temperature controls sampling randomness, not the number of tokens processed, so it changes output style rather than cost.