Reduce OpenAI API Costs

Problem this prompt solves

OpenAI bills climb for boring reasons: every button press hits the largest model, prompts repeat the same system instructions, outputs are unbounded, and identical inputs are regenerated. Cost control is a product and engineering problem — not only a pricing-page problem — and it should land before public beta, not after the first invoice shock.

When to use it

• Your OpenAI invoice grew faster than active users.
• You defaulted to a flagship model for every task, including classification and short rewrites.
• You lack per-feature token accounting and cannot see which route spends money.
• You are preparing pricing tiers and need a lower cost-per-successful-action.

Audit this Next.js + OpenAI integration for cost leaks and implement high-impact reductions without destroying output quality.

Phase 1 — Measure (do this first in code):
- Ensure each AI call persists model, prompt_tokens, completion_tokens, estimated_cost (if pricing table available), route/feature name, user id, and latency
- Add a simple admin/dev-only summary or SQL view: cost and tokens by feature for the last 7 days
- Identify the top 3 cost drivers

Phase 2 — Reduce (implement the applicable items):
1) Model right-sizing: route cheap tasks to a smaller/cheaper model; reserve the expensive model for hard tasks (implement a clear mapping in code)
2) Prompt trimming: remove unused context, repeated boilerplate, and giant few-shot blocks; move static system prompts to concise versions
3) Output caps: set max_tokens (or equivalent) per feature based on real output needs
4) Caching: for deterministic or repeatable inputs, add an exact-match cache (Redis or Supabase table) keyed by hash(model + prompt/version + input). Respect user ownership and TTL.
5) Deduplicate: debounce client UI; ignore duplicate in-flight requests for the same user+input
6) Short-circuit: if validation can solve the task without a model, skip the API
7) Batching: where the product allows, combine multiple small extractions into one call instead of N calls

Phase 3 — Guardrails:
- Per-user daily token or request budget for MVP (env-configurable)
- Clear user-facing errors when budget exceeded
- Document remaining risks (e.g. cache staleness)

Constraints:
- Do not weaken auth/RLS
- Do not log full prompts to a public channel
- Prefer surgical refactors; do not rewrite the entire app
- Summarise expected savings qualitatively per change (e.g. “classification moved to mini — largest win”)

Deliver a short COST.md describing what changed, how to read the usage summary, and recommended next steps (model routing prompt, rate limits, pricing).

Expected result

Instrumented AI calls, concrete code changes (smaller models where safe, caps, cache/dedupe, budgets), and a COST.md that explains the top drivers and what was done — ready to re-check with the cost calculator.

Implementation notes

• Measure before micro-optimising temperature; feature-level token totals change priorities fast.
• Version your prompts in the cache key so prompt edits do not serve stale answers forever.
• Cheaper models need evaluation on a fixed sample set — keep 10–20 golden inputs to compare quality.
• Caching user-specific content must still enforce ownership on read.
• max_tokens too low creates retries (users click again) and can increase spend — tune with real outputs.
• Pair this work with rate limiting; cost caps and rate limits solve different abuse modes.

Testing checklist

• Run the same input twice — second call hits cache when intended and does not increment OpenAI usage.
• Confirm cheap-task routes use the smaller model in persisted ai_runs.
• Verify max_tokens (or caps) are applied on each feature’s request payload.
• Hit the daily user budget and see a controlled error without upstream calls.
• Compare token totals for a golden input before/after prompt trimming.
• Ensure failed runs still record token usage when the API returns partial usage data.
• Confirm no cross-user cache reads under a second account.

Common mistakes

• Switching everything to the cheapest model without quality checks — users churn and you “save” money on zero usage.
• Caching without ownership checks — data leak across tenants.
• Counting only requests, not tokens — one fat prompt dwarfs 50 tiny ones.
• Retry loops on timeouts without idempotency — triple charge on one click.
• Keeping huge conversation history in every call when the product is single-shot.
• Optimising marketing-page copy generation before the core loop that runs 100× more often.