Customer Support & Troubleshooting Guide
For support specialists: how to diagnose, resolve, and escalate user issues
1. Escalation Matrix
flowchart TD
Issue["User reports issue"] --> T1{"Tier 1:
Self-service / Quick fix?"}
T1 -->|Yes| Resolve1["Resolve: guide user
through steps"]
T1 -->|No| T2{"Tier 2:
Configuration / Permission issue?"}
T2 -->|Yes| Resolve2["Resolve: adjust settings,
permissions, or data"]
T2 -->|No| T3{"Tier 3:
System / Infrastructure issue?"}
T3 -->|Yes| Resolve3["Escalate to engineering
with diagnostics"]
T3 -->|No| Eng["Escalate to engineering
as bug report"]
| Tier | Handles | Examples | Resolves within |
|---|---|---|---|
| Tier 1 | User errors, UI guidance, basic questions | Duplicate name, unsaved changes, navigation, "how do I...?" | Immediately |
| Tier 2 | Config, permissions, data issues | Missing permissions, broken variable references, API key issues, computation errors | Same day |
| Tier 3 | Infrastructure, bugs, provider outages | S3 errors, CloudFront issues, database locks, LLM provider down | Escalate to engineering |
What to Include When Escalating
- User: email, company name, permission group
- What they tried: exact action (create campaign, run computation, etc.)
- What they saw: exact error message text
- Campaign ID (from URL:
/campaign/{campaignId}) - Timestamp of when the issue occurred
- Browser + OS (for frontend issues)
- Health check result:
/api/public/healthresponse - Version:
/api/public/versionresponse
2. System Health Checks
Grafana Dashboard: "16 - Customer Support"
A dedicated Grafana dashboard is available (grafana/dashboards/16-customer-support.json). It provides real-time views of everything below, organized into 6 rows. Start every troubleshooting session by opening this dashboard.
| Check | How | Grafana panel | Expected | If it fails |
|---|---|---|---|---|
| System up? | GET /api/public/health | "System Up?" stat | "UP" green | T3 System down |
| Active computations | Grafana | "Active Computations" | 0-5 normal | >10 or stuck >10min = investigate |
| Stuck tasks | Grafana | "Stuck Tasks (>5 min)" | 0 (green) | Non-zero = deadlock/timeout, T3 |
| Computation success | Grafana | "Computation Success Rate" | >95% green | <95% = check error panels |
| Login success | Grafana | "Login Success Rate" | >99% green | <90% = auth issue, T3 |
| LLM errors | Grafana | "LLM Provider Errors / min" | 0-2 normal | >20/min = provider outage |
| API key pool | Grafana | "API Key Pool Status" | Active > 0 per provider | All passive = AI down. Add keys. |
| Pool exhaustion | Grafana | "LLM Pool Exhaustion Events" | 0 | Non-zero = AI failing. T2 |
| Chat errors | Grafana | "Chat Errors" | 0 | Non-zero = chat broken |
| WebSocket | Browser DevTools -> WS tab | - | Active /secured/ws-integration | Refresh; check proxy |
| Version | GET /api/public/version | - | Expected version | Wrong version deployed |
Dashboard Rows Quick Guide
| Row | Shows | Check when |
|---|---|---|
| System Health | Up/down, computations, success rates, LLM errors | First thing on every ticket |
| User-Facing Errors | Auth failures/min, computation errors by type, LLM errors by reason | "Something is broken" |
| AI/LLM Health | Response latency P50/95/99, key pool, pool exhaustion, chat errors | "AI is slow" / "AI not working" |
| Performance | Campaign duration, chat response time, file upload latency | "Everything is slow" |
| User Activity | Campaigns/hr, computations/hr, chat msgs/hr, logins/hr | Baseline comparison |
| Deployment | Microsite errors, upload errors, doc gen errors, CDN latency | "Assets not publishing" |
| LLM Provider Detail (collapsed) | Per-provider errors, reasons, retries, key passivation, RPM throttling | Expand when LLM issues detected above |
3. Error Message Reference
When a user reports an error, find the exact message text below to understand the cause and resolution.
Campaign Errors
| Error Message | When | Resolution | Tier |
|---|---|---|---|
| "Campaign with name {name} already exist!" | Creating or renaming a campaign to a name already used in the same company | Choose a different name. Campaign names must be unique within each company. | T1 |
| "This campaign's name in the company exists. Change campaign name. The result will not be saved." | Updating a campaign with a duplicate name | Same as above - pick a unique name. | T1 |
| "Provided Campaign is not a published template or does not exist." | Setting an MCT ID that doesn't point to a valid published template | Verify the template exists and its type is "Published Template". | T2 |
| "Failed to rename campaign." / "Failed to delete campaign." | Campaign operation failed (often: locked by computation) | Wait for active computation to finish. Check if another user is working on it. | T1 |
| "Campaign not found!" | Campaign ID in URL doesn't match any existing campaign | Campaign may have been deleted. Return to home page and check campaign list. | T1 |
Computation Errors
| Error Message | When | Resolution | Tier |
|---|---|---|---|
| "There are some active computation elements in progress, please wait before execute" | Trying to start computation while one is already running | Wait for the current computation to finish. Monitor progress via the progress bar or WebSocket. | T1 |
| "Can't run new one campaign computation. Campaign locked by another process" | Another user or tab is already computing this campaign | Wait for the other process. If stuck for >10 minutes, ask engineering to check for hung locks. | T2 |
| "Prompt key '{key}' not defined." | A prompt variable references an environment variable that doesn't exist | Check environment variables: the referenced prompt template must exist. Create it or fix the reference. | T2 |
| "[key]: ReferenceError: [variable] is not defined" | JavaScript function references a variable that doesn't exist or hasn't been computed | Check variable names and computation order. The referenced variable may need to be computed first. | T2 |
| "Parent(s) not computed: [key list]" | A variable depends on others that haven't been computed | Run computation at a broader scope (step or phase level) to include dependencies. | T2 |
| "Broken template reference [templateId]" | A variable references a document template that was deleted | Reassign a valid document template to the step, or remove the asset variable. | T2 |
AI / LLM Provider Errors
| Error Category | What the user might see | Cause | Resolution | Tier |
|---|---|---|---|---|
| Rate limited (429) | "Too many requests" or computation stalls | Too many AI calls sent to the provider in a short time | Wait 1-2 minutes and retry. Long-term: add more API keys or increase RPM in model config. | T1 |
| Invalid API key (401/403) | "Invalid API key" or "Unauthorized" | API key is expired, revoked, or incorrectly formatted | Check API key in LLM API Keys settings. OpenAI keys must start with sk-. Verify in provider dashboard that the key is active. | T2 |
| Context too long | "context_length_exceeded" or "max_tokens" | Prompt + input data exceeds the model's token limit | Reduce prompt size, use fewer variables in the prompt template, or switch to a model with a larger context window. | T2 |
| Content filtered | "content_filter" or "moderation" error | AI refused to process due to content safety rules | Revise the prompt to remove sensitive/flagged content. | T2 |
| Model not found (404) | "model_not_found" | Model name in configuration doesn't match provider's available models | Check model name in AI Model Configuration. Model may have been deprecated by the provider. | T2 |
| Provider down (502/503) | "Bad Gateway" or timeouts | AI provider is experiencing an outage | Check provider status page (status.openai.com, etc.). Wait and retry. System auto-retries up to 3 times. | T3 |
| Network timeout (0) | Computation hangs, then errors | Network connectivity issue between our system and the AI provider | Check server network. Increase read timeout in model configuration. Escalate if persistent. | T3 |
File Storage & Asset Errors
| Error Message | When | Resolution | Tier |
|---|---|---|---|
| "File too large! Max allowed size is 100MB" | Uploading a file larger than 100 MB | Compress or split the file. The limit is 100 MB per file. | T1 |
| "Folder size more than 100 MB" | A storage folder has exceeded the 100 MB limit | Delete unused files from the folder, or move files to subfolders. | T1 |
| "File for an asset {title} does not exist in public storage!" | An asset's source file was deleted from storage | Re-upload the asset file, or regenerate the asset by rerunning computation. | T2 |
| "Private asset can not be published!" | Trying to deploy an asset that's in private storage to the microsite | Move the file to public company storage, then retry the deployment. | T2 |
| "Unable to load the required data" | Data fetch failed (storage, campaign, or step data) | Refresh the page. Check network connectivity. Verify user has permissions. | T1 |
Authentication & Permission Errors
| Error Message | When | Resolution | Tier |
|---|---|---|---|
| "Access Denied: You do not have the necessary roles to access this resource." | User doesn't have the required permission | Check which permission is needed (see Part 1: Permissions). Ask admin to add the permission to the user's group or add an Allow override. | T2 |
| "Access Denied: ... Required roles: [list]" | Same, but with specific role names shown | The message tells you exactly which permissions are needed. Request them from the admin. | T2 |
| "Username not found" | User account doesn't exist in the system | Create the user account, or check if the email/username is correct. | T2 |
| Page redirects to login | Session expired | Log in again. OAuth sessions last 24 hours; API key sessions last only 5 minutes. | T1 |
Validation Errors
| Error Message | Where | How to fix |
|---|---|---|
| "Required field!" | Any required form field left empty | Fill in the field before submitting |
| "The name must not contain slashes!" | Campaign name | Remove / characters from the name |
| "The subdomain must contain only lowercase letters, digits, and hyphens..." | Microsite subdomain | Use only a-z, 0-9, hyphens; don't start/end with hyphen |
| "Invalid url!" | URL fields | Enter a valid URL with protocol (https://...) |
| "The email is not valid!" | Email fields | Enter a valid email format |
| "Site Context must not start or end with a slash." | Microsite site context | Remove leading/trailing slashes |
| "Allowed characters: A-Z, a-z, numbers, or -/_+=.@" | Various fields | Remove special characters not in the allowed list |
| "Invalid api key" (must start with sk-) | OpenAI API key | Use the correct key format from OpenAI dashboard |
| "You have unsaved changes. Please click Apply or Cancel." | Form fields with Apply/Cancel buttons | Click "Apply" to save or "Cancel" to discard before navigating |
Customer App Specific Errors
| Error Message | When | Resolution | Tier |
|---|---|---|---|
| "No GTM Foundation available. Contact support..." | Step 1 of campaign wizard, no templates exist | Admin must create and publish a GTM Foundation (campaign template) first. | T2 |
| "Failed to start AI Assist. Please try again." | Clicking AI Assist button fails | Check LLM configuration. Ensure at least one chat-enabled AI model exists. Retry. | T2 |
| "Asset with ID {id} not found." | Referencing a deleted asset in the assets table | Reload the step data. The asset may have been deleted by another user. | T1 |
| "An error has occurred" (full-screen modal) | Computation failed during campaign activation | Check computation status for specific error. See AI/computation error tables above. | T2 |
| "Something went wrong" | Generic fallback error | Ask user to check browser console (F12 -> Console) for details. Refresh and retry. | T2 |
4. Troubleshooting Scenarios
"My computation is stuck / not finishing"
Tier 2
- Check computation status: is it showing progress (items completing) or completely frozen?
- If progress is moving but slow: likely rate limiting. The system is waiting for API tokens. Check RPM setting on the AI model - increase it or add more API keys.
- If completely frozen for >5 minutes: check if it's a lock issue. Another tab or user may have started a competing computation.
- Try cancelling the computation (Stop button) and restarting.
- If cancel doesn't work: the computation may be stuck in a deadlock. Escalate to engineering with campaign ID and timestamp.
- After a system restart, all in-progress items auto-reset to "created" - user can re-trigger computation.
"User can't see a campaign / feature / button"
Tier 2
- Check the user's company context: are they looking at the right company? Campaigns are isolated per company.
- Check the user's permission group: go to Users -> select user -> Company Security. Look for the missing permission.
- Check for DENY overrides: these block specific permissions even if the group allows them.
- For feature buttons (Image Generation, Audit, etc.): check if the feature toggle permission is enabled (Imaging Enabled, Audit Enabled, etc.).
- For hidden phases/steps: user needs the "Show Hidden" permission.
- In Customer App: campaigns only appear if they have an
applicationState. Template-only campaigns are hidden. - Quick fix: use "Login As" (admin feature) to see exactly what the user sees.
"AI is generating bad / wrong content"
Tier 2
- Check the prompt template (environment variable): is it well-written with clear instructions?
- Check the model: is the right AI model assigned? Different models have different strengths. GPT-4 is more capable than GPT-3.5 but slower.
- Check temperature: high temperature (>1.0) = more creative/random. Low temperature (<0.5) = more deterministic/focused.
- Check max tokens: if too low, output may be cut off mid-sentence.
- For structured output (
json_prompt): check the JSON schema - it may be too restrictive or have wrong field types. - Check if variables feeding into the prompt have correct values - bad input = bad output.
- User can override any AI result manually - suggest this as an immediate fix while prompt is being refined.
"Asset won't publish to microsite"
Tier 2 / Tier 3
- Check asset state: it must be "Publication Approved" before it can be published. Walk through the approval chain: Created -> Ready for Review -> Asset Approved -> Publication Approved -> Published.
- Check asset storage location: "Private asset can not be published!" means the file is in private storage. Move to public storage first.
- Check if the microsite exists and is in "Created" state. If it's "Initial" or "Error", the infrastructure isn't ready.
- Check if the asset's source file still exists in public storage. It may have been deleted.
- If all above is fine: Escalate - may be an S3/CloudFront issue.
"Customer App: Activate button doesn't work / computation fails"
Tier 2
- Check all 5 steps are properly filled: the system validates that each step has required fields before activation.
- Step 5 requires at least 1 external asset in the table.
- If computation starts but fails: check the error modal. It should show a specific error (AI, template, variable reference).
- Check WebSocket connection: if the progress bar doesn't move at all, the real-time connection may be broken. Ask user to refresh the page.
- Check AI model configuration: ensure at least one model is set as system default.
- If computation completes partially: the user can view the generated state and re-run failed items using field handlers.
"User logged out unexpectedly"
Tier 1
- OAuth sessions expire after 24 hours. The user simply needs to log in again.
- API key sessions expire after just 5 minutes of inactivity. If using automated tools, they must handle re-authentication.
- If an admin used "Logout User" on this account, all active sessions are terminated.
- If the user is logged out immediately after logging in: check their account status (is it disabled?) and company assignment (are they assigned to at least one company?).
"Changes not saving / data seems stale"
Tier 1 / Tier 2
- Check for "Apply" buttons: many fields in the system require clicking "Apply" to save. If the user navigates away without clicking Apply, changes are lost.
- Check for concurrent editing: if two users edit the same campaign, the last save wins. The system uses optimistic locking - if there's a conflict, the second user may see an error.
- Check browser cache: ask user to hard-refresh (Ctrl+Shift+R) to bypass cached data.
- Check WebSocket connection: if real-time updates aren't arriving, the user may see stale computation results.
"Microsite shows old content after updating assets"
Tier 2
- If the microsite uses HTTPS (CloudFront): there's a CDN cache. After deploying, changes may take a few minutes to propagate globally.
- The system auto-invalidates CloudFront cache when the origin path changes, but not on every asset deployment.
- Ask user to try hard-refreshing the microsite page (Ctrl+Shift+R) or try in an incognito window.
- If still stale after 15 minutes: Escalate for manual CloudFront cache invalidation.
5. Step-by-Step Guides
How to Create a Campaign (Admin Console)
- Navigate to Campaigns page
- Click "Create Campaign" button
- Enter a unique campaign name
- (Optional) Select a published template to start from
- Click Create - the campaign appears in "Not Started" status
- Add Phases (logical groups, e.g., "Research", "Content")
- Add Steps within each phase
- Add Variables (CC Items) within each step - choose types: string, prompt, function, etc.
- Set up Environment Variables - prompt templates, functions, shared config
- Assign Document Templates to steps that need rendered output
- Configure the default AI model for the campaign
- Click Compute to run the computation engine
- Review results, override if needed, compute again
- Generate and publish Assets
How to Fix a Failed Computation
- Identify which variables failed: look for red "error" state in the campaign grid
- Click on the failed variable to see the error message
- Common fixes:
- Missing prompt key: create the environment variable the prompt references
- Broken reference: fix the variable key name or path
- AI error: check API key, model config, and prompt size (see AI errors table)
- Script error: fix the JavaScript function code
- After fixing, either:
- Recompute the single failed variable (right-click -> Compute)
- Or reset the state to "created" and rerun the full computation
- If the error persists, check the dependency graph (upstream/downstream views) for circular or broken dependencies
How to Grant Permissions to a User
- Go to Users page -> find the user
- Click user -> Company Security tab
- Select the company
- Either:
- Assign a Permission Group (recommended): select a pre-built group like "Campaign Manager" or "Content Editor"
- Add individual overrides: add Allow overrides for specific permissions
- User must refresh their browser for new permissions to take effect (session caches permissions)
Warning: DENY overrides are permanent blockers
If a user has a DENY override for a permission, it blocks that permission regardless of their permission group. Always check for DENY overrides first when troubleshooting access issues.
How to Add / Replace an LLM API Key
- Go to Model Configs page (or LLM API Keys section)
- Click "Create"
- Select the provider (OpenAI, Anthropic, Google, etc.)
- Paste the API key (OpenAI keys start with
sk-) - The key enters the "In Pool" state and starts being used immediately
- If a key is rate-limited, it's auto-disabled temporarily and re-checked every 60 seconds
- To replace: disable the old key, add the new one
How to Guide a User Through the Customer App
- Home Page: shows campaign cards. Click "New Campaign" to start, or click an existing card to continue.
- Step 1 (Fundamentals): campaign name, GTM Foundation (template selection), desired outcomes, timeframe, funnel stage. All required.
- Step 2 (Audience): select target personas (at least 1), choose segment, add any additional info.
- Step 3 (Strategy): fill in campaign objective, point of view, content themes, messaging pillars.
- Step 4 (Execution): describe content strategy, select content formats and distribution channels.
- Step 5 (Assets): add at least 1 external asset manually. Click "Activate Campaign" to trigger AI generation.
- After activation: a progress bar shows real-time computation. Wait for completion.
- Generated state: Steps 1-3 become read-only. Steps 4-5 remain editable with field handlers for incremental updates.
- AI Assist (if available for the step): click the AI icon to open an embedded chat for help refining content.
6. Known Limitations
| Limitation | Value | What happens when exceeded |
|---|---|---|
| Single file upload size | 100 MB | "File too large!" error, upload rejected |
| Folder size in storage | 100 MB | "Folder size more than 100 MB" error, new uploads blocked |
| Concurrent computations per campaign | 1 | "Campaign locked by another process" error |
| Total computation threads | 150 (system-wide) | New computations queue until threads free up |
| Threads per campaign | 30 max | AI calls within one campaign are limited to 30 parallel |
| OAuth session | 24 hours | User must re-login |
| API key session | 5 minutes inactivity | Session expires, must re-authenticate |
| AI retry attempts | 3 (default, configurable per model) | After 3 retries, variable goes to "error" state |
| Campaign name | Unique per company | Duplicate rejected with error |
| Phase name | Unique per campaign | Duplicate rejected |
| Step name | Unique per phase | Duplicate rejected |
| Variable key | Unique per step | Duplicate rejected |
Things the System Cannot Do
- No undo/rollback for computations - use audit trail to find previous values and manually override
- No real-time collaborative editing - two users editing the same campaign may overwrite each other's changes (last save wins)
- No automatic AI prompt optimization - prompt quality depends entirely on what the user writes
- No cross-company sharing - campaigns, templates, and storage are isolated per company. Use export/import to transfer.
- No partial rollback after "Start from Template" - once a campaign is created from a template, it's independent. Template changes don't propagate.
- No direct database access for users - all operations go through the REST API
- Cannot publish private storage files - must be in public storage for microsite deployment
7. Admin Console UI Reference
Quick reference of what each main screen shows and what the user can do there.
| Page | URL pattern | What it shows | Key actions |
|---|---|---|---|
| Campaigns | /campaigns | List of all campaigns for selected company | Create, edit, clone, delete, archive, publish, export |
| Campaign Editor | /campaigns/company/:id/campaign/:id | Campaign grid: phases -> steps -> variables | Add/edit/delete phases, steps, variables. Run computation. View results. |
| Variable Grid | (within campaign editor) | All CC items for a step in a table | Add, rename, reorder (drag), clone, override, execute, view graph |
| Document View | (within campaign editor) | Rendered document from template + data | Inline text editing, image replacement, link management |
| Chat | /campaigns/company/:id/chat/:chatId | AI chat for a step | Send messages, receive AI responses, apply overrides |
| Dependency Graph | /graph/:ids or /graph/downstream/:ids | Interactive upstream/downstream variable graph | Pan, zoom, click nodes to navigate |
| Storage | /campaigns/storage/* or /campaigns/company/:id/storage/* | File browser with folders | Upload, download, delete, copy, rename, create folders |
| Companies | /companies | List of all companies | Create, edit, manage users, configure microsite, audit config |
| Users | /users | List of all users | Create, edit, enable/disable, assign to companies, manage permissions |
| Security Groups | /security-groups | Permission group definitions | Create, edit, clone groups. Assign permissions. |
| Model Configs | /model-configs | AI model configurations | Create, edit, delete, set default |
| Document Templates | /templates/documents | Document template list | Create, edit, preview, HTML-to-XSLT conversion |
| Labels (i18n) | /campaigns/company/:id/campaign/:id/labels | Campaign/global labels | Add, edit, delete, bulk replace |
| Campaign Apps | /apps | Application configurations | View app details and storage |
| API Keys | /account/api-keys | User's personal API keys | View, manage |
8. Quick Reference Card
Keyboard Shortcuts
| Shortcut | Where | What it does |
|---|---|---|
| Cmd/Ctrl + Enter | Chat, JSON editor, inline editors | Save / Apply changes |
| ESC | Chat, editors, modals | Cancel editing / Close |
| Drag & Drop | Variable grid, phase list, step list | Reorder items |
Status Color Coding
| Color/Badge | Meaning |
|---|---|
| Gray | Not started / Created |
| Yellow | Waiting / In Progress |
| Purple | Processing |
| Green | Computed / Success |
| Red | Error / Failed |
Useful URLs
| URL | Purpose |
|---|---|
/api/public/health | System health check (returns "OK") |
/api/public/version | Build version and timestamp |
/api/secured/{campaignId}/computation/status | Computation progress for a campaign |