Back to Blog Home

How to Build a Safe AI Onboarding Sandbox for New Hires

Illustration of a bounded practice area with training cards separated by a wall from a live production zone

A practice sandbox is safe only if production is unreachable by construction, not by instruction. We recommend four rules: isolate it with its own identities, endpoints and storage; start every attempt from a fresh copy of the scenario instead of building a reset; use the real tool's own sandbox where one exists and write a simulator only where none does; and include a stale-instruction case. Without that boundary, a training prompt can expose a customer record, create a real ticket, send a message or grant access before the employee understands the process.

How do you keep a practice sandbox away from production?

Reading a policy shows that someone can find it, not that they can do the work. A company brain makes the material easier to find, but retrieval alone gives the hire no place to rehearse a decision. AWS notes that retrieval augmented generation lets a model present information with source attribution, and that users can look up the source documents themselves. That is what you want from a practice assistant. It does not make a connected action safe: if the same assistant can query live records or call production tools, a realistic exercise can become a real event.

The failure usually starts with convenience: the training assistant gets pointed at the production index, reuses an integration token, and new hires are told not to submit anything. That instruction is a weak control, because a misunderstood prompt, an interface slip or a model error can still cross the boundary.

Write the sandbox contract first

Answer four questions before choosing tools: which tasks can the employee practice, which information can appear in a scenario, which actions are simulated, blocked or sent for review, and what evidence allows promotion to live access. Keep the first release narrow. Pick one recurring task with a clear source, visible decision points and a verifiable outcome. A support example might ask the employee to classify a fictional request, find the approved refund rule, draft a response and choose the escalation route, without sending the response or opening a production customer record.

For each exercise, record the owner, intended audience, source documents, allowed data classes, simulated tools, forbidden actions, completion test and access decision. If any field is unclear, the task is not ready. The NIST AI Risk Management Framework asks (MEASURE 2.6) that a system be shown to be safe and to fail safely, particularly when made to operate beyond its knowledge limits. The boundary and escalation cases are where a sandbox shows that.

Keep training knowledge apart from production knowledge

A prompt instruction to use only training documents is not a boundary. Create a distinct training collection and enforce the boundary before retrieval. Copy only the approved material the exercise needs, with its canonical source URL, version, effective date, owner and review date, and remove unrelated sections that expose confidential context. If a task depends on a policy exception, write a synthetic exception instead of copying a real employee case.

The company brain retrieves from this collection under a sandbox identity and cites the source beside each procedural claim, so the hire learns where an answer comes from. Kipwise's new hire onboarding workflow provides product context for assigned reading and searchable company knowledge. In a sandbox, those capabilities should point to the approved practice set, not every page the employee may eventually use. Keep a manifest for each exercise:

exercise_id: support-refund-01
knowledge_collection: onboarding-sandbox-support-v3
allowed_sources:
  - refund-policy-training-v3
  - escalation-guide-training-v2
synthetic_dataset: support-cases-fixture-v4
attempt_policy: fresh-copy-per-attempt
allowed_tools:
  - lookup_case_simulated
  - draft_reply_simulated
  - refund_in_vendor_sandbox
forbidden_tools:
  - send_reply
  - issue_refund

Resolve this configuration in application code. Do not let the model choose another collection or tool because a retrieved page mentions it.

Fixtures that preserve decisions

Synthetic data should keep the shape and difficulty of the task without copying a real person, customer, secret or transaction. Replacing a name is not enough if the record still holds a unique email, account ID, free text or incident timeline. Build each fixture from a schema with required fields, valid ranges, relationships and edge cases. Generate fictional identifiers that production systems cannot accept, mark every record with a sandbox tenant and exercise ID, and reject any record that lacks the markers.

A good fixture tests a decision. A support scenario can include a purchase date, plan, request reason, prior exception and policy version. A People Operations scenario can include a fictional location, start date, role and equipment need. Write at least four variants:

  • a routine case;
  • a boundary case;
  • a case that requires human escalation, because the correct outcome is sometimes to stop;
  • a stale-instruction case.

The fourth case is easy to leave out. The task arrives with an old instruction attached, such as a forwarded email or a colleague's saved checklist that quotes a superseded limit, while the cited current page says otherwise. The correct move is to notice the conflict, follow the current page and say so. It trains the habit the live assistant needs from the hire: check a page's date and version before acting on what someone pasted.

Use the real tool's sandbox before writing a simulator

Check whether the tool already ships one. Stripe is the example for a refund exercise: its documentation says sandboxes simulate creating real objects, Refund objects included, without moving real money, and a general sandbox with Private access lets you invite a person to the sandbox only, without access to live mode. A vendor maintains its sandbox next to the real API, so it should drift less than a simulator you write, although not every feature is simulated: Stripe says supported simulations vary by product and integration. Use a general sandbox and not the older test mode sandbox, which shares some settings with live mode. Write your own simulators only for internal tools that have no sandbox.

A simulator validates intent and returns a realistic result without performing the action:

def simulate_issue_refund(input, context):
    require(context.environment == "sandbox")
    require(input.case_id.startswith("TRAINING-"))
    require(context.exercise_id)

    decision = evaluate_against_fixture(input)
    record_simulation(context.user, input, decision)
    return simulated_receipt(decision)

Wrap every tool, simulated or vendor-sandboxed, in the same controls:

  • issue a sandbox identity with no production group membership;
  • allow network access only to training services;
  • reject production tenant IDs, domains and resource names;
  • require the exercise ID on every request;
  • store outputs in disposable sandbox storage;
  • expire the sandbox entitlement after the training window.

A visual banner is useful but is not a security boundary. Production gateways must reject sandbox identities, and sandbox gateways must reject production resources, so that one configuration error does not collapse both controls. For the expiry, Microsoft's Entra entitlement management lists identities that need time-limited access for a particular task as a case for access packages, which is what a training window is. Make the sandbox a named package with an owner and an end condition, not an informal copy of production access.

Start every attempt from a fresh copy

Do not build a reset button. Start each attempt from a new copy of the scenario under its own exercise ID, and let finishing mean discarding the copy. There is no reset code to get wrong, no leftover state from the previous learner, and a failed attempt can be replayed from the stored copy when the owner reviews a dispute. Vendors point the same way: Stripe notes that card-level state from one test can affect later tests and recommends a different test card for each independent scenario, and it says a sandbox is temporarily unusable while its test data is deleted.

Accounts also cap general sandboxes (five at Stripe), so in a vendor sandbox create new test objects per attempt and not a new sandbox.

Design scenarios around observable decisions

A useful exercise has a start state, a goal, decision points, expected evidence and an end state, and it does not grade wording against a model answer. Define the sources the employee is expected to consult, the acceptable decisions and reasons, and the actions that must be refused or escalated. A named team owner maintains each scenario and handles disputed outcomes. The assistant guides the exercise but does not own it. Give feedback at the point of error: if the hire selects the wrong policy, show the governing passage and ask them to revise the decision, and if they attempt a forbidden action, explain the boundary and require the correct escalation. A score alone shows the hire neither.

Keep assessment separate from access

Completion evidence can support an access decision, but the model must not grant access. Store structured events such as the source consulted, the decision selected, the escalation chosen, the simulator result, the retry count and a reviewer note, and avoid retaining chat text you do not need. The GitLab onboarding handbook treats a checked box as meaning either that the task is done or that the person checked and it does not apply to them. Record the same distinction here: completed in the sandbox, or waived by the manager as not needed for this role. Promotion rules read both.

Define the rules in advance. A rule might require success on the routine and boundary cases, correct escalation of the high-risk case, and manager confirmation that the role still needs production access. A failed exercise produces targeted practice, not permanent denial. The access owner reviews the evidence and assigns the production entitlement through the normal identity workflow, so the employee practices in one environment while an accountable system and person control the move to production.

Plan for the failures an isolated sandbox still has

  • Stale training knowledge: freeze the exercise when an underlying policy changes, and update the source manifest, expected decisions and fixtures together.
  • Simulator drift: compare simulator rules with the production contract on a schedule.
  • Synthetic data leakage: scan fixtures before release and after edits, and reject real domains, account formats, secrets and copied free text.
  • Hidden production dependency: run the exercise with production network routes blocked. If it breaks, find the dependency instead of opening the route.
  • Expired access: make the expiry visible before the session ends and keep completion evidence separately, rather than extending access with a shared credential.

Verify isolation and learning

Test before assigning the environment to a hire. First prove that approved training retrieval and simulated actions work. Then attempt to retrieve an unapproved document, use a production identifier, call a live endpoint, omit the exercise ID, reuse an expired entitlement and cross into another sandbox tenant. Every attempt should fail visibly and create an event for the owner. Run the same exercise twice in parallel under two sandbox identities and confirm that neither run sees the other's objects.

Review learning quality separately: did the scenario make the employee find the source, apply it to a realistic case, recognize a boundary and choose a safe next step? If it rewards guessing or memorizing a phrase, rewrite it. Track blocked boundary attempts, disputed outcomes and time to owner response to repair the environment, not as employee performance metrics.

Pick the task that needs live access today

Choose one onboarding task that currently needs live observation or risky early access. Write its forbidden action list first, build four synthetic cases, connect only the approved training collection, sandbox every action tool, and prove that production is unreachable. Have the task owner run the verification checklist a new hire will face before any employee gets the sandbox.

Topics

Related posts

Your team’s knowledge, answered in Slack

Add the Kipwise Agent. It answers from your pages with the source, and asks before it changes anything. 7-day trial, no card needed.