AI DEVELOPMENT / DEDICATED BUILD
Reinforcement Learning
Learn action policies from interaction, rewards, and state transitions while explicitly testing exploration, reward design, stability, and safety.
What it is
Learn action policies from interaction, rewards, and state transitions while explicitly testing exploration, reward design, stability, and safety.
This page separates the concept, implementation path, interactive experiment, evaluation criteria, failure modes, and sources so the topic can be studied without relying on a summary tile.
Core concepts
- A Markov decision process models states, actions, transitions, rewards, and discounting.
- A policy maps observations or states to action probabilities.
- Value functions estimate expected long-term return.
- Exploration gathers information; exploitation uses what has already been learned.
- Reward design defines what is optimized, not necessarily what a human intended.
Architecture / workflow
- Define environment state, allowed actions, reward, termination, and safety constraints.
- Start with a tabular or scripted baseline.
- Collect transitions and update a value function or policy.
- Evaluate on fixed seeds and unseen environment variations.
- Stress-test reward hacking, unsafe exploration, and distribution shift.
Technical foundations
- Return Gₜ=Σₖγᵏrₜ₊ₖ₊₁.
- Q-learning update: Q(s,a)←Q(s,a)+α[r+γmaxₐ′Q(s′,a′)−Q(s,a)].
- Policy-gradient methods optimize expected return with gradients estimated from sampled trajectories.
- Off-policy methods can learn from data generated by a different behavior policy.
Evaluation checklist
- Average return with confidence intervals across seeds.
- Success rate and constraint violations.
- Sample efficiency and wall-clock training cost.
- Robustness to changed dynamics, observations, and reward scales.
- Behavior inspection, not reward alone, to detect shortcuts.
Failure modes and limits
- Agents can exploit unintended reward loopholes.
- Exploration can produce unsafe actions in real systems.
- Results may be highly seed-sensitive.
- Simulation policies may not transfer to physical systems without domain-gap testing.
Practical build path
- Implement tabular Q-learning in a small gridworld.
- Visualize Q-values and exploration rate.
- Change the reward structure and observe policy changes.
- Add constraints and compare reward-only optimization with safety-aware evaluation.
Primary / authoritative references
AUTHORED STUDY COMPANION / 3.4
Compare a learned policy with an exact plan
Prerequisites. States, actions, rewards, discounted returns and exploration.
The environment is a five-state chain, starting at 0 and ending at 4. Actions move left or right with clamping at the endpoints. Reaching state 4 gives reward 1; other transitions cost 0.02. Q-learning updates Q(s,a) toward r+γmaxₐ′Q(s′,a′), except that terminal transitions have no bootstrapped continuation.
The companion uses seed 7, 300 episodes, learning rate 0.3, discount 0.9 and ε-greedy exploration at 0.2. It then evaluates the greedy policy without exploration and compares its trajectory with the known shortest plan [0,1,2,3,4]. The undiscounted reward on that path is 0.94; the optimization target uses a discounted return, so those quantities must not be conflated.
This deterministic chain is deliberately easy. A successful seed does not establish robust generalization, safe exploration or a policy for a different transition model. Distinguish termination from a time limit when bootstrapping. Reward design can also encourage shortcuts that conflict with the intended task; a planner baseline helps expose avoidable complexity and wrong objectives.
| Element | Model | Interpretation |
|---|---|---|
| Experience | (s,a,r,s′,terminal) | Observed transition |
| Update | r+γmax Q(s′,·) | No bootstrap after true termination |
| Reference | Exact shortest plan | Separate learning from planning |
Predict, run, compare
The default learned policy should reach the goal along the four-move path. Evaluate multiple seeds separately from training and retain any failed runs.
Download executable notebooks and source Inspect source ↗
Open reinforcement.ipynb in the package. Run cells in order, compare with the reference, then change the stated input. Restart the kernel to reset.
Computed reference result — local CPU fixture
{
"q": [
[
0.5873199332715219,
0.6747999999999996
],
[
0.5872563055867727,
0.7719999999999997
],
[
0.674795616205395,
0.8799999999999998
],
[
0.7719989868564793,
0.9999999999999999
],
[
0.0,
0.0
]
],
"trajectory": [
0,
1,
2,
3,
4
],
"undiscounted_return": 0.94,
"optimal_plan": [
0,
1,
2,
3,
4
],
"seed": 7,
"reached_goal": true,
"scope": "five-state deterministic chain; compare multiple seeds and held-out dynamics before generalizing"
}This is the recorded result of the included code under the package environment. It is not a live service or device measurement.
Extend the experiment
Introduce a stochastic slip probability and evaluate a frozen policy on several seeds. Compare average return, failure rate and an exact planner that knows the changed transition model.
Check your reasoning: At a true terminal transition the Q-learning target is:
- r only — Correct. There is no continuation after termination.
- r+γmax Q of an arbitrary next state — That bootstraps value beyond the defined episode.
- Always zero — The terminal reward still matters.
This is an explained self-check, not a certification of mastery.
Claim-specific sources
Reviewed 2026-10-02: this worked example and its explicit assumptions. Source links and notebook results do not establish production or hardware validation.