InteractiveMap of Operations Research
Back to Uncertainty & Stochastic Systems
MethodAdvanced03.05
Uncertainty & Stochastic Systems

Markov Decision Processes

Controlled stochastic systems with state, action, and reward.

Overview

Markov Decision Processes focuses on controlled stochastic systems with state, action, and reward. In the map of OR, it connects States, Actions, Rewards to decisions that must be modeled, solved, explained, and revised as evidence changes.

MDPs formalize sequential decision-making under uncertainty and connect dynamic programming, reinforcement learning, inventory, queues, and maintenance. The practical use case is clearest in Inventory control, Maintenance, Pricing, Routing, RL, where the method helps turn constraints and tradeoffs into a decision artifact someone can inspect.

Core ideas

States

States is a core checkpoint for Markov Decision Processes: define it concretely, attach units or rules where possible, and test whether stakeholders interpret it the same way.

Actions

Actions is a core checkpoint for Markov Decision Processes: define it concretely, attach units or rules where possible, and test whether stakeholders interpret it the same way.

Rewards

Rewards is a core checkpoint for Markov Decision Processes: define it concretely, attach units or rules where possible, and test whether stakeholders interpret it the same way.

Policies

Policies is a core checkpoint for Markov Decision Processes: define it concretely, attach units or rules where possible, and test whether stakeholders interpret it the same way.

Value iteration

Value iteration is a core checkpoint for Markov Decision Processes: define it concretely, attach units or rules where possible, and test whether stakeholders interpret it the same way.

How to use it

  1. 1Start with Inventory control: write the decision, time horizon, actors, and objective in operational language.
  2. 2Translate the problem into States, Actions, and Rewards; define units and data sources for each one.
  3. 3Build a small instance of Markov Decision Processes that can be solved or simulated by hand inspection before using full production data.
  4. 4Compare the recommendation against a baseline policy, not just against mathematical optimality.
  5. 5Document assumptions, sensitivity results, and the conditions under which the recommendation should be revisited.

Applications

Inventory controlMaintenancePricingRoutingRL
  • Inventory control: compare feasible policies, quantify the operating tradeoffs, and make the assumptions behind the recommendation visible.
  • Maintenance: compare feasible policies, quantify the operating tradeoffs, and make the assumptions behind the recommendation visible.
  • Pricing: compare feasible policies, quantify the operating tradeoffs, and make the assumptions behind the recommendation visible.
  • Routing: compare feasible policies, quantify the operating tradeoffs, and make the assumptions behind the recommendation visible.
  • RL: compare feasible policies, quantify the operating tradeoffs, and make the assumptions behind the recommendation visible.

Common pitfalls

  • Applying Markov Decision Processes because the label sounds appropriate while leaving the actual decision boundary vague.
  • Treating States as a technical detail instead of a modeling choice that affects the recommendation.
  • Reporting one answer without showing sensitivity to demand, capacity, costs, or behavioral assumptions.
  • Ignoring implementation details such as data quality, explainability, ownership, and how users will override bad recommendations.

Resources