Enhancing Experiment-Driven Analytics using Markov Decision Processes

The rapid growth of data has made experiment design increasingly complex, limiting the effectiveness of traditional heuristic or trial-and-error approaches. This thesis introduces a formal and interpretable framework for automated experiment design based on Markov Decision Processes (MDPs). The framework evolves through three versions. MDP~1.0 provides a baseline with uniform transitions and constraint-based rewards. MDP~2.0 incorporates popularity priors, biasing decisions toward widely adopted models. MDP~3.0 adds feedback-aware transitions, enabling personalization and dynamic adaptation to user evaluations. A path-level ranking mechanism further improves interpretability by ranking entire workflows rather than isolated models. The framework was implemented in Python and validated using real and synthetic datasets. The real dataset, derived from COCO benchmark results on anomaly detection, contains neural network models with associated algorithms, hardware requirements, and performance metrics such as accuracy and precision. The synthetic data set mirrors this structure, but scales to 100,000 rows for scalability of stress testing. Policy iteration was employed as the solution method, ensuring convergence to optimal policies under the defined models. Results show that the framework generates valid, scalable, and interpretable workflows, adapting to popularity signals and user feedback. This work lays the foundation for future human-centered AutoML systems and suggests extensions such as dynamic reward learning, customized transitions, and memory-based adaptation.