Deep Q-Learning Aircraft Landing Control Project and Study

This project explores how different hyperparameters affect deep q-learning performance through systematic experimentation. using a custom gymnasium environment simulating aircraft descent, I investigated the complex relationships between batch size, learning rate, discount factor, and exploration strategies.

The neural network architecture consisted of an input layer, 3 hidden layers (256 neurons each), and an output layer, utilizing ReLu activation functions. this setup enabled the agent to learn control policies for managing descent rate, angle, and positioning while maintaining stability throughout the landing sequence. Check out the full 19-page study below.

Full Study

Details

Developed a custom gymnasium environment:
- Designed a custom gymnasium environment for aircraft descent simulation with physics-based reward shaping
Conducted comprehensive parameter studies across:
- batch sizes (32 and 64)
- learning rates (α): 1×10^-4, 5×10^-4, 1×10^-3
- discount factors (γ): 0.999, 0.990, 0.950
- ε-decay rate: 0.995, 0.980
Analyzed stability-performance trade-offs:
- Utilized surface plots and statistical analysis to evaluate how different hyperparameters influence model stability and performance.
Optimal parameter combinations:
- Conducted parameter analysis (batch sizes, learning/decay rates, discount factors) to realize stability and performance trade-offs
- Identified a configuration performing 51% above average (397.9 vs 263.5) displaying the significance of parameter selection (displayed in the top video)

What I learned

Deep q-learning implementation and optimization: Gained hands-on experience in implementing deep q-learning algorithms and optimizing their performance through hyperparameter tuning.
Hyperparameter tuning and its impact on learning dynamics: Learned how different hyperparameters influence the learning process, including convergence rates and stability of the learning agent.
Batch size effects on gradient estimation and learning stability: Understood how varying batch sizes affect gradient estimation accuracy and the overall stability of the learning process.
Statistical analysis and visualization of experimental results: Developed skills in analyzing experimental data using statistical methods and presenting findings through effective visualizations.
Reward shaping (frustrating!!!) strategies for control tasks: Explored techniques for designing reward functions that guide the agent towards desired behaviors in control environments. Even for such a simple task it took a long time to find the sweet spot. Failure is a part of the process, and it took many iterations to converge on rewards that did not break every law of physics, while also landing properly.