# Project export: OpenFinance This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Democratizing Wall Street’s black box - OpenFinance transforms proprietary hedge-fund algorithms into transparent, teachable AI systems, empowering everyone with institutional-grade financial intel. - Devpost: https://devpost.com/software/alphaexec-risk-averse-ddpg - GitHub: https://github.com/UniversalDreams/AI-IS-TAKING-OVER-.git - Demo: https://calhacksfinance.netlify.app/ - Video: https://player.vimeo.com/video/1130689173?byline=0&portrait=0&title=0#t= - Team: 2 GitHub contributor(s) — Jacob L. Johnston (12 commits), Cristofer Arellano (3 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration comes from the technological and community impact: limitations of classical finance when handling large, volatile trades and the idea that finance should not be limited to just those with accessible means, allowing for equitable opportunity. The foundation of OpenFinance is challenging the status quo of institutional trading. Major players like JPMorgan Chase, Goldman Sachs, Citadel Securities, Virtu Financial, Jane Street, and Hudson River Trading all use proprietary optimal execution algorithms, leaving a gap for open, interpretable innovation and questions on the community impact. The Limitation of Analytical Solutions Classical Optimal Execution models, like the original Almgren-Chriss (A-C) framework, rely on mathematical tractability, yielding a non-adaptive, deterministic solution like TWAP (Time-Weighted Average Price). But markets are stochastic - TWAP can't react to price trends, volatility spikes, or microstructure shifts. What We Learned Institutional traders don't just care about average returns (mean); they also want to minimize execution variance. This tradeoff between expected cost and uncertainty defines risk aversion, captured through the Mean-Variance Utility Criterion. Our agent explicitly balances profit v.s. risk, adapting its behavior to volatility in real time. ### What it does OpenFinance is an advanced Deep Reinforcement Learning Agent that determines the optimal pace of trade execution for large stock orders, with risk management as its central principle. Continuous Control: Standard RL agents act discretely (Buy/Hold/Sell). Our DDPG agent outputs a continuous action, e.g., "Sell 12.4% of remaining inventory." Risk Aversion Objective: The agent's reward is a Mean-Variance Utility Function: R_t = -(E[C_t] + lambda * Var[C_t]). Setting lambda = 2.0 makes the policy risk-averse, forcing faster liquidation under volatility while still seeking favorable prices. Performance: Achieved an average normalized reward of -0.0747, meaning an average execution cost of 7 cents per share (~ $90,000 on a $5M order). This is a 92% improvement over naive execution methods like TWAP that might lose several dollars per share due to poor timing and market impact, demonstrating institutional-grade performance and adaptive intelligence. To put it into context, selling 10,000 shares at $500 a share (a $5m trade) would cost only about $747 total in execution costs with our agent. Professional firms consider under 5 cents per share excellent performance, so our result is competitive with industry standards. ### How we built it OpenFinance was built over a weekend using a highly parallelized Actor-Critic (DDPG) architecture, trained entirely on a custom high-fidelity Almgren-Chriss simulator. We built everything from scratch and did not rely on existing APIs, frameworks, or wrappers. Unlike prior research implementations that assume perfect data and stationary dynamics, we redesigned the Almgren-Chriss simulator for RL compatibility, introducing stochastic volatility, high-frequency real market data, and adaptive reward scaling. We also reinterpretated A-C as a learnable environment, enabling the agent to interact with market microstructure rather than merely backtest static data. This design shift from theoretical replication to interactive, teachable finance physics is what makes OpenFinance an innovation in accessible quantitative research, not a reproduction of it. Most people use A-C as a static mathematical benchmark; we turned it into a simulated world for an interactive agent (with step(), state vectors, and stochastic noise). Our 10-dimensional state vector is another custom design, not copied. It's our way of approximating the belief state for a POMPDP. RL Architecture (DDPG Core): Implemented a Deep Deterministic Policy Gradient framework with: 1) Actor, Critic, and two Target Networks, 2) Actor outputs continuous actions, 3) Critic evaluates them via Q-values, 4) Target networks stabilize the "Deadly Triad" (bootstrapping, off-policy learning, function approximation). Market Simulator (Environment Physics): Built the Discrete-Time Almgren-Chriss model - temporary impact = 3e-6, permanent impact = 1e-7, and a noise term. We used 50,000 real 1-minute NVIDIA price bars to ensure realism. Each episode simulates a full liquidation horizon under stochastic volatility. Stabilization: Experience Replay Buffer to break temporal correlations. Polyak Averaging (tau = 0.0005) for stable target updates. Early Stopping (500-episode patience) to prevent catastrophic forgetting. State Representation: A 10-dimensional augmented state vector, including: 5 lags of normalized lag returns, inventory fraction remaining, temporary and permanent impact history, and recent volatility estimates. This approximates the Belief State in a Partially Observable MDP (POMPDP). ### Challenges we ran into Continuous Control Convergence: DDPG is unstable due to the aforementioned Deadly Triad. We mitigated this with small tau updates, decaying Ornstein-Uhlenbeck noise, and conservative learning rates. Market Physics Calibration: Tuning A-C parameters against a high tau created sensitive market behavior. Treating the simulator itself as a stochastic control system yielded stable, realistic liquidation dynamics. Data Fidelity vs Quantity: API pagination limits forced us to use 9 months of 1-minute, high-resolution data instead of multiple years - but this improved realism over longer, low-fidelity samples. ### Accomplishments we're proud of Institutional-Level Performance: Cost reduced by 92% compared to TWAP; converged at 7 cents a share, 1.5 basis points execution cost. Adaptive Intelligence: The agent occasionally achieved positive returns (+1.79) by opportunistically selling into favorable price movements - impossible for deterministic benchmarks. Theoretical Discipline: Fully grounded in utility theory, policy gradient optimization, and finite-horizon MDPs as outlined in Foundations of Reinforcement Learning with Applications in Finance, a textbook recommended by our professor. Open Innovation: We didn’t just implement the Almgren–Chriss model. We turned it into an interactive RL environment that learns market dynamics from real 1-minute data. We engineered a custom 10-dimensional state representation capturing historical trends and impact effects, bridging theoretical finance with modern reinforcement learning. Our innovation wasn’t inventing a new model, but making institutional-grade execution research open, transparent, and teachable. ### What we learned Policy Gradient Power: Continuous-action problems require policy gradient methods. DDPG's Actor-Critic loop proved indispensable. Risk-Return Tradeoff (lambda): Lambda is the control knob of behavior. Low lambda means patient, opportunistic strategy. High lambda means defensive, volatility-averse strategy. We demonstrated direct control over trading temperment. State History as Belief: Adding history lags was an effective way to encode memory, approximating the belief state in real-world POMPDPs under hackathon constraints. ### What's next for AlphaExec - Risk Averse DDPG We want to continue to improve existing public quantitative finance research and knowledge and showcase their relevance and applicability, urging and encouraging others outside of Wall Street to build their own, and further remove the disparity between the ultra-wealthy and regular people when it comes to trading algorithms. ## README (from the GitHub repository) This project implements a Deep Deterministic Policy Gradient (DDPG) agent that learns to optimally liquidate large stock positions by minimizing trading costs while balancing execution speed against market impact. The agent uses an actor-critic neural network architecture trained on a custom Almgren-Chriss trading environment that simulates real market dynamics including temporary and permanent price impact, volatility, and execution risk. We used a 10-dimensional state space to capture price, inventory, time remaining, volatility, price trends, and last market impact, while the agent outputs continuous actions representing the percentage of remaining shares to sell at each 5-minute interval. Through 2000 training episodes with experience replay and decaying exploration noise, the agent learns sophisticated strategies including dynamic position sizing, volatility adaptation, and implementation shortfall minimization compared to naive execution baselines. ## How to Run cd Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum python main_ddpg.py The agent will train for up to 2000 episodes with early stopping and save trained models to `tmp/ddpg/`. Training progress and performance curves are saved to `plots/optimal_execution.png`. ## Visual Representation https://calhacksfinance.netlify.app/ *Framework inspired by "Foundations of Reinforcement Learning with Applications in Finance" by Ashwin Rao and Tikhon Jelvis* ## Detected evidence (automated analysis) Indexed codebase: 13 recognized source files, 57 KB. - Python (language) — detected in the code - React (technology) — claimed on Devpost, not found in the code - Tailwind CSS (technology) — claimed on Devpost, not found in the code - TensorFlow (technology) — claimed on Devpost, not found in the code - TypeScript (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (16 of 16) ``` .gitignore Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/buffer.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/create_visualizations.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/ddpg_tf2.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /data/NVDA_historical.csv Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /scripts/fetch_historical_data.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /src/__init__.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /src/envs/trading_env.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /src/utils/data_loader.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/main_ddpg.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/networks.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/utils.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/pytorch/lunar-lander/ddpg_torch.py Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/pytorch/lunar-lander/main_torch.py README.md requirements.txt ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update README.md - Update README.md - Add project inspiration credit to README - Update DDPG optimal execution implementation with test evaluation and graphs - Add DDPG optimal execution agent - Avg reward -0.0747 (92% better than TWAP) - Adds normalization and improves metrics tracking - Adds episode metrics tracking, enhanced info dict, and TWAP methods - Adds slippage - Creates test file for env - Creates data_loader and trading_env - Typos - Removes unused files - Removes unused files - Fetch historical data - Removes redundant IDE files - Initial project structure - Initial project structure ## Key source files (fetched from GitHub, selected and truncated for size) ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/utils.py ```python import matplotlib.pyplot as plt import numpy as np import os def plot_learning_curve(x, scores, figure_file): """Plot the learning curve showing episode scores over time.""" running_avg = np.zeros(len(scores)) for i in range(len(running_avg)): running_avg[i] = np.mean(scores[max(0, i-100):(i+1)]) # Create plots directory if it doesn't exist os.makedirs(os.path.dirname(figure_file), exist_ok=True) plt.figure(figsize=(10, 6)) plt.plot(x, running_avg, label='Running Average (100 episodes)', linewidth=2) plt.plot(x, scores, alpha=0.3, label='Episode Score') plt.title('DDPG Training Progress - Optimal Execution') plt.xlabel('Episode') plt.ylabel('Score (Revenue - Risk Penalty)') plt.legend() plt.grid(True, alpha=0.3) plt.savefig(figure_file) print(f'Plot saved to {figure_file}') ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/pytorch/lunar-lander/main_torch.py ```python from ddpg_torch import Agent import gym import numpy as np from utils import plotLearning env = gym.make('LunarLanderContinuous-v2') agent = Agent(alpha=0.000025, beta=0.00025, input_dims=[8], tau=0.001, env=env, batch_size=64, layer1_size=400, layer2_size=300, n_actions=2) #agent.load_models() np.random.seed(0) score_history = [] for i in range(1000): obs = env.reset() done = False score = 0 while not done: act = agent.choose_action(obs) new_state, reward, done, info = env.step(act) agent.remember(obs, act, reward, new_state, int(done)) agent.learn() score += reward obs = new_state #env.render() score_history.append(score) #if i % 25 == 0: # agent.save_models() print('episode ', i, 'score %.2f' % score, 'trailing 100 games avg %.3f' % np.mean(score_history[-100:])) filename = 'LunarLander-alpha000025-beta00025-400-300.png' plotLearning(score_history, filename, window=100) ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/buffer.py ```python import numpy as np class ReplayBuffer: def __init__(self, max_size, input_shape, n_actions): self.mem_size = max_size self.mem_cntr = 0 self.state_memory = np.zeros((self.mem_size, *input_shape)) self.new_state_memory = np.zeros((self.mem_size, *input_shape)) self.action_memory = np.zeros((self.mem_size, n_actions)) self.reward_memory = np.zeros(self.mem_size) self.terminal_memory = np.zeros(self.mem_size, dtype=np.bool) def store_transition(self, state, action, reward, state_, done): index = self.mem_cntr % self.mem_size self.state_memory[index] = state self.new_state_memory[index] = state_ self.action_memory[index] = action self.reward_memory[index] = reward self.terminal_memory[index] = done self.mem_cntr += 1 def sample_buffer(self, batch_size): max_mem = min(self.mem_cntr, self.mem_size) batch = np.random.choice(max_mem, batch_size, replace=False) states = self.state_memory[batch] states_ = self.new_state_memory[batch] actions = self.action_memory[batch] rewards = self.reward_memory[batch] dones = self.terminal_memory[batch] return states, actions, rewards, states_, dones ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/networks.py ```python import os import tensorflow as tf import tensorflow.keras as keras from tensorflow.keras.layers import Dense class CriticNetwork(keras.Model): def __init__(self, fc1_dims=512, fc2_dims=512, name='critic', chkpt_dir='tmp/ddpg'): super(CriticNetwork, self).__init__() self.fc1_dims = fc1_dims self.fc2_dims = fc2_dims self.model_name = name self.checkpoint_dir = chkpt_dir # Create checkpoint directory if it doesn't exist os.makedirs(self.checkpoint_dir, exist_ok=True) self.checkpoint_file = os.path.join(self.checkpoint_dir, self.model_name+'_ddpg.weights.h5') self.fc1 = Dense(self.fc1_dims, activation='relu') self.fc2 = Dense(self.fc2_dims, activation='relu') self.q = Dense(1, activation=None) def call(self, state, action): action_value = self.fc1(tf.concat([state, action], axis=1)) action_value = self.fc2(action_value) q = self.q(action_value) return q class ActorNetwork(keras.Model): def __init__(self, fc1_dims=512, fc2_dims=512, n_actions=2, name='actor', chkpt_dir='tmp/ddpg'): super(ActorNetwork, self).__init__() self.fc1_dims = fc1_dims self.fc2_dims = fc2_dims self.n_actions = n_actions self.model_name = name self.checkpoint_dir = chkpt_dir # Create checkpoint directory if it doesn't exist os.makedirs(self.checkpoint_dir, exist_ok=True) self.checkpoint_file = os.path.join(self.checkpoint_dir, self.model_name+'_ddpg.weights.h5') self.fc1 = Dense(self.fc1_dims, activation='relu') self.fc2 = Dense(self.fc2_dims, activation='relu') self.mu = Dense(self.n_actions, activation='tanh') def call(self, state): prob = self.fc1(state) prob = self.fc2(prob) mu = self.mu(prob) # Rescale tanh output from [-1, 1] to [0, 1] for percentage actions # mu_scaled = (mu + 1) / 2 mu_scaled = (mu + 1.0) * 0.5 return mu_scaled ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/ddpg_tf2.py ```python import tensorflow as tf import tensorflow.keras as keras from tensorflow.keras.optimizers import Adam from buffer import ReplayBuffer from networks import ActorNetwork, CriticNetwork #alpha is the learning rate for the actor #beta is the learning rate for the critic class Agent: def __init__(self, input_dims, alpha=0.0001, beta=0.001, env=None, gamma=1.0, n_actions=1, max_size=1000000, tau=0.005, fc1=256, fc2=128, batch_size=64, noise=0.2): self.gamma = gamma self.tau = tau self.memory = ReplayBuffer(max_size, input_dims, n_actions) self.batch_size = batch_size self.n_actions = n_actions self.noise = noise # Handle custom environments without action_space attribute if env is not None and hasattr(env, 'action_space'): self.max_action = env.action_space.high[0] self.min_action = env.action_space.low[0] else: # Default for optimal execution: action is percentage (0 to 1) self.max_action = 1.0 self.min_action = 0.0 self.actor = ActorNetwork(n_actions=n_actions, name='actor') self.critic = CriticNetwork(name='critic') self.target_actor = ActorNetwork(n_actions=n_actions, name='target_actor') self.target_critic = CriticNetwork(name='target_critic') self.actor.compile(optimizer=Adam(learning_rate=alpha)) self.critic.compile(optimizer=Adam(learning_rate=beta)) self.target_actor.compile(optimizer=Adam(learning_rate=alpha)) self.target_critic.compile(optimizer=Adam(learning_rate=beta)) self.update_network_parameters(tau=1) def update_network_parameters(self, tau=None): if tau is None: tau = self.tau weights = [] targets = self.target_actor.weights for i, weight in enumerate(self.actor.weights): weights.append(weight * tau + targets[i]*(1-tau)) self.target_actor.set_weights(weights) weights = [] targets = self.target_critic.weights for i, weight in enumerate(self.critic.weights): weights.append(weight * tau + targets[i]*(1-tau)) self.target_critic.set_weights(weights) def remember(self, state, action, reward, new_state, done): self.memory.store_transition(state, action, reward, new_state, done) def save_models(self): print('... saving models ...') # Only save if all networks have been built (called with data at least once) if not (self.actor.built and self.target_actor.built and self.critic.built and self.target_critic.built): print('Networks not built yet, skipping save') return self.actor.save_weights(self.actor.checkpoint_file) self.target_actor.save_weights(self.target_actor.checkpoint_file) self.critic.save_weights(self.critic.checkpoint_file) self.target_critic.save_weights(self.target_critic.checkpoint_file) def load_models(self): print('... loading models ...') self.actor.load_weights(self.actor.checkpoint_file) self.target_actor.load_weights(self.target_actor.checkpoint_file) self.critic.load_weights(self.critic.checkpoint_file) self.target_critic.load_weights(self.target_critic.checkpoint_file) def choose_action(self, observation, evaluate=False): state = tf.convert_to_tensor([observation], dtype=tf.float32) actions = self.actor(state) if not evaluate: actions += tf.random.normal(shape=[self.n_actions], mean=0.0, stddev=self.noise) # note that if the env has an action > 1, we have to multiply by # max action at some point actions = tf.clip_by_value(actions, self.min_action, self.max_action) return actions[0] def learn(self): if self.memory.mem_cntr < self.batch_size: return state, action, reward, new_state, done = \ self.memory.sample_buffer(self.batch_size) states = tf.convert_to_tensor(state, dtype=tf.float32) states_ = tf.convert_to_tensor(new_state, dtype=tf.float32) rewards = tf.convert_to_tensor(reward, dtype=tf.float32) actions = tf.convert_to_tensor(action, dtype=tf.float32) with tf.GradientTape() as tape: target_actions = self.target_actor(states_) critic_value_ = tf.squeeze(self.target_critic( states_, target_actions), 1) critic_value = tf.squeeze(self.critic(states, actions), 1) target = rewards + self.gamma*critic_value_*(1-done) critic_loss = keras.losses.MSE(target, critic_value) critic_network_gradient = tape.gradient(critic_loss, self.critic.trainable_variables) self.critic.optimizer.apply_gradients(zip( critic_network_gradient, self.critic.trainable_variables)) with tf.GradientTape() as tape: new_policy_actions = self.actor(states) actor_loss = -self.critic(states, new_policy_actions) actor_loss = tf.math.reduce_mean(actor_loss) actor_network_gradient = tape.gradient(actor_loss, self.actor.trainable_variables) self.actor.optimizer.apply_gradients(zip( actor_network_gradient, self.actor.trainable_variables)) self.update_network_parameters() ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/main_ddpg.py ```python import sys import os #----TRAINING THE AGENT---- # Store original directory for imports original_dir = os.path.dirname(os.path.abspath(__file__)) # Add Environment directory to path (use local copy with all files) env_path = os.path.join(original_dir, 'Environment ') sys.path.insert(0, env_path) sys.path.insert(0, original_dir) # Change to Environment directory so data loading works os.chdir(env_path) import numpy as np from ddpg_tf2 import Agent #graphs of how agent performance improves over training episodes from utils import plot_learning_curve from src.envs.trading_env import OptimalExecutionEnv ## Helper function for state normalization (matches partner's implementation) def normalize_state_manual(state, env): """Normalize state to help neural network training - matches root env logic.""" # Calculate price statistics from data if not cached if not hasattr(env, '_price_stats_cached'): env._price_mean = env.data_df['close'].mean() env._price_std = env.data_df['close'].std() # Typical impact per step (average volume per step × permanent impact factor) env._typical_impact = env.THETA * (env.X0 / env.T) env._price_stats_cached = True return np.array([ (state.current_price - env._price_mean) / env._price_std, state.inventory_left / env.X0, state.time_remaining / env.T, state.volatility / env.sigma, *state.price_trend_vector, state.last_perm_impact / max(env._typical_impact, 1e-8) ], dtype=np.float32) ##Import the environment you want to use if __name__ == '__main__': # Initialize custom environment with parameters # 50 steps × 5 min/step = 250 minutes (~4 hours of trading) # More realistic horizon for learning optimal liquidation strategy env = OptimalExecutionEnv(initial_shares=10000, total_time_steps=50) # State dimension: price + inventory + time + volatility + trend_vector + last_impact #we have the 6 features we need but the trend vector has a length of 5 state_dim = 10 agent = Agent(input_dims=[state_dim], env=None, # env=None since custom env n_actions=1) # 1 action: percentage to sell # Reduced episodes for hackathon with early stopping protection n_games = 2000 best_score = -np.inf # Start with very low score score_history = [] load_checkpoint = False # Train from scratch with protections #load saved progress if load_checkpoint: #this is to sample from a randomized batch n_steps = 0 while n_steps <= agent.batch_size: #making sure memory isn't empty state = env.reset() observation = normalize_state_manual(state, env) action = np.random.uniform(0, 1) # Random action between 0 and 1 state_, reward, done, info = env.step(state, action) observation_ = normalize_state_manual(state_, env) agent.remember(observation, [action], reward, observation_, done) n_steps += 1 agent.learn() agent.load_models() evaluate = True else: evaluate = False #training of the agent no_improvement_count = 0 # Track episodes without improvement for i in range(n_games): #num of episosdes state = env.reset() #initial state at the start of each episode observation = normalize_state_manual(state, env) # Matches partner's normalization done = False score = 0 terminal_reward = 0.0 # Decay exploration noise over time (0.2 → 0.05) agent.noise = max(0.05, 0.2 * (0.995 ** i)) while not done: action = agent.choose_action(observation, evaluate) action_value = float(action[0]) if hasattr(action, '__iter__') else float(action) state_, reward, done, info = env.step(state, action_value) #next state after action observation_ = normalize_state_manual(state_, env) # Matches partner's normalization score += reward if done and reward != 0: # Terminal reward terminal_reward = reward agent.remember(observation, [action_value], reward, observation_, done) #don't want to train during evaluation if not load_checkpoint: agent.learn() observation = observation_ state = state_ score_history.append(score) avg_score = np.mean(score_history[-100:]) if avg_score > best_score: best_score = avg_score no_improvement_count = 0 # Reset counter #only save modals during training if not load_checkpoint: agent.save_models() else: no_improvement_count += 1 # Early stopping if no improvement for 500 episodes (was too aggressive at 300) if no_improvement_count >= 500: print(f"\nEarly stopping at episode {i}: No improvement for 500 episodes") print(f"Best avg score achieved: {best_score:.4f}") break # Print with terminal reward info if i < 20 or i % 50 == 0: # More frequent early logging print(f'ep {i:4d} | reward {terminal_reward:7.4f} | avg {avg_score:7.4f} | noise {agent.noise:.3f}') else: print(f'ep {i:4d} | reward {terminal_reward:7.4f} | avg {avg_score:7.4f}') if not load_checkpoint: # Use actual number of episodes trained (not n_games) for plotting x = [i+1 for i in range(len(score_history))] figure_file = 'plots/optimal_execution.png' plot_learning_curve(x, score_history, figure_file) ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/pytorch/lunar-lander/ddpg_torch.py ```python import os import torch as T import torch.nn as nn import torch.nn.functional as F import torch.optim as optim import numpy as np class OUActionNoise(object): def __init__(self, mu, sigma=0.15, theta=.2, dt=1e-2, x0=None): self.theta = theta self.mu = mu self.sigma = sigma self.dt = dt self.x0 = x0 self.reset() def __call__(self): x = self.x_prev + self.theta * (self.mu - self.x_prev) * self.dt + \ self.sigma * np.sqrt(self.dt) * np.random.normal(size=self.mu.shape) self.x_prev = x return x def reset(self): self.x_prev = self.x0 if self.x0 is not None else np.zeros_like(self.mu) def __repr__(self): return 'OrnsteinUhlenbeckActionNoise(mu={}, sigma={})'.format( self.mu, self.sigma) class ReplayBuffer(object): def __init__(self, max_size, input_shape, n_actions): self.mem_size = max_size self.mem_cntr = 0 self.state_memory = np.zeros((self.mem_size, *input_shape)) self.new_state_memory = np.zeros((self.mem_size, *input_shape)) self.action_memory = np.zeros((self.mem_size, n_actions)) self.reward_memory = np.zeros(self.mem_size) self.terminal_memory = np.zeros(self.mem_size, dtype=np.float32) def store_transition(self, state, action, reward, state_, done): index = self.mem_cntr % self.mem_size self.state_memory[index] = state self.new_state_memory[index] = state_ self.action_memory[index] = action self.reward_memory[index] = reward self.terminal_memory[index] = 1 - done self.mem_cntr += 1 def sample_buffer(self, batch_size): max_mem = min(self.mem_cntr, self.mem_size) batch = np.random.choice(max_mem, batch_size) states = self.state_memory[batch] actions = self.action_memory[batch] rewards = self.reward_memory[batch] states_ = self.new_state_memory[batch] terminal = self.terminal_memory[batch] return states, actions, rewards, states_, terminal class CriticNetwork(nn.Module): def __init__(self, beta, input_dims, fc1_dims, fc2_dims, n_actions, name, chkpt_dir='tmp/ddpg'): super(CriticNetwork, self).__init__() self.input_dims = input_dims self.fc1_dims = fc1_dims self.fc2_dims = fc2_dims self.n_actions = n_actions self.checkpoint_file = os.path.join(chkpt_dir,name+'_ddpg') self.fc1 = nn.Linear(*self.input_dims, self.fc1_dims) f1 = 1./np.sqrt(self.fc1.weight.data.size()[0]) T.nn.init.uniform_(self.fc1.weight.data, -f1, f1) T.nn.init.uniform_(self.fc1.bias.data, -f1, f1) #self.fc1.weight.data.uniform_(-f1, f1) #self.fc1.bias.data.uniform_(-f1, f1) self.bn1 = nn.LayerNorm(self.fc1_dims) self.fc2 = nn.Linear(self.fc1_dims, self.fc2_dims) f2 = 1./np.sqrt(self.fc2.weight.data.size()[0]) #f2 = 0.002 T.nn.init.uniform_(self.fc2.weight.data, -f2, f2) T.nn.init.uniform_(self.fc2.bias.data, -f2, f2) #self.fc2.weight.data.uniform_(-f2, f2) #self.fc2.bias.data.uniform_(-f2, f2) self.bn2 = nn.LayerNorm(self.fc2_dims) self.action_value = nn.Linear(self.n_actions, self.fc2_dims) f3 = 0.003 self.q = nn.Linear(self.fc2_dims, 1) T.nn.init.uniform_(self.q.weight.data, -f3, f3) T.nn.init.uniform_(self.q.bias.data, -f3, f3) #self.q.weight.data.uniform_(-f3, f3) #self.q.bias.data.uniform_(-f3, f3) self.optimizer = optim.Adam(self.parameters(), lr=beta) self.device = T.device('cuda:0' if T.cuda.is_available() else 'cuda:1') self.to(self.device) def forward(self, state, action): state_value = self.fc1(state) state_value = self.bn1(state_value) state_value = F.relu(state_value) state_value = self.fc2(state_value) state_value = self.bn2(state_value) action_value = F.relu(self.action_value(action)) state_action_value = F.relu(T.add(state_value, action_value)) state_action_value = self.q(state_action_value) return state_action_value def save_checkpoint(self): print('... saving checkpoint ...') T.save(self.state_dict(), self.checkpoint_file) def load_checkpoint(self): print('... loading checkpoint ...') self.load_state_dict(T.load(self.checkpoint_file)) class ActorNetwork(nn.Module): def __init__(self, alpha, input_dims, fc1_dims, fc2_dims, n_actions, name, chkpt_dir='tmp/ddpg'): super(ActorNetwork, self).__init__() self.input_dims = input_dims self.fc1_dims = fc1_dims self.fc2_dims = fc2_dims self.n_actions = n_actions self.checkpoint_file = os.path.join(chkpt_dir,name+'_ddpg') self.fc1 = nn.Linear(*self.input_dims, self.fc1_dims) f1 = 1./np.sqrt(self.fc1.weight.data.size()[0]) T.nn.init.uniform_(self.fc1.weight.data, -f1, f1) T.nn.init.uniform_(self.fc1.bias.data, -f1, f1) #self.fc1.weight.data.uniform_(-f1, f1) #self.fc1.bias.data.uniform_(-f1, f1) self.bn1 = nn.LayerNorm(self.fc1_dims) self.fc2 = nn.Linear(self.fc1_dims, self.fc2_dims) #f2 = 0.002 f2 = 1./np.sqrt(self.fc2.weight.data.size()[0]) T.nn.init.uniform_(self.fc2.weight.data, -f2, f2) T.nn.init.uniform_(self.fc2.bias.data, -f2, f2) #self.fc2.weight.data.uniform_(-f2, f2) #self.fc2.bias.data.uniform_(-f2, f2) self.bn2 = nn.LayerNorm(self.fc2_dims) #f3 = 0.004 f3 = 0.003 self.mu = nn.Linear(self.fc2_dims, self.n_actions) T.nn.init.uniform_(self.mu.weight.data, -f3, f3) T.nn.init.uniform_(self.mu.bias.data, -f3, f3) #self.mu.weight.data.uniform_(-f3, f3) #self.mu.bia [truncated — 7053 more characters] ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/create_visualizations.py ```python """ Comprehensive Visualization Script for DDPG Optimal Execution Agent Generates 4 key graphs for presentation/demo """ import sys import os import numpy as np import matplotlib.pyplot as plt from matplotlib.gridspec import GridSpec # Setup paths original_dir = os.path.dirname(os.path.abspath(__file__)) env_path = os.path.join(original_dir, 'Environment ') sys.path.insert(0, env_path) sys.path.insert(0, original_dir) os.chdir(env_path) from ddpg_tf2 import Agent from src.envs.trading_env import OptimalExecutionEnv # Set style plt.style.use('seaborn-v0_8-whitegrid') plt.rcParams['figure.figsize'] = (12, 8) plt.rcParams['font.size'] = 10 def normalize_state_manual(state, env): """Normalize state for agent input - matches main_ddpg.py""" # Calculate price statistics from data if not cached if not hasattr(env, '_price_stats_cached'): env._price_mean = env.data_df['close'].mean() env._price_std = env.data_df['close'].std() # Typical impact per step (average volume per step × permanent impact factor) env._typical_impact = env.THETA * (env.X0 / env.T) env._price_stats_cached = True return np.array([ (state.current_price - env._price_mean) / env._price_std, state.inventory_left / env.X0, state.time_remaining / env.T, state.volatility / env.sigma, *state.price_trend_vector, state.last_perm_impact / max(env._typical_impact, 1e-8) ], dtype=np.float32) def graph1_learning_curve(score_history, filename='graph1_learning_curve.png'): """ Graph 1: DDPG Agent Convergence - From Random to Optimal Shows raw episode rewards + 100-episode rolling average """ print("Creating Graph 1: Learning Curve...") fig, ax = plt.subplots(figsize=(14, 8)) episodes = np.arange(1, len(score_history) + 1) # Calculate rolling average window = 100 running_avg = np.zeros(len(score_history)) for i in range(len(score_history)): running_avg[i] = np.mean(score_history[max(0, i-window+1):i+1]) # Plot raw scores as scatter ax.scatter(episodes, score_history, alpha=0.3, s=20, c='lightblue', label='Episode Reward (Raw)', edgecolors='none') # Plot rolling average ax.plot(episodes, running_avg, linewidth=3, color='darkblue', label='100-Episode Rolling Average') # Mark early stopping point ax.axvline(x=len(score_history), color='red', linestyle='--', linewidth=2, label=f'Early Stopping (Episode {len(score_history)})') # Mark final performance final_avg = running_avg[-1] ax.axhline(y=final_avg, color='green', linestyle='--', linewidth=2, alpha=0.7, label=f'Final Avg: {final_avg:.4f}') # Mark TWAP baseline ax.axhline(y=0.0, color='orange', linestyle='--', linewidth=2, alpha=0.7, label='TWAP Baseline (0.0)') ax.set_xlabel('Episode', fontsize=14, fontweight='bold') ax.set_ylabel('Reward (Negative Cost)', fontsize=14, fontweight='bold') ax.set_title('DDPG Agent Convergence: From Random to Optimal\n' + 'Proof of Learning Through Trial and Error', fontsize=16, fontweight='bold', pad=20) ax.legend(loc='lower right', fontsize=11) ax.grid(True, alpha=0.3) # Add annotation ax.annotate('Initial Exploration\n(High Variance)', xy=(50, -2.5), fontsize=11, ha='center', bbox=dict(boxstyle='round,pad=0.5', facecolor='yellow', alpha=0.3)) ax.annotate('Convergence\n(Stable Policy)', xy=(600, final_avg), fontsize=11, ha='center', bbox=dict(boxstyle='round,pad=0.5', facecolor='lightgreen', alpha=0.3)) plt.tight_layout() plt.savefig(filename, dpi=300, bbox_inches='tight') print(f"✓ Saved: {filename}") plt.close() def graph2_execution_profile(agent, env, filename='graph2_execution_profile.png'): """ Graph 2: Adaptive Liquidation vs. Passive TWAP Shows agent's adaptive selling vs TWAP's fixed schedule with price overlay """ print("Creating Graph 2: Execution Profile...") # Run one episode with trained agent state = env.reset() observation = normalize_state_manual(state, env) timesteps = [] prices = [] agent_volumes = [] twap_volumes = [] agent_inventory = [] twap_inventory = [] done = False step = 0 current_inventory = env.X0 # initial_shares twap_inventory_current = env.X0 twap_sell_per_step = env.X0 / env.T # total_time_steps while not done and step < env.T: # Agent action action = agent.choose_action(observation, evaluate=True) action = np.clip(action, 0, 1)[0] # Record data timesteps.append(step) prices.append(state.current_price) # Agent sells agent_sell = action * state.inventory_left agent_volumes.append(agent_sell) agent_inventory.append(current_inventory) current_inventory -= agent_sell # TWAP sells twap_sell = min(twap_sell_per_step, twap_inventory_current) twap_volumes.append(twap_sell) twap_inventory.append(twap_inventory_current) twap_inventory_current -= twap_sell # Step environment state, reward, done, info = env.step(state, action) observation = normalize_state_manual(state, env) step += 1 # Create figure with two y-axes fig, ax1 = plt.subplots(figsize=(14, 8)) ax2 = ax1.twinx() # Plot volumes (left y-axis) ax1.bar([t - 0.2 for t in timesteps], twap_volumes, width=0.4, alpha=0.6, color='orange', label='TWAP (Fixed 200/step)') ax1.bar([t + 0.2 for t in timesteps], agent_volumes, width=0.4, alpha=0.8, color='darkblue', label='DDPG Agent (Adaptive)') # Plot price (right y-axis) ax2.plot(timesteps, [truncated — 8100 more characters] ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /scripts/fetch_historical_data.py ```python import requests import pandas as pd import numpy as np import os import time from dotenv import load_dotenv # --- Configuration Constants (Modify if needed) --- TICKER = "NVDA" MULTIPLIER = 1 # 1-minute bars TIMESPAN = "minute" # Adjusted date range to ensure total bar count is <= 50,000 # (Approx. 8 months of trading days should fit) START_DATE = "2024-04-01" END_DATE = "2025-01-01" OUTPUT_DIR = "data" OUTPUT_FILE_PATH = os.path.join(OUTPUT_DIR, f"{TICKER}_historical.csv") MAX_LIMIT = 50000 def fetch_polygon_data_and_save(): """Fetches Polygon.io aggregate data and saves it to a local CSV.""" # 1. Load envs variables from.env file load_dotenv() # 2. Retrieve API Key POLYGON_API_KEY = os.getenv("POLYGON_API_KEY") if not POLYGON_API_KEY: print("FATAL ERROR: POLYGON_API_KEY not found. Ensure it is set in your.env file.") return print(f"Starting fetch for {TICKER} ({MULTIPLIER}-{TIMESPAN} bars) from {START_DATE} to {END_DATE}...") # Polygon Aggregates API Endpoint url = f"https://api.polygon.io/v2/aggs/ticker/{TICKER}/range/{MULTIPLIER}/{TIMESPAN}/{START_DATE}/{END_DATE}" params = { "adjusted": "true", "sort": "asc", "apiKey": POLYGON_API_KEY, "limit": MAX_LIMIT } try: response = requests.get(url, params=params) response.raise_for_status() data = response.json() if not data.get('results'): print("No data returned from Polygon.io. Check ticker and date range.") return # Convert results list to DataFrame df = pd.DataFrame(data['results']) # Rename columns: 'c' is close price, 't' is Unix timestamp df.rename(columns={'c': 'close', 'v': 'volume', 't': 'timestamp'}, inplace=True) # Convert Unix timestamp (ms) to datetime df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms') df.set_index('timestamp', inplace=True) # Save the processed DataFrame to a local file os.makedirs(OUTPUT_DIR, exist_ok=True) df.to_csv(OUTPUT_FILE_PATH) print(f"\n--- SUCCESS ---") print(f"Successfully fetched {len(df)} records for {TICKER}.") print(f"Data saved to: {OUTPUT_FILE_PATH}") except requests.exceptions.RequestException as e: print(f"\nFATAL API ERROR: Could not connect to Polygon.io or bad response.") print(f"Details: {e}") except Exception as e: print(f"\nAn unexpected error occurred: {e}") if __name__ == '__main__': fetch_polygon_data_and_save() ``` ### Agent/Agent/ReinforcementLearning/PolicyGradient/DDPG/DDPG/pendulum/Environment /src/utils/data_loader.py ```python import pandas as pd import numpy as np import os from typing import Tuple, Dict # --- 1. Fixed Market Microstructure Parameters (Optimized for NVIDIA) --- AC_PARAMS: Dict[str, float] = { "PERMANENT_IMPACT_THETA": 1e-7, # Low (NVDA very liquid) "TEMPORARY_IMPACT_ETA": 3e-6, # Low-moderate (tight spreads) "RISK_AVERSION_LAMBDA": 2.0, # Conservative starting point "TIME_STEP_DELTA": 5.0 # Trade every 5 minutes } DATA_PATH = os.path.join("data", "NVDA_historical.csv") def load_and_calculate_market_params() -> Tuple[pd.DataFrame, float]: """ Loads historical price data and calculates sigma. NOTE: P_REF (reference price) is NOT calculated here because it should be episode-specific (set to each episode's starting price), not a global constant. Returns: (DataFrame with price data, Estimated Volatility Sigma) """ print(f"Loading environment data from {DATA_PATH} to calculate sigma...") try: df = pd.read_csv(DATA_PATH, index_col='timestamp', parse_dates=True) except FileNotFoundError: raise FileNotFoundError( f"Simulation data not found at {DATA_PATH}. Check fetch script run." ) # Calculate Log Returns df['log_return'] = np.log(df['close'] / df['close'].shift(1)) # Clean up NA values created by shift(1) df.dropna(inplace=True) # Calculate Volatility (Sigma): Standard deviation of log returns # Sigma is the core parameter for the A-C stochastic noise volatility_sigma = df['log_return'].std().item() print(f"Market Volatility Sigma calculated: {volatility_sigma:.8f}") return df, volatility_sigma if __name__ == '__main__': try: # Unpack two return values: data_df, sigma_val = load_and_calculate_market_params() print(f"\nTotal Trading Steps Loaded: {len(data_df)}") print(f"Calculated Sigma: {sigma_val:.8f}") print(f"\nA-C Parameters:") for key, value in AC_PARAMS.items(): print(f" {key}: {value}") # Show sample price range for context print(f"\nPrice Range in Dataset:") print(f" Min: ${data_df['close'].min():.2f}") print(f" Max: ${data_df['close'].max():.2f}") print(f" First: ${data_df['close'].iloc[0]:.2f}") print(f" Last: ${data_df['close'].iloc[-1]:.2f}") except Exception as e: print(f"Validation failed: {e}") ``` [1 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]