# Project export: LLM Stock Trading Benchmark This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Benchmark LLM performance at trading stocks by reacting to news with a quick, easy to use UI - Devpost: https://devpost.com/software/llm-stock-trading-benchmark - GitHub: https://github.com/vkethana/calhacks-24/ - Video: https://www.youtube.com/embed/_B3GyWRfAzw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Mark Ogata (23 commits), Vijay Kethanaboyina (12 commits), James Dinh (9 commits) ## Devpost submission (written by the team) ### Overview The key to improving Large Language Models (LLMs) in specific domains lies in comprehensive, high-quality evaluations. Our project, the LLM Stock Trader, addresses this need in the financial sector. By creating a tool for evaluating LLMs' stock trading capabilities, we're paving the way for future research into making language agents better at stock trading. Our tool allows LLMs to simulate buying and selling stocks. Users can: Select custom start and end dates for trading simulations Watch as the AI makes real-time buying and selling decisions based on market data and newspaper headlines Analyze the AI's performance and decision-making process For each day in the specified time range, the LLM receives its current portfolio and crucial news headlines. Then, we tell the LLM to output a JSON of the trades it wants to make. For each trade, it also outputs a justification for why it made the trade (chain of thought prompting). This makes the model less likely to hallucinate nonexistent stock tickers or make poor decisions, because it's forced to explain out loud why it made a given decision. API Rate Limits: Groq: Optimized our trading bot's efficiency to reduce API calls and avoid rate limits yfinance: Implemented local caching of stock data to minimize repeated downloads and avoid yfinance's rate limits Frontend Development: Tackled numerous UI bugs, which tested our patience in debugging very precise issues. Debugging front end bugs took longer than any other portion of the project. ## README (from the GitHub repository) # calhacks-24 2024 Calhacks Submission ## Detected evidence (automated analysis) Indexed codebase: 50 recognized source files, 698 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (63 of 63) ``` .gitignore app.py checkyfinance.py datafetch_2.py datafetch.py datascraper_3.py drudgereport_archives/drudgereport_20230117.html drudgereport_archives/drudgereport_20230118.html drudgereport_archives/drudgereport_20230119.html drudgereport_archives/drudgereport_20230120.html drudgereport_archives/drudgereport_20230121.html drudgereport_archives/drudgereport_20230122.html drudgereport_archives/drudgereport_20230123.html drudgereport_archives/drudgereport_20230124.html drudgereport_archives/drudgereport_20230125.html drudgereport_archives/drudgereport_20230126.html drudgereport_archives/drudgereport_20230127.html drudgereport_archives/drudgereport_20230128.html drudgereport_archives/drudgereport_20230129.html drudgereport_archives/drudgereport_20230130.html drudgereport_archives/drudgereport_20230131.html drudgereport_archives/drudgereport_20230201.html drudgereport_archives/drudgereport_20230202.html drudgereport_archives/drudgereport_20230203.html drudgereport_archives/drudgereport_20230204.html eval_model_test.py eval_model.py fetch_headlines.py frontend/.gitignore frontend/package.json frontend/public/index.html frontend/public/manifest.json frontend/public/robots.txt frontend/README.md frontend/src/App.test.tsx frontend/src/App.tsx frontend/src/curr-leader.tsx frontend/src/dashboard.tsx frontend/src/headlines.tsx frontend/src/index.css frontend/src/index.tsx frontend/src/leaderboard.tsx frontend/src/logtable.tsx frontend/src/react-app-env.d.ts frontend/src/reportWebVitals.ts frontend/src/setupTests.ts frontend/src/styles/general.scss frontend/src/timeline.tsx frontend/src/util/functions.ts frontend/tsconfig.json LICENSE llm_trader.py prompts.txt README.md realtimeTest.ipynb static/script.js static/style.css templates/index.html test/AAPL_stock_prices_dec_2023.csv test/getTestData.py utils.py wsj_2022_headlines.csv wsj_2022_headlines.json ``` ### Dependencies - frontend/package.json: @emotion/react@^11.13.3, @emotion/styled@^11.13.0, @fortawesome/fontawesome-svg-core@^6.6.0, @fortawesome/free-solid-svg-icons@^6.6.0, @fortawesome/react-fontawesome@^0.2.2, @mui/material@^6.1.4, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @types/jest@^27.5.2, @types/node@^16.18.114, @types/react@^18.3.11, @types/react-dom@^18.3.1, axios@^1.7.7, cors@^2.8.5, react@^18.3.1, react-dom@^18.3.1, react-scripts@5.0.1, sass@^1.80.3, typescript@^4.9.5, web-vitals@^2.1.4 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/vkethana/calhacks-24 - fixed styles - fixed styles - reduced api call load - added percent change - Merge branch 'main' of https://github.com/vkethana/calhacks-24 - fix rankings - fix - Merge branch 'main' of https://github.com/vkethana/calhacks-24 - fix - Merge branch 'main' of https://github.com/vkethana/calhacks-24 - package lock - fasdf - Merge branch 'main' of https://github.com/vkethana/calhacks-24 - . - sadf - fix: bug - fix: date - fix: styles - fix: bug ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/package.json ``` { "name": "calhacks-24", "version": "0.1.0", "private": true, "dependencies": { "@emotion/react": "^11.13.3", "@emotion/styled": "^11.13.0", "@fortawesome/fontawesome-svg-core": "^6.6.0", "@fortawesome/free-solid-svg-icons": "^6.6.0", "@fortawesome/react-fontawesome": "^0.2.2", "@mui/material": "^6.1.4", "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "@types/jest": "^27.5.2", "@types/node": "^16.18.114", "@types/react": "^18.3.11", "@types/react-dom": "^18.3.1", "axios": "^1.7.7", "cors": "^2.8.5", "react": "^18.3.1", "react-dom": "^18.3.1", "react-scripts": "5.0.1", "sass": "^1.80.3", "typescript": "^4.9.5", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### app.py ```python from flask import Flask, render_template, request, jsonify from datetime import datetime from flask_cors import CORS, cross_origin from utils import trade from eval_model import calculate_pnl_with_real_data from llm_trader import run_llm_trade import json app = Flask(__name__) CORS(app, resources={r"/*": {"origins": "*", "methods": ["GET", "POST", "OPTIONS"], "allow_headers": "*"}}) # Load news data from the JSON file with open('wsj_2022_headlines.json') as news_file: news_data = json.load(news_file) @app.route('/') def index(): return render_template('index.html') @app.route('/api/get_news', methods=['POST']) def get_news(): data = request.get_json() trading_date = data.get('trading_date') return jsonify(news_data.get(trading_date, ["ERROR: No news data for " + trading_date])) @cross_origin() @app.route('/api/get_trades_and_pnl', methods=['POST', 'OPTIONS']) def get_trades_and_pnl(): if request.method == 'OPTIONS': return '', 200 # Respond to preflight requests print("call to get value") data = request.get_json() trading_date = data.get('trading_date') evaluation_date = data.get('evaluation_date') print("TRADING DATE", trading_date) print("EVALUATION DATE", evaluation_date) # Run LLM to generate trades based on trading_date and headlines trades_data = run_llm_trade(trading_date, trading_history=None, headlines=news_data) print("LLM CHOICE", trades_data) # print("daddaaa", trades_data.keys()) # Convert incoming trade data to trade objects trades_list = [] for cur_trade in trades_data.get("trades", []): print("CUR TRADE", cur_trade) if cur_trade.get('action') is None: print("NO ACTION") continue timestamp = trading_date action = cur_trade.get('action') volume = cur_trade.get('volume') ticker = cur_trade.get('ticker', 'AAPL') # Default ticker symbol if not provided new_trade = trade(datetime.strptime(timestamp, '%Y-%m-%d'), action, volume, ticker) trades_list.append(new_trade) # Calculate the result based on the evaluation date and trades try: print("TRADES LIST", trades_list) pnl_result = calculate_pnl_with_real_data(trades_list, datetime.strptime(evaluation_date, '%Y-%m-%d')) print("PnL RESULT", pnl_result) print("TRADES DATA", trades_data) return jsonify(response={ 'trades': trades_data, 'result': pnl_result }, headers={ "Content-Type": "application/json", "Access-Control-Allow-Origin": '*', "Access-Control-Allow-Methods": 'PUT, GET, POST, DELETE, OPTIONS', "Access-Control-Allow-Headers": 'Origin, Accept, Content-Type, X-Requested-With, X-CSRF-Token' }, content_type='application/json') except ValueError as e: return jsonify({'error': str(e)}), 400 except Exception as e: print(f"Error: {e}") # Log the error return jsonify({'error': str(e)}), 500 # Return error response if __name__ == '__main__': app.run(debug=True) ``` ### frontend/src/index.tsx ```typescript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot( document.getElementById('root') as HTMLElement ); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### frontend/src/App.tsx ```typescript import React, { useMemo, useState, useEffect } from "react"; import "./styles/general.scss"; import Dashboard from "./dashboard"; import Timeline from "./timeline"; import { fetchApiTrades } from "./util/functions"; export interface Trade { action: string; volume: number; ticker: string; pnl: number; } export interface Agent { name: string; trades: Trade[]; pnl: number; } function App() { const [agents, setAgents] = useState([ { name: "llama-3.1-8b-instant", trades: [], pnl: 0, }, ]); const [trades, setTrades] = useState([]); const [pnl, setPnl] = useState(0); const [isLoading, setIsLoading] = useState(true); const [date, setDate] = useState("2022-01-01"); const [evalDate, setEvalDate] = useState("2022-01-01"); const [largestTrade, setLargestTrade] = useState({ action: "", volume: 0, ticker: "", pnl: 0, }); useEffect(() => { const fetchTrades = async () => { setIsLoading(true); const fetchedTrades = await fetchApiTrades(date, evalDate); // Get trades from API if (fetchedTrades && fetchedTrades.response.trades.trades) { console.log(fetchedTrades.response); setTrades((prevTrades) => [ ...prevTrades, ...fetchedTrades.response.trades.trades, ]); // Append fetched trades to the existing trades } if (fetchedTrades && fetchedTrades.response.result) { console.log(fetchedTrades.response.result); setPnl(fetchedTrades.response.result); } setIsLoading(false); }; fetchTrades(); }, [date, evalDate]); useEffect(() => { if (trades.length === 0) return; // Early return if there are no trades let updatedAgents = [...agents]; // Create a copy of the current agents trades.forEach((trade) => { // Check if there are any agents available if (updatedAgents[0]) { updatedAgents[0] = { ...updatedAgents[0], trades: [ ...(updatedAgents[0].trades || []), trade, // Append the current trade ], }; } if (trade.volume > largestTrade.volume) { setLargestTrade(trade); } }); setAgents(updatedAgents); // Update the agents state once }, [trades, largestTrade]); // Include 'agents' as a dependency if it can change useEffect(() => { if (!agents[0]) return; let updatedAgents = [...agents]; updatedAgents[0] = { ...updatedAgents[0], pnl: pnl, // Update the agent's pnl whenever pnl changes }; setAgents(updatedAgents); }, [pnl]); // Trigger update when pnl changes return (
); } export default App; ``` ### utils.py ```python from datetime import datetime class trade: def __init__(self, time, action, volume, ticker): #convert time to datetime object self.time = time self.action = action self.volume = volume self.ticker = ticker def __str__(self): return f"timestamp: {self.time}, action: {self.action}, volume: {self.volume} ticker: {self.ticker}" def __repr__(self): return f"timestamp: {self.time}, action: {self.action}, volume: {self.volume} ticker: {self.ticker}" ``` ### datafetch.py ```python import requests import os import requests def get_financial_articles(api_key): url = "https://api.nytimes.com/svc/search/v2/articlesearch.json" query = "finance OR stock market OR Federal Reserve OR economy OR inflation OR earnings OR mergers OR interest rates" params = { "q": query, "begin_date": "20230101", "end_date": "20231231", "fq": "news_desk:(\"Business\" \"Finance\" \"Economy\")", "api-key": api_key } response = requests.get(url, params=params) data = response.json() # Extract headlines from the response articles = [ { "headline": article['headline']['main'], "pub_date": article['pub_date'], "web_url": article['web_url'] } for article in data['response']['docs'] ] return articles # Example usage api_key = "YOUR_API_KEY" articles = get_financial_articles(api_key) print(articles) ``` ### datafetch_2.py ```python import requests import datetime import time import os def get_wayback_url(url, timestamp): return f"http://web.archive.org/web/{timestamp}/{url}" def download_page(url, output_path): response = requests.get(url) if response.status_code == 200: with open(output_path, 'w', encoding='utf-8') as f: f.write(response.text) print(f"Downloaded: {output_path}") else: print(f"Failed to download: {url}") def main(): target_url = "https://drudgereport.com/" output_dir = "drudgereport_archives" os.makedirs(output_dir, exist_ok=True) start_date = datetime.date(2023, 1, 17) end_date = datetime.date(2023, 12, 31) current_date = start_date while current_date <= end_date: timestamp = current_date.strftime("%Y%m%d") wayback_url = get_wayback_url(target_url, timestamp) output_path = os.path.join(output_dir, f"drudgereport_{timestamp}.html") download_page(wayback_url, output_path) current_date += datetime.timedelta(days=1) time.sleep(1) # Be respectful to the Wayback Machine's servers if __name__ == "__main__": main() ``` ### checkyfinance.py ```python import yfinance as yf from datetime import datetime def calculate_trade_profit(ticker, shares, buy_date, sell_date): # Convert date strings to datetime objects buy_date = datetime.strptime(buy_date, "%Y-%m-%d") sell_date = datetime.strptime(sell_date, "%Y-%m-%d") # Create a Ticker object stock = yf.Ticker(ticker) # Fetch historical data hist = stock.history(start=buy_date, end=sell_date) if hist.empty: return "Error: No data available for the specified date range." # Get buy and sell prices buy_price = hist['Close'][0] # First day's closing price sell_price = hist['Close'][-1] # Last day's closing price # Calculate profit/loss initial_investment = buy_price * shares final_value = sell_price * shares profit = final_value - initial_investment percent_return = (profit / initial_investment) * 100 return { "ticker": ticker, "shares": shares, "buy_date": buy_date.strftime("%Y-%m-%d"), "sell_date": sell_date.strftime("%Y-%m-%d"), "buy_price": round(buy_price, 2), "sell_price": round(sell_price, 2), "initial_investment": round(initial_investment, 2), "final_value": round(final_value, 2), "profit": round(profit, 2), "percent_return": round(percent_return, 2) } # Example usage result = calculate_trade_profit("AAPL", 100, "2019-10-18", "2020-11-09") print(result) ``` ### datascraper_3.py ```python import requests from bs4 import BeautifulSoup from datetime import datetime, timedelta import csv import time import os import json def get_headlines_for_date(date): url = f"https://www.wsj.com/news/archive/{date.strftime('%Y/%m/%d')}" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" } response = requests.get(url, headers=headers) soup = BeautifulSoup(response.content, 'html.parser') headlines = [] # Look for common HTML elements that might contain headlines for headline_elem in soup.find_all(['h2', 'h3', 'a']): # Check if the element has text and a certain length if headline_elem.text and len(headline_elem.text.strip()) > 20: headlines.append(headline_elem.text.strip()) # Remove duplicates and limit to top 5 return list(dict.fromkeys(headlines))[40:] def main(): start_date = datetime(2022, 1, 15) end_date = datetime(2022, 12, 31) current_date = start_date json_file = 'wsj_2022_headlines.json' # Load existing data if the file exists if os.path.exists(json_file): with open(json_file, 'r') as file: all_headlines = json.load(file) else: all_headlines = {} while current_date <= end_date: date_str = current_date.strftime('%Y-%m-%d') print(f"Scraping headlines for {date_str}") # Skip if we already have data for this date if date_str not in all_headlines: headlines = get_headlines_for_date(current_date) all_headlines[date_str] = headlines # Write to JSON file after each day with open(json_file, 'w') as file: json.dump(all_headlines, file, indent=4) current_date += timedelta(days=1) time.sleep(2) # Be respectful with request frequency print("Scraping completed.") if __name__ == "__main__": main() ``` ### eval_model_test.py ```python import unittest from unittest.mock import MagicMock from datetime import datetime import pandas as pd import yfinance as yf from eval_model import calculate_pnl from utils import * class TestCalculatePnL(unittest.TestCase): def setUp(self): # Define the ticker symbol and date range for downloading stock data ticker = "AAPL" start_date = "2023-12-01" end_date = "2023-12-07" # Download the stock data using yfinance self.stock_prices = [ ["2023-12-01", 100], ["2023-12-04", 105], ["2023-12-05", 102], ["2023-12-06", 110], ] # Create trades object self.trades_obj = trades(ticker="AAPL", startDate='2023-12-01', endDate='2023-12-07') self.trades_obj.trades.append(trade('2023-12-01 00:00:00', 'buy', 5)) self.trades_obj.trades.append(trade('2023-12-04 00:00:00', 'buy', 3)) self.trades_obj.trades.append(trade('2023-12-05 00:00:00', 'sell', 8)) def test_calculate_pnl_successful(self): endDate = '2023-12-06' # Expected result: Sell at 110, buy average price is ((10 * 100) + (5 * 102)) / 15 = 100.67 expected_pnl = 1 pnl = calculate_pnl(self.trades_obj, self.stock_prices, endDate) self.assertAlmostEqual(pnl, expected_pnl, places=2) def test_no_price_data_for_trade_date(self): endDate = '2023-12-06' # Modify the first trade's timestamp to an unavailable date self.trades_obj.trades[0].timestamp = '2023-12-10 00:00:00' with self.assertRaises(ValueError) as context: calculate_pnl(self.trades_obj, self.stock_prices, endDate) self.assertEqual(str(context.exception), "No stock price data available for 2023-12-10") def test_no_trades(self): endDate = '2023-12-06' # Empty trades list self.trades_obj.trades = [] pnl = calculate_pnl(self.trades_obj, self.stock_prices, endDate) self.assertEqual(pnl, 0) if __name__ == '__main__': unittest.main() ``` [39 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]