# Project export: Nebula This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: 3d search for network graphs - Devpost: https://devpost.com/software/semantic-search-of-3d-graph-of-high-signal-social-network - GitHub: https://github.com/berkleyaihackathon2025DANI/working - Team: 2 GitHub contributor(s) — DK (2 commits), Nicolas Dickenmann (2 commits) ## Devpost submission (written by the team) ### Inspiration We believe that great work is accomplished when great talent unites. This was our attempt at mapping the academic landscape to hopefully encourage more collaboration within academia. Heavily inspired by Bell Labs. What you learned Learned how to work with the unpkg open source 3d graph visualizer and edit features such as a lighting up specific nodes or connections. Also learned how to build a great dataset from scratch scrapping directly from Google Scholar. Challenges A large challenge was pulling the whole project together, getting our vectordb results to light up related nodes. Another challenge was the slow speed of scrapping Google Scholar IDs, we spent at least 4 hours running scripts just to collect data. ## README (from the GitHub repository) # Author-Co-Author Network Navigator This project provides a web-based interface to explore a network of academic authors and their co-authors. It features a semantic search engine to find authors based on their research abstracts and a 3D force-directed graph to visualize the co-authorship network. ## Features * **Semantic Search**: Find authors based on natural language queries related to their research. * **AI-Powered Explanations**: Get AI-generated explanations of why an author is a good match for your search query. * **3D Network Visualization**: Explore the co-authorship network in an interactive 3D graph. * **Flask-Based API**: A simple and extensible API for search and data retrieval. ## Prerequisites Before you begin, ensure you have the following installed: * Python 3.7+ * pip (Python package installer) ## Setup 1. **Clone the repository:** ```bash git clone cd ``` 2. **Install dependencies:** ```bash pip install -r requirements.txt ``` 3. **Configure API Key:** * Rename the `config.env.example` file to `config.env`. * Open `config.env` and add your Google Gemini API key: ``` GOOGLE_API_KEY=your_google_api_key ``` ## Data Preparation The search engine and graph visualization rely on pre-processed data files. 1. **Input Data:** The primary data source is a JSON file containing author information, including abstracts and co-author relationships. The project expects this file at `nicolasdata/author_abstracts_5.json`. 2. **Generate Vector Database:** Run the `embedding_database.py` script to create the vector database from your input data. This will generate the `static/vectorbig.json` file by default. ```bash python embedding_database.py load nicolasdata/author_abstracts_5.json ``` You can specify a different path for the database: ```bash python embedding_database.py --db /path/to/your/vector_database.json load nicolasdata/author_abstracts_5.json ``` 3. **Generate Force Graph Data:** Run the `convert_author_abstracts_4_to_graph.py` script to create the data for the 3D visualization. This will generate the `static/forcegraph_data_3.json` file. Provide the input and output file paths as arguments. ```bash python convert_author_abstracts_4_to_graph.py nicolasdata/author_abstracts_5.json static/forcegraph_data_3.json ``` ## Running the Application Once the data preparation is complete, you can start the web server: ```bash python search_api.py ``` By default, the server runs on `0.0.0.0:5000` and uses the vector database at `static/vectorbig.json`. You can change the host, port, and database path using command-line arguments: ```bash python search_api.py --host 127.0.0.1 --port 8080 --db /path/to/your/vector_database.json ``` The application will be available at `http://:`. * The 3D graph visualization will be at the root URL: `http://:/` * The search API is available at the `/search` endpoint. ## API Endpoints ### `/search` * **Method**: `POST` * **Description**: Performs a semantic search for authors based on a query. * **Body**: ```json { "query": "your search query here" } ``` * **Response**: A JSON object with a list of matching authors. ### `/explain_match` * **Method**: `POST` * **Description**: Generates an AI-powered explanation for why an author matches a search query. * **Body**: ```json { "query": "your search query here", "author_id": "author_id_from_search_results" } ``` * **Response**: A JSON object with the explanation text. ## Detected evidence (automated analysis) Indexed codebase: 15 recognized source files, 281 KB. - Flask (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (32 of 32) ``` .DS_Store config.env.example convert_author_abstracts_4_to_graph.py datascripts/.DS_Store datascripts/convert_author_abstracts.py datascripts/convert_to_forcegraph.py datascripts/converted_author_data.json datascripts/merge_and_update.py datascripts/merged_author_abstracts.json embedding_database.py fonts/.DS_Store fonts/JetBrainsMono-2.304/.DS_Store fonts/JetBrainsMono-2.304/AUTHORS.txt fonts/JetBrainsMono-2.304/OFL.txt initial faculty/.DS_Store initial faculty/bing_scholar_finder.py initial faculty/enriched_professors_data.json initial faculty/extract_faculty_pages.py initial faculty/extract_professors.py initial faculty/faculty_page_0.html initial faculty/faculty_page_1.html initial faculty/faculty_page_2.html initial faculty/faculty_page_3.html initial faculty/faculty_professors.json nicolasdata/author_abstracts_5.json README.md requirements.txt search_api.py static/.DS_Store static/force_graph.html static/forcegraph_data_3.json static/vectorbig.json ``` ### Dependencies - requirements.txt: beautifulsoup4@==4.12.2, fake-useragent@==1.4.0, flask@==3.1.1, flask-cors@==6.0.1, google-generativeai@==0.8.5, numpy@==2.3.1, python-dotenv@==1.1.0, requests@==2.31.0, selenium@==4.15.2, undetected-chromedriver@==3.5.4 ### Recent commits (newest first) - cleanup - fix:API key and add readme - Implement graph reload on clear, UI/UX improvements, and Gemini explanations - UI/UX: Sidebar result cards, Gemini explanations, and CORS/debug improvements - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` requests==2.31.0 beautifulsoup4==4.12.2 undetected-chromedriver==3.5.4 selenium==4.15.2 fake-useragent==1.4.0 flask==3.1.1 flask-cors==6.0.1 python-dotenv==1.1.0 google-generativeai==0.8.5 numpy==2.3.1 ``` ### convert_author_abstracts_4_to_graph.py ```python #!/usr/bin/env python3 """ Convert author_abstracts_4.json to force graph format Creates nodes and links for 3D force-directed graph visualization """ import json import os def convert_author_abstracts_4_to_graph(input_file, output_file): """ Convert author_abstracts_4.json to force graph format Args: input_file (str): Path to author_abstracts_4.json output_file (str): Path to output force graph JSON """ print(f"📖 Loading data from {input_file}...") # Load the author abstracts data with open(input_file, 'r') as f: data = json.load(f) # Extract components author_names = data.get('author_names', {}) co_authors = data.get('co_authors', {}) author_abstracts = data.get('author_abstracts', {}) author_levels = data.get('author_levels', {}) summary = data.get('summary', {}) print(f"📊 Data summary:") print(f" - Total authors: {summary.get('total_authors', 0)}") print(f" - Input authors: {summary.get('input_authors_count', 0)}") print(f" - Direct co-authors: {summary.get('direct_co_authors_count', 0)}") print(f" - Second level co-authors: {summary.get('second_level_co_authors_count', 0)}") print(f" - Authors with abstracts: {summary.get('authors_with_abstracts', 0)}") print(f" - Total abstracts: {summary.get('total_abstracts', 0)}") # Build nodes print("\n🔨 Building nodes...") nodes = [] id_to_node = {} for author_id, author_name in author_names.items(): # Get research papers for this author papers = author_abstracts.get(author_id, []) # Get top 3 most recent papers sorted_papers = sorted(papers, key=lambda x: x.get('year', 0), reverse=True)[:3] # Format papers for display formatted_papers = [] for paper in sorted_papers: formatted_papers.append({ "title": paper.get('title', ''), "abstract": paper.get('abstract', ''), "year": paper.get('year', 0), "authors": paper.get('authors', '') }) # Determine author level for styling author_level = "unknown" if author_id in author_levels.get('input_authors', []): author_level = "input" elif author_id in author_levels.get('direct_co_authors', []): author_level = "direct" elif author_id in author_levels.get('second_level_co_authors', []): author_level = "second" # Create node node = { "id": author_id, "name": author_name, "level": author_level, "papers": formatted_papers, "paper_count": len(papers) } nodes.append(node) id_to_node[author_id] = node print(f"✅ Created {len(nodes)} nodes") # Build links (co-authorship connections) print("\n🔗 Building links...") links_set = set() # To avoid duplicate links links = [] for author_id, connections in co_authors.items(): # Only process if this author exists in our nodes if author_id not in id_to_node: continue for target_id in connections: # Only add link if both nodes exist if target_id in id_to_node: # Use tuple with sorted ids to deduplicate undirected links link_tuple = tuple(sorted([author_id, target_id])) if link_tuple not in links_set: links_set.add(link_tuple) links.append({ "source": link_tuple[0], "target": link_tuple[1] }) print(f"✅ Created {len(links)} links") # Create the force graph data structure force_graph_data = { "nodes": nodes, "links": links, "metadata": { "source_file": input_file, "total_authors": len(nodes), "total_connections": len(links), "summary": summary } } # Save to output file print(f"\n💾 Saving to {output_file}...") with open(output_file, 'w') as f: json.dump(force_graph_data, f, indent=2) print(f"✅ Successfully converted to force graph format!") print(f"📁 Output saved to: {output_file}") # Print some statistics print(f"\n📊 Final graph statistics:") print(f" - Nodes: {len(nodes)}") print(f" - Links: {len(links)}") print(f" - Input authors: {len([n for n in nodes if n['level'] == 'input'])}") print(f" - Direct co-authors: {len([n for n in nodes if n['level'] == 'direct'])}") print(f" - Second level co-authors: {len([n for n in nodes if n['level'] == 'second'])}") return force_graph_data def main(): """Main function""" print("🚀 Converting author_abstracts_5.json to force graph format") print("=" * 60) # File paths input_file = "nicolasdata/author_abstracts_5.json" output_file = "static/forcegraph_data_3.json" # Check if input file exists if not os.path.exists(input_file): print(f"❌ Input file not found: {input_file}") return # Convert the data try: force_graph_data = convert_author_abstracts_4_to_graph(input_file, output_file) print(f"\n🎉 Conversion complete!") print(f"💡 You can now use {output_file} with your 3D force graph visualization") except Exception as e: print(f"❌ Error during conversion: {e}") return if __name__ == "__main__": main() ``` ### search_api.py ```python from flask import Flask, request, jsonify, send_from_directory from flask_cors import CORS, cross_origin import google.generativeai as genai import numpy as np import json import os from dotenv import load_dotenv # Load environment variables from config.env load_dotenv('config.env') app = Flask(__name__) CORS(app) # Enable CORS for all routes # Configure Gemini API from environment GOOGLE_API_KEY = os.getenv('GOOGLE_API_KEY') if not GOOGLE_API_KEY: raise ValueError("GOOGLE_API_KEY not found in config.env") genai.configure(api_key=GOOGLE_API_KEY) EMBEDDING_MODEL = 'embedding-001' # Load vector database def load_vector_database(): try: with open('static/vectorbig.json', 'r') as f: return json.load(f) except Exception as e: print(f"Error loading vector database: {e}") return [] vector_database = load_vector_database() print(f"Loaded {len(vector_database)} entries from vector database") def get_embedding(text): """Generate embedding for text using Gemini API""" try: result = genai.embed_content( model=f"models/{EMBEDDING_MODEL}", content=text, task_type="RETRIEVAL_QUERY" ) return result['embedding'] except Exception as e: print(f"Error getting embedding: {e}") return None def cosine_similarity(vec_a, vec_b): """Calculate cosine similarity between two vectors""" vec_a = np.array(vec_a) vec_b = np.array(vec_b) dot_product = np.dot(vec_a, vec_b) norm_a = np.linalg.norm(vec_a) norm_b = np.linalg.norm(vec_b) if norm_a == 0 or norm_b == 0: return 0.0 return dot_product / (norm_a * norm_b) # Add this helper function to collect all texts for an author def get_author_texts(author_id): texts = [] for item in vector_database: if author_id in item['author_ids']: texts.append(item['text']) return texts @app.route('/search', methods=['POST']) def search(): """Search endpoint that performs semantic search""" try: data = request.get_json() query = data.get('query', '').strip() if not query: return jsonify({'error': 'Query is required'}), 400 print(f"Searching for: {query}") # Generate embedding for the query query_embedding = get_embedding(query) if not query_embedding: return jsonify({'error': 'Failed to generate query embedding'}), 500 # Calculate similarities with all database entries results = [] for item in vector_database: similarity = cosine_similarity(query_embedding, item['vector']) if similarity > 0.1: # Only include results with similarity > 0.1 for author_id in item['author_ids']: results.append({ 'author_id': author_id, 'similarity': similarity, 'text': item['text'][:200] + '...' if len(item['text']) > 200 else item['text'] }) # Group by author_id and take the highest similarity for each author author_results = {} for result in results: author_id = result['author_id'] if author_id not in author_results or result['similarity'] > author_results[author_id]['similarity']: author_results[author_id] = result # Convert back to list and sort by similarity final_results = list(author_results.values()) final_results.sort(key=lambda x: x['similarity'], reverse=True) # Take top 20 results final_results = final_results[:200] print(f"Found {len(final_results)} results") return jsonify({ 'query': query, 'results': final_results, 'total_found': len(final_results) }) except Exception as e: print(f"Error in search: {e}") return jsonify({'error': str(e)}), 500 @app.route('/health', methods=['GET']) def health(): """Health check endpoint""" return jsonify({ 'status': 'healthy', 'database_entries': len(vector_database) }) @app.route('/') def serve_force_graph(): return send_from_directory('static', 'force_graph.html') @app.route('/explain_match', methods=['POST', 'OPTIONS']) @cross_origin(origins="*") def explain_match(): if request.method == 'OPTIONS': return '', 200 data = request.json query = data.get('query') author_id = data.get('author_id') # Gather all research texts for this author author_texts = get_author_texts(author_id) if not author_texts: return jsonify({'explanation': "No research texts found for this professor."}) # Compose a prompt for Gemini prompt = ( "You are an expert academic assistant. Your output will be shown on a card for a specific professor. " "Always make an effort to connect the user's search query to the professor's research interests, even if the connection is not obvious. " "Be creative and imaginative in finding possible links between the search and the research. " "Do NOT simply say there is no similarity; instead, try to find any plausible or tangential connection. " "Only explain why THIS professor matches the user's search, quoting or paraphrasing relevant research below. " "Do NOT suggest searching for other professors or topics. " "Be friendly, helpful, and use first or second person (e.g., 'You might be interested in this professor's work...'). " f"\n\nUser's search: '{query}'\n" f"Professor's research abstracts:\n" + "\n---\n".join(author_texts[:5]) + "\n\nIn 2-3 sentences, explain to the user why this professor matches their search, quoting or paraphrasing relevant research." ) # Call Gemini try: model = genai.Genera [truncated — 506 more characters] ``` ### embedding_database.py ```python import google.generativeai as genai import numpy as np import os import json # Using json for saving/loading the database # --- Configuration and Setup --- # Configure the Gemini API key. # It's best practice to use an environment variable. # If the environment variable is not set, you can paste your key directly. try: # Get the API key from an environment variable GOOGLE_API_KEY = os.getenv('GOOGLE_API_KEY') genai.configure(api_key=GOOGLE_API_KEY) except KeyError: print("----------------------------------------------------------------------") print("API Key not found in environment variables.") print("Please set the GOOGLE_API_KEY environment variable.") # Or uncomment the next line and paste your key. # genai.configure(api_key="YOUR_API_KEY") # For now, we will exit if no key is found. exit("Exiting: No API key configured.") print("----------------------------------------------------------------------") # The embedding model to use EMBEDDING_MODEL = 'embedding-001' DB_FILE_PATH = 'vectorbig.json' # --- Core Functions --- def get_embedding(text): """ Generates an embedding for the given text using the Gemini API. Args: text (str): The text to embed. Returns: list[float]: The embedding vector, or None if an error occurs. """ try: # The API handles requests where the text is too long by chunking it. # Here we assume the text fits within the model's context window for simplicity. result = genai.embed_content( model=f"models/{EMBEDDING_MODEL}", content=text, task_type="RETRIEVAL_DOCUMENT" # Use 'RETRIEVAL_DOCUMENT' for items in DB # and 'RETRIEVAL_QUERY' for search queries. ) return result['embedding'] except Exception as e: print(f"Error getting embedding: {e}") return None def cosine_similarity(vec_a, vec_b): """ Calculates the cosine similarity between two vectors. Args: vec_a (np.array): The first vector. vec_b (np.array): The second vector. Returns: float: The cosine similarity score. """ # Ensure vectors are numpy arrays for efficient calculation vec_a = np.array(vec_a) vec_b = np.array(vec_b) dot_product = np.dot(vec_a, vec_b) norm_a = np.linalg.norm(vec_a) norm_b = np.linalg.norm(vec_b) # Avoid division by zero if norm_a == 0 or norm_b == 0: return 0.0 return dot_product / (norm_a * norm_b) # --- Database Management --- def load_database(): """Loads the vector database from a JSON file.""" if os.path.exists(DB_FILE_PATH): with open(DB_FILE_PATH, 'r') as f: return json.load(f) return [] # Return an empty list if the file doesn't exist def save_database(db): """Saves the vector database to a JSON file.""" with open(DB_FILE_PATH, 'w') as f: json.dump(db, f, indent=4) print(f"\nDatabase saved to {DB_FILE_PATH}") def find_existing_chunk(db, text): """ Finds if a chunk with the same text already exists in the database. Args: db (list): The database text (str): The text to search for Returns: int: Index of the existing chunk, or -1 if not found """ for i, item in enumerate(db): if item['text'] == text: return i return -1 def add_text_to_db(db, text, author_id): """ Adds text and its embedding to the database, handling duplicates. Args: db (list): The database text (str): The text to add author_id (str): The author ID to associate with this text """ # Check if this exact text already exists existing_index = find_existing_chunk(db, text) if existing_index >= 0: # Text already exists, just add the new author_id to the existing entry if author_id not in db[existing_index]['author_ids']: db[existing_index]['author_ids'].append(author_id) print(f"Added author ID {author_id} to existing chunk.") save_database(db) else: print(f"Author ID {author_id} already associated with this chunk.") else: # New text, create embedding and add to database print(f"\nGetting embedding for: '{text[:100]}...'...") embedding = get_embedding(text) if embedding: db.append({ 'text': text, 'vector': embedding, 'author_ids': [author_id] }) print("New chunk added to the database.") save_database(db) else: print("Failed to add text to the database.") def load_author_abstracts_from_json(json_file_path): """ Loads author abstracts from a JSON file and returns the data. Args: json_file_path (str): Path to the JSON file Returns: dict: The loaded JSON data """ try: with open(json_file_path, 'r') as f: data = json.load(f) return data except Exception as e: print(f"Error loading JSON file: {e}") return None def process_author_abstracts(json_file_path): """ Processes author abstracts from JSON file and adds them to the database. Args: json_file_path (str): Path to the JSON file containing author abstracts """ print(f"Loading author abstracts from {json_file_path}...") data = load_author_abstracts_from_json(json_file_path) if not data or 'author_abstracts' not in data: print("No author_abstracts found in the JSON file.") return database = load_database() author_abstracts = data['author_abstracts'] total_papers = sum(len(papers) for papers in author_abstracts.values()) processed = 0 print(f"Found {len(author_abstracts)} authors with {total_papers} total papers.") for aut [truncated — 4992 more characters] ``` ### datascripts/convert_to_forcegraph.py ```python import json def convert_to_forcegraph(): # Load the converted author data with open("converted_author_data.json", "r") as f: authors = json.load(f) # Build nodes nodes = [] id_to_node = {} for author in authors: node = { "id": author["id"], "name": author["name"], # Optionally add more fields here (e.g., research_papers) } nodes.append(node) id_to_node[author["id"]] = node # Build links (deduplicate, undirected) links_set = set() links = [] for author in authors: source_id = author["id"] for target_id in author.get("connections", []): # Only add link if both nodes exist if target_id in id_to_node: # Use tuple with sorted ids to deduplicate undirected links link_tuple = tuple(sorted([source_id, target_id])) if link_tuple not in links_set: links_set.add(link_tuple) links.append({"source": link_tuple[0], "target": link_tuple[1]}) # Output out = {"nodes": nodes, "links": links} with open("forcegraph_data.json", "w") as f: json.dump(out, f, indent=2) print(f"✅ Wrote {len(nodes)} nodes and {len(links)} links to forcegraph_data.json") if __name__ == "__main__": convert_to_forcegraph() ``` ### datascripts/convert_author_abstracts.py ```python import json def convert_author_abstracts(): """Convert author_abstracts.json to a new format with id, name, 3 research papers, and connections""" # Load the original data with open("nicolasdata/author_abstracts_4.json", "r") as f: data = json.load(f) # Extract the components all_authors = data.get("all_authors", []) author_names = data.get("author_names", {}) co_authors = data.get("co_authors", {}) author_abstracts = data.get("author_abstracts", {}) # Convert to new format converted_data = [] for author_id in all_authors: author_name = author_names.get(author_id, "Unknown") # Get research papers from author_abstracts papers = author_abstracts.get(author_id, []) # Sort by year (most recent first), then take top 3 sorted_papers = sorted(papers, key=lambda x: x.get("year", 0), reverse=True)[:3] # Format papers formatted_papers = [] for paper in sorted_papers: formatted_papers.append({ "title": paper.get("title", ""), "abstract": paper.get("abstract", ""), "year": paper.get("year", 0), "authors": paper.get("authors", "") }) # Get connections (co-authors) connections = co_authors.get(author_id, []) # Create the new entry entry = { "id": author_id, "name": author_name, "research_papers": formatted_papers, "connections": connections } converted_data.append(entry) # Save the converted data output_file = "fornicolas.json" with open(output_file, "w") as f: json.dump(converted_data, f, indent=2) print(f"✅ Converted {len(converted_data)} authors") print(f"📊 Output saved to: {output_file}") # Show a sample entry if converted_data: print("\n📋 Sample entry:") sample = converted_data[0] print(f"ID: {sample['id']}") print(f"Name: {sample['name']}") print(f"Papers: {len(sample['research_papers'])}") print(f"Connections: {len(sample['connections'])}") if sample['research_papers']: print(f"First paper: {sample['research_papers'][0]['title'][:50]}...") if __name__ == "__main__": convert_author_abstracts() ``` ### initial faculty/extract_professors.py ```python from bs4 import BeautifulSoup import json import re def extract_professors_from_html(html_file): """ Extract professor names and interests from the HTML file. Returns a list of dictionaries with 'name' and 'interests' keys. """ # Read the HTML file with open(html_file, 'r', encoding='utf-8') as f: html_content = f.read() # Parse HTML soup = BeautifulSoup(html_content, 'html.parser') # Find all professor articles professor_articles = soup.find_all('article', class_='node--type-faculty') professors = [] for article in professor_articles: # Extract name name_element = article.find('span', class_='field--name-title') if name_element: name = name_element.text.strip() else: continue # Skip if no name found # Extract interests interests_element = article.find('div', class_='field--name-field-areas-of-expertise') interests = [] if interests_element: # Find all interest links interest_links = interests_element.find_all('a') for link in interest_links: interest = link.text.strip() if interest: # Only add non-empty interests interests.append(interest) # Create professor entry professor = { 'name': name, 'interests': interests } professors.append(professor) return professors def save_professors_to_file(professors, output_file): """ Save professors data to a file in JSON format. """ with open(output_file, 'w', encoding='utf-8') as f: json.dump(professors, f, indent=2, ensure_ascii=False) print(f"Saved {len(professors)} professors to {output_file}") def save_professors_to_python_array(professors, output_file): """ Save professors data to a Python file as an array. """ with open(output_file, 'w', encoding='utf-8') as f: f.write("# Professors data extracted from batch1.html\n") f.write("professors = [\n") for i, professor in enumerate(professors): f.write(" {\n") f.write(f' "name": "{professor["name"]}",\n') f.write(' "interests": [\n') for j, interest in enumerate(professor["interests"]): if j == len(professor["interests"]) - 1: f.write(f' "{interest}"\n') else: f.write(f' "{interest}",\n') if i == len(professors) - 1: f.write(" ]\n") f.write(" }\n") else: f.write(" ]\n") f.write(" },\n") f.write("]\n") f.write(f"\n# Total professors: {len(professors)}\n") print(f"Saved {len(professors)} professors to {output_file}") def print_summary(professors): """ Print a summary of the extracted data. """ print(f"\n=== SUMMARY ===") print(f"Total professors found: {len(professors)}") # Count total interests total_interests = sum(len(prof['interests']) for prof in professors) print(f"Total interests found: {total_interests}") # Show first few professors as example print(f"\n=== FIRST 3 PROFESSORS ===") for i, professor in enumerate(professors[:3]): print(f"{i+1}. {professor['name']}") print(f" Interests: {', '.join(professor['interests'])}") print() def main(): # Extract professors from batch1.html print("Extracting professors from batch1.html...") professors = extract_professors_from_html('batch1.html') if not professors: print("No professors found in the HTML file!") return # Print summary print_summary(professors) # Save to JSON file save_professors_to_file(professors, 'professors_data.json') # Save to Python array file save_professors_to_python_array(professors, 'professors_array.py') print("\n=== EXTRACTION COMPLETE ===") print("Files created:") print("- professors_data.json (JSON format)") print("- professors_array.py (Python array format)") if __name__ == "__main__": main() ``` ### initial faculty/extract_faculty_pages.py ```python from bs4 import BeautifulSoup import json import glob import os def extract_professors_from_html(html_file): """ Extract professor names and interests from a faculty page HTML file. Returns a list of dictionaries with 'name' and 'interests' keys. """ # Read the HTML file with open(html_file, 'r', encoding='utf-8') as f: html_content = f.read() # Parse HTML soup = BeautifulSoup(html_content, 'html.parser') # Find all professor articles professor_articles = soup.find_all('article', class_='node--type-faculty') professors = [] for article in professor_articles: # Extract name name_element = article.find('span', class_='field--name-title') if name_element: name = name_element.text.strip() else: continue # Skip if no name found # Extract interests interests_element = article.find('div', class_='field--name-field-areas-of-expertise') interests = [] if interests_element: # Find all interest links interest_links = interests_element.find_all('a') for link in interest_links: interest = link.text.strip() if interest: # Only add non-empty interests interests.append(interest) # Create professor entry professor = { 'name': name, 'interests': interests } professors.append(professor) return professors def process_all_faculty_pages(): """ Process all faculty_page HTML files and extract professors. """ # Find all faculty page HTML files faculty_files = glob.glob('faculty_page_*.html') faculty_files.sort() # Sort to process in order if not faculty_files: print("No faculty_page HTML files found!") return print(f"Found {len(faculty_files)} faculty page files: {faculty_files}") all_professors = [] total_new = 0 # Process each faculty page for faculty_file in faculty_files: print(f"Processing {faculty_file}...") # Extract professors from the file professors = extract_professors_from_html(faculty_file) print(f"Found {len(professors)} professors in {faculty_file}") # Create a set of existing professor names for quick lookup existing_names = {prof['name'].lower() for prof in all_professors} # Filter out duplicates (based on name) unique_professors = [] duplicates = 0 for prof in professors: if prof['name'].lower() not in existing_names: unique_professors.append(prof) else: duplicates += 1 if duplicates > 0: print(f"Skipped {duplicates} duplicate professors") # Add unique professors to the main list all_professors.extend(unique_professors) total_new += len(unique_professors) return all_professors, total_new def save_professors_to_json(professors, output_file): """ Save professors data to a JSON file. """ with open(output_file, 'w', encoding='utf-8') as f: json.dump(professors, f, indent=2, ensure_ascii=False) print(f"Saved {len(professors)} professors to {output_file}") def print_summary(professors): """ Print a summary of the extracted data. """ print(f"\n=== SUMMARY ===") print(f"Total professors: {len(professors)}") # Count total interests total_interests = sum(len(prof['interests']) for prof in professors) print(f"Total interests: {total_interests}") # Show first few professors as example print(f"\n=== FIRST 5 PROFESSORS ===") for i, professor in enumerate(professors[:5]): print(f"{i+1}. {professor['name']}") print(f" Interests: {', '.join(professor['interests'])}") print() def main(): # Process all faculty pages print("Extracting professors from faculty page HTML files...") all_professors, total_new = process_all_faculty_pages() if not all_professors: print("No professors found in the HTML files!") return # Save to JSON file save_professors_to_json(all_professors, 'faculty_professors.json') # Print summary print_summary(all_professors) print("\n=== EXTRACTION COMPLETE ===") print(f"Total unique professors: {len(all_professors)}") print(f"File created: faculty_professors.json") if __name__ == "__main__": main() ``` [5 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]