Project Info

ScraPy

This project did not submit a demo video on Devpost.

Inspiration

The inspiration for this project came from the need to efficiently send out research emails and collect professor email addresses. Instead of sending specific emails to each person, the idea was to streamline the process by asking for coffee chats, which seemed more effective.

What it does

ScraPy automates the process of sending out research emails and collecting professor emails. It uses web scraping techniques to gather email addresses and automates the email-sending process.

How we built it

Technology Stack: The project is built using Python. Web Scraping: Utilizes BeautifulSoup and requests libraries. Email Automation: Uses smtplib to automate email sending. Data Management: Stores collected emails in CSV files. Setup: Requires setting up a virtual environment, installing necessary packages, and configuring email settings.

Challenges we ran into

Dynamic Content: Handling websites that use JavaScript to load content dynamically. Anti-Scraping Measures: Overcoming measures put in place by websites to prevent scraping. Email Deliverability: Ensuring emails are not marked as spam.

Accomplishments we're proud of

Successfully automating the email-sending process. Efficiently scraping professor email addresses. Implementing error handling to manage various issues.

What we learned

Advanced web scraping techniques and ethical scraping practices. Automating tasks using Python. Handling and processing large datasets efficiently.

What's next

Implementing an AI agent to tailor emails specifically to each professor. Enhancing the scraping techniques to handle more complex websites. Adding more features to improve the overall functionality and efficiency of the tool. For more details, you can visit the README file.

ScraPy ☕📧

ScraPy is a Python-based tool designed to send out research emails efficiently. This repository contains scripts that automate the process of scraping data and sending emails. Instead of sending specific emails to each person, I've opted for a more streamlined approach by asking for coffee chats, believing it would be more effective. In the future, I intend to implement an AI agent that will scrape each professor's entry and tailor the emails specifically to them. 🤖✨

NOTE: Clicking on the "Wiki" section at the top of this repo will show you some of the results this project yielded.

Prerequisites ✅

Before you can run any scripts in this project, make sure you have the latest version of Python 3 installed.

Install Python 3: Follow the links above to download and install the latest versions.
Create a Virtual Environment:
- Navigate to your project directory:
```
cd path/to/your/project
```
- Create a virtual environment:
```
python -m venv .venv
```
- Activate the virtual environment:
  - On Windows:
    - Open PowerShell as Administrator and run:
```
Set-ExecutionPolicy Unrestricted
```
    - Then activate the virtual environment:
```
.\.venv\Scripts\activate
```
  - On macOS/Linux:
    - Activate the virtual environment using:
```
source .venv/bin/activate
```
Clone the Repository Inside the .venv Directory:
- After activating your virtual environment, clone the ScraPy repository:
- Navigate into the cloned repository:
```
cd ScraPy
```
Install Required Packages: You can install the required packages using pip:
```
pip install -r requirements.txt
```

Here are the required packages listed in requirements.txt:

beautifulsoup4==4.12.3
certifi==2024.6.2
charset-normalizer==3.3.2
et-xmlfile==1.1.0
idna==3.7
openpyxl==3.1.3
python-dotenv==1.0.1
requests==2.32.3
soupsieve==2.5
urllib3==2.2.1

Important File Setup 📋

Replace DUMMYDOC.xlsx: Ensure you replace DUMMYDOC.xlsx with an empty Excel workbook, replacing its current existence in the project folder.

Demo Run-through 🚀

Here's a step-by-step guide to running a script from the scraperMagic folder:

Navigate to the Project Directory:
```
cd scraperMagic
```
Run a Script: Assuming you have a script named example_script.py in the scraperMagic folder, you can run it as follows:
```
python example_script.py
```

Example: Running `emailer.py` ✉️

If you have a script named send_emails.py in the scraperMagic folder, you can run it to send out research emails. Make sure to configure any necessary environment variables or input files as required by the script.

Configure Gmail for emailer.py: To send emails from a personal Gmail address, you'll need to create an app password. Please read the instructions in the emailer.py script to set this up correctly.
Attaching Files: If you want to attach any files to the email, read through the corresponding code block in emailer.py and follow the comments for instructions on how to do this.

python emailer.py

Disclaimer ⚠️

I am not liable for any damage, spam, or liabilities this project may cause. This includes but is not limited to any unintended consequences of using this tool, such as your email being flagged as spam or any impact on your reputation.

I have chosen not to include extensive documentation exploring all features and scripts, as the level of abstraction was necessary to limit its usability to those with a specific purpose who have the capabilities to modify the script and use it for good. By using this tool, you acknowledge that you understand the risks involved and agree to take full responsibility for any actions taken as a result of using this project.

License 📄

This project is licensed under the CCC License - see the LICENSE file for details.

Contributing 🤝

Feel free to fork this repository and submit pull requests. For major changes, please open an issue first to discuss what you would like to change.

Analysis

Compare with all teams

View

Metric

No commits on this project resolved to a GitHub account.

Technology

Found in codeClaimed only

PythonIn code

1 of 1 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

23 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

3 Verified4 Code-supported4 Claimed only

Automated email sending via SMTPVerified
Uses smtplib to automate email sending
Claimed on readmehigh confidence
- scraperMagic/emailer.py:36 — send_email uses smtplib.SMTP with Gmail server, starttls, login, and sendmail to send emails
- scraperMagic/emailer.py:48 — send_emails_from_excel iterates recruits from an Excel sheet and calls send_email for each row
Duplicate name/email checking toolVerified
Handling and processing large datasets efficiently (supporting tool for data management)
Claimed on Devpostmedium confidence
- tools/check-duplicates.py:12 — identifies duplicate names/emails across combined sheets and interactively removes them, then saves an updated workbook
Web scraping of professor names and emails from university faculty pagesVerified
Uses web scraping techniques (BeautifulSoup and requests) to gather professor email addresses
Claimed on readmehigh confidence
- scraperMagic/fac-sci-scraper.py:22 — scrape_mcmaster_faculty uses requests.get and BeautifulSoup to parse faculty names from a university page
- scraperMagic/fac-sci-scraper.py:36 — scrape_email_from_profile extracts mailto: links from profile pages
- scraperMagic/fac-hum-scraper.py:1 — additional faculty-specific scraper present, same pattern
- scraperMagic/fac-soc-sci-scraper.py:1 — additional faculty-specific scraper present, same pattern
- scraperMagic/fac-bizscraper.py:1 — additional faculty-specific scraper present, same pattern
- scraperMagic/eng-scraper.py:1 — additional faculty-specific scraper present, same pattern
Data stored/read from Excel files (not CSV)Code-supported
Stores collected emails in CSV files; data management via CSV
Claimed on Devposthigh confidence
- scraperMagic/fac-sci-scraper.py:6 — openpyxl.load_workbook is used to read/write an .xlsx workbook, not CSV files, contradicting the CSV claim though data persistence itself is implemented
- excel-sheets/DUMMYDOC.xlsx — the actual data file present in the repo is an Excel workbook, not a CSV
- tools/check-duplicates.py:4 — duplicate-checking tool also reads/writes .xlsx via pandas.read_excel and ExcelWriter
Email deliverability / anti-spam measuresCode-supported
Ensuring emails are not marked as spam
Claimed on Devpostlow confidence
- scraperMagic/emailer.py:42 — a time.sleep(10) delay is added between emails, a minimal throttling measure that could relate to deliverability, but no other spam-avoidance logic exists
Environment/virtualenv based configuration setupCode-supported
Setup requires virtual environment, package installation, and configuring email settings
Claimed on readmemedium confidence
- requirements.txt — lists required packages (beautifulsoup4, requests, openpyxl, python-dotenv, etc.) matching the setup instructions
- scraperMagic/emailer.py:10 — load_dotenv() is called, indicating support for .env-based configuration, though sender_email/password are hardcoded placeholders in __main__ rather than pulled from env vars
Error handling to manage various issuesCode-supported
Implementing error handling to manage various issues
Claimed on Devpostmedium confidence
- scraperMagic/emailer.py:35 — try/except/finally around the SMTP send logic catches and prints exceptions
- scraperMagic/fac-bizscraper.py:1 — contains try/except constructs per grep match
AI agent to tailor emails to each professorClaimed only
What's next: implementing an AI agent to tailor emails specifically to each professor
Claimed on readmehigh confidence
Anti-scraping measure evasionClaimed only
Overcoming measures put in place by websites to prevent scraping
Claimed on Devposthigh confidence
Coffee-chat style outreach email contentClaimed only
Streamlines outreach by asking for coffee chats instead of highly specific emails
Claimed on readmemedium confidence
Handling dynamic (JavaScript-rendered) website contentClaimed only
Challenges included handling websites that use JavaScript to load content dynamically
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.

Project Info

ScraPy

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments we're proud of

What we learned

What's next

ScraPy ☕📧

Prerequisites ✅

Important File Setup 📋

Demo Run-through 🚀

Example: Running `emailer.py` ✉️

Disclaimer ⚠️

License 📄

Contributing 🤝

Analysis

Technology

AI coding agents

Codebase size

Structure

Application logic

Languages

Dependencies

Feature verification

ScraPy

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments we're proud of

What we learned

What's next

ScraPy ☕📧

Prerequisites ✅

Important File Setup 📋

Demo Run-through 🚀

Example: Running emailer.py ✉️

Disclaimer ⚠️

License 📄

Contributing 🤝

Technology

AI coding agents

Codebase size

Structure

Application logic

Languages

Dependencies

Feature verification

Example: Running `emailer.py` ✉️