# Project export: NinjAttach This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: NinjAttach is an AI-powered app that automates email workflows by extracting data, filling forms, and generating drafts—streamlining document tasks and boosting productivity. - Devpost: https://devpost.com/software/automail - GitHub: https://github.com/louis-salletourne/NinjAttach - Video: https://www.youtube.com/embed/phJiA5K13EA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Louis Sallé-Tourne (31 commits), VictorRicaux (30 commits), Alice Ternois (13 commits) ## Devpost submission (written by the team) ### Inspiration NinjAttach was inspired by our frustrations during past work experiences, where we wasted time filling repetitive PDF attachments and feared missing urgent emails in a flood of incoming messages. We built NinjAttach to automate these tedious tasks, ensuring users can manage their inboxes efficiently and focus on what matters most. ### What it does Email Reading: Automatically reads the latest emails and extracts attachments. PDF Information Extraction: Extracts missing fields from PDF attachments. User Profile Management: Updates user profiles with extracted information. Form Generation: Generates a dynamic form for users to fill in missing information. Email Draft Creation: Creates an email draft with the updated PDF attached for the user to send. Task Scheduling: Automatically saves the task in the user’s Google Calendar as a reminder to send the email. ### How we built it We structured the project into three core components: Gmail Integration: Communicating with Gmail via the API to automatically read emails and extract attachments. Web App Development: Creating a user-friendly interface where users can review and complete extracted information from PDFs. PDF Processing: Automatically filling PDFs in the correct fields based on extracted data. ### Challenges we ran into One of the main challenges we faced was our limited coding background. None of us had developed an app before, so we had to rapidly upskill to bring NinjaAttach to life. The PDF auto-filling feature proved particularly challenging as we needed to understand the structure of widgets and PDF formatting. Additionally, handling complex PDFs required us to explore computer vision tools we had never used before. Despite these hurdles, our team leveraged our diverse backgrounds and worked cohesively, allowing us to overcome the technical challenges and grow our skill sets quickly. ### Accomplishments we're proud of We’re proud of how quickly our team upskilled, moving from minimal coding experience to developing NinjAttach. We learned how to deploy a web app, work with Google APIs, and tackle the complexities of PDF auto-filling. One key feature we’re especially proud of is the ability to remember and reuse the user’s previously entered key information, streamlining future PDF completions. Despite barely knowing each other before the hackathon, we quickly built a strong team dynamic, leveraging each member’s strengths to collaborate effectively. Our ability to unite and execute efficiently is something we take great pride in. ### What we learned Throughout the project, we gained valuable experience in coding and deploying a web app using Streamlit Cloud. We deepened our understanding of working with APIs, particularly the Gmail API and LLM APIs, and improved our skills in handling PDF structure and automated form filling. Additionally, we developed stronger team management and communication practices, ensuring efficient collaboration. Lastly, we learned the importance of resilience in overcoming challenges and delivering a successful project. ### What's next for AutoMail Next for NinjAttach is expanding its capabilities beyond email attachments and replies. We plan to introduce a more intuitive, user-friendly interface and a dynamic dashboard, allowing users to track tasks, emails, and document workflows in real-time. In addition, we aim to integrate advanced AI features, such as smart email prioritization and seamless connections with task management tools. Our vision is to build an intelligent assistant that evolves with user needs, boosting efficiency and productivity. ## README (from the GitHub repository) # Email Agent - Powered by Gemini AI ## Overview The Email Agent is a Streamlit application designed to automate the process of extracting information from emails and PDF attachments, updating user profiles, and generating email drafts with the updated information. This project leverages various functionalities to streamline the workflow and improve efficiency. ## Features - **Email Reading**: Automatically reads the latest email and extracts attachments. - **PDF Information Extraction**: Extracts missing fields from PDF attachments. - **User Profile Management**: Updates the user profile with the extracted information. - **Form Generation**: Generates a dynamic form for users to fill in missing information. - **Email Draft Creation**: Creates an email draft with the updated PDF attached. - **Task Scheduling**: Automatically saves the task in the user’s Google Tasks as a reminder to send the email. ## Installation To run this project, you need to have Python and Streamlit installed. Follow these steps to set up the project: 1. **Clone the repository**: ```bash git clone https://github.com/louis-salletourne/NinjAttach.git cd NinjAttach ``` 2. **Install the dependencies**: ```bash pip install -r requirements.txt ``` 3. **Set up the environment variables**: - Create a `.env` file in the root directory. - Add the following environment variables to the `.env` file: ```bash GEMINI_API_KEY=your-gemini-api-key GOOGLE_APPLICATION_CREDENTIALS=path/to/your/google-credentials.json ``` - Activate the Gmail API and download the `credentials.json` file from the Google Cloud Console. 4. **Run the Streamlit app**: ```bash streamlit run webapp.py ``` ## Detected evidence (automated analysis) Indexed codebase: 14 recognized source files, 31 KB. - LangChain (technology) — detected in the code - Python (language) — detected in the code - Streamlit (technology) — detected in the code ## Codebase structure (from repository index) ### Files (19 of 19) ``` .devcontainer/devcontainer.json .gitignore easy_pdf/fill_missing_files.py easy_pdf/list_missing_fields.py easy_pdf/main.py email_read.py gemini_request/clean_fields.py gemini_request/gemini_request.py pdf/export_missing_fields.py pdf/missing_field.py pdf/read_pdf.py pdf/to_json.py README.md requirements_for_devs.txt requirements.txt tabs/profile_settings.py tasks.py user_profile.json webapp.py ``` ### Dependencies - requirements.txt: google-api-python-client, google-auth, google-auth-httplib2, google-auth-oauthlib, google-generativeai, langchain, langchain-google-genai, pymupdf, PyPDF2, python-dotenv, reportlab, streamlit ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - readme upadte - webapp last update - readme update - Merge branch 'main' of https://github.com/louis-salletourne/cal_hacks - requiremnts modif - Added Dev Container Folder - deleted test pdfs - gitignore - added tasks - gitignore - Merge pull request #8 from louis-salletourne/logo - logo - logo - Merge pull request #7 from louis-salletourne/profile - tabs - webapp update - user_profile_example ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` langchain python-dotenv google-generativeai langchain-google-genai PyPDF2 google-generativeai reportlab google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client streamlit pymupdf ``` ### easy_pdf/main.py ```python from easy_pdf.list_missing_fields import list_widgets pdf_path = 'tests/test_pdf/test/test_1.pdf' output_path = 'output.pdf' def export_missing_fields(pdf_file: str) -> list: widgets = list_widgets(pdf_path = pdf_file) return widgets ``` ### tasks.py ```python from google.oauth2 import service_account from google_auth_oauthlib.flow import InstalledAppFlow from google.auth.transport.requests import Request from googleapiclient.discovery import build import pickle import os from datetime import datetime, timezone, timedelta import time # Define scopes for Gmail and Tasks SCOPES = [ 'https://www.googleapis.com/auth/gmail.readonly', 'https://www.googleapis.com/auth/tasks' ] # Authentication function def authenticate(): creds = None # Token.pickle stores the user's access and refresh tokens, and is created automatically when the authorization flow completes for the first time if os.path.exists('token.pickle'): with open('token.pickle', 'rb') as token: creds = pickle.load(token) # If there are no valid credentials available, let the user log in. if not creds or not creds.valid: if creds and creds.expired and creds.refresh_token: creds.refresh(Request()) else: flow = InstalledAppFlow.from_client_secrets_file( 'credentials.json', SCOPES) creds = flow.run_local_server(port=0) # Save the credentials for the next run with open('token.pickle', 'wb') as token: pickle.dump(creds, token) return creds # Function to get Gmail drafts def get_gmail_drafts(service): results = service.users().drafts().list(userId='me').execute() drafts = results.get('drafts', []) return drafts # Function to create Google Task def create_google_task(service, task_title): task = { 'title': task_title, 'notes': 'This task was created from a Gmail draft.', 'due': (datetime.now(timezone.utc) + timedelta(days=1)).strftime('%Y-%m-%dT%H:%M:%S.%fZ') } result = service.tasks().insert(tasklist='@default', body=task).execute() print(f'Task created: {result.get("title")}, Task ID: {result.get("id")}') return result.get("id") # Function to mark Google Task as completed def mark_task_as_completed(service, task_id): task = service.tasks().get(tasklist='@default', task=task_id).execute() task['status'] = 'completed' updated_task = service.tasks().update(tasklist='@default', task=task_id, body=task).execute() print(f'Task marked as completed: {updated_task.get("title")}') # Function to check if draft email was sent def check_sent_email(service, draft_subject): results = service.users().messages().list(userId='me', labelIds=['SENT']).execute() messages = results.get('messages', []) for message in messages: msg = service.users().messages().get(userId='me', id=message['id']).execute() headers = msg['payload']['headers'] for header in headers: if header['name'].lower() == 'subject' and header['value'] == draft_subject: return True return False def create_task(): # Authenticate and create service for Gmail and Tasks creds = authenticate() gmail_service = build('gmail', 'v1', credentials=creds) tasks_service = build('tasks', 'v1', credentials=creds) # Get Gmail drafts drafts = get_gmail_drafts(gmail_service) if drafts: for draft in drafts: draft_id = draft['id'] draft_message = gmail_service.users().drafts().get(userId='me', id=draft_id).execute() subject = None headers = draft_message['message']['payload']['headers'] for header in headers: if header['name'].lower() == 'subject': subject = header['value'] print("Subject :", subject) break if subject: # Create Google Task with the draft subject task_id = create_google_task(tasks_service, subject) # Monitor the sent folder for the draft to be sent print(f'Waiting for the draft with subject "{subject}" to be sent...') while True: sent = check_sent_email(gmail_service, subject) if sent: # Mark the task as completed once the email is sent mark_task_as_completed(tasks_service, task_id) break time.sleep(10) # Wait for 10 seconds before checking again else: print('No drafts found.') ``` ### webapp.py ```python import streamlit as st import json import os from easy_pdf.main import export_missing_fields from email_read import read_email, create_draft from gemini_request.gemini_request import found_and_missing_infos from gemini_request.clean_fields import clean_fields from easy_pdf.fill_missing_files import fill_missing_fields from tabs import profile_settings from tasks import create_task @st.cache_resource def cached_read_email(): return read_email() @st.cache_resource def cached_create_draft(_output, completed_file): return create_draft(_output, completed_file) @st.cache_resource def cached_found_and_missing_fields(user_profile, path): missing_fields = export_missing_fields(path) return found_and_missing_infos(user_profile, missing_fields) # JSON file path json_file_path = "user_profile.json" # Initialize or load the JSON file def load_json(): if not os.path.exists(json_file_path): return {} with open(json_file_path, 'r') as f: return json.load(f) # Save the updated JSON data def save_json(data): with open(json_file_path, 'w') as f: json.dump(data, f, indent=4) def IndirectObject(*args): return None # Main Streamlit app def main(): # Set the favicon and page title st.set_page_config( page_title="Email Agent - Powered by Gemini AI", page_icon="images/WhatsApp Image 2024-10-20 at 02.56.05.jpeg", # Adjust the path as needed layout="wide" ) st.title("NinjAttach - Powered by Gemini AI") st.write("Fetching the latest email...") # Create tabs tab1, tab2 = st.tabs(["Process Email", "Profile Settings"]) with tab1: # Read the email to extract attachments output = cached_read_email() attachments = output['Attachments'] if not attachments: st.write("No attachments found.") return # Consider the first attachment pdf_path = attachments[0] st.write("Found and saved PDF attachment") # Load current user profile original_user_profile = load_json() user_profile = original_user_profile.copy() # Display current user profile if available if user_profile: st.subheader("Current User Profile") st.json(user_profile) else: st.write("No user profile found.") # Extract missing fields from the PDF found_info, missing_info = cached_found_and_missing_fields(user_profile, pdf_path) if not missing_info: st.write("All required information is already present in the user profile.") create_draft(output, "files/certificate_of_presence_erasmus_2023-2024 (1)_completed.pdf") st.write("Draft email created with the updated PDF attached.") return # Create a form for the user to fill in missing fields st.subheader("Fill in the missing fields:") # Use a form to delay execution until the submit button is pressed with st.form(key='info_form'): user_inputs = {} for field, answer in found_info.items(): user_inputs[field] = st.text_input(field, value=answer, key=field) for field, info in missing_info.items(): if info.get('/FT') == '/Tx': help_text = info.get('/TU', '') user_inputs[field] = st.text_input(info.get('/T', field), placeholder=help_text) # Submit button submitted = st.form_submit_button(label='Submit') # Only process the form when submitted if submitted: for field, value in user_inputs.items(): if value.strip(): user_profile[field] = value user_profile_new_part = {key: user_profile[key] for key in missing_info.keys() if key in user_profile} new_part_clean = clean_fields(user_profile_new_part) original_user_profile.update(new_part_clean) save_json(original_user_profile) saved_profile = load_json() st.success("Profile updated successfully!") st.json(saved_profile) output_file_name = pdf_path.replace('.pdf', '') + "_completed.pdf" _ = fill_missing_fields(pdf_path=pdf_path, user_profile=user_profile, output_file_name=output_file_name) cached_create_draft(output, output_file_name) create_task() st.write("Draft email created with the updated PDF attached.") with tab2: # Call the display_tab function from the profile_settings module profile_settings.display_tab() if __name__ == "__main__": main() ``` ### email_read.py ```python import os.path import json import base64 from google.oauth2.credentials import Credentials from google_auth_oauthlib.flow import InstalledAppFlow from google.auth.transport.requests import Request from googleapiclient.discovery import build from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from email.mime.base import MIMEBase from email import encoders from dotenv import load_dotenv import os import google.generativeai as genai load_dotenv() genai.configure(api_key=os.getenv("GEMINI_API_KEY")) import shutil # If modifying these SCOPES, delete the file token.json. SCOPES = ['https://www.googleapis.com/auth/gmail.modify'] def read_email(): """Shows basic usage of the Gmail API. Lists the user's Gmail labels and reads emails. """ creds = None # The file token.json stores the user's access and refresh tokens, and is # created automatically when the authorization flow completes for the first time. if os.path.exists('token.json'): with open('token.json', 'r') as token: # Load the file as a dictionary token_data = json.load(token) creds = Credentials.from_authorized_user_info(token_data, SCOPES) # If there are no (valid) credentials available, let the user log in. if not creds or not creds.valid: if creds and creds.expired and creds.refresh_token: creds.refresh(Request()) else: flow = InstalledAppFlow.from_client_secrets_file( 'credentials.json', SCOPES) creds = flow.run_local_server(port=0) # Save the credentials for the next run with open('token.json', 'wb') as token: token.write(creds.to_json().encode()) # If credentials are valid, build the Gmail API service if not creds: print("Authentication failed.") return None service = build('gmail', 'v1', credentials=creds) # Search query to find all emails received during a time # Format: after:2023/09/30 before:2023/11/01 query = 'in:inbox' # Get the user's inbox and retrieve emails based on the query results = service.users().messages().list(userId='me', q=query, maxResults=1).execute() messages = results.get('messages', []) if not messages: print("No messages found.") return None # Get the message ID message_id = messages[0]['id'] # Use the message ID to get the email details message_data = service.users().messages().get(userId='me', id=message_id, format='full').execute() # Extract information from the email headers headers = message_data['payload']['headers'] subject = get_header(headers, 'Subject') sender = get_header(headers, 'From') recipient = get_header(headers, 'To') date = get_header(headers, 'Date') # Extract the email body email_body = get_message_body(message_data['payload']) # Check for attachments attachments = [] if 'parts' in message_data['payload']: for part in message_data['payload']['parts']: if part.get('filename'): # Attachment is present attachment_id = part['body']['attachmentId'] attachment = service.users().messages().attachments().get(userId='me', messageId=message_id, id=attachment_id).execute() file_data = base64.urlsafe_b64decode(attachment['data'].encode('UTF-8')) # Save the attachment attachment_file = save_attachment(part['filename'], file_data) attachments.append(attachment_file) # Decode the email body (Base64 encoded) if email_body: decoded_body = base64.urlsafe_b64decode(email_body).decode('utf-8') else: decoded_body = "(No body content)" # Output the email details print(f"From: {sender}") print(f"To: {recipient}") print(f"Date: {date}") print(f"Subject: {subject}") print(f"Body: {decoded_body}") if attachments: print("Attachments:", attachments) else: print("No attachments found.") print("\n" + "="*50 + "\n") # Separator between emails output = { "From": sender, "To": recipient, "Date": date, "Subject": subject, "Body": decoded_body, "Attachments": attachments, "service": service, } return output def get_header(headers, name): """Helper function to retrieve header values.""" for header in headers: if header['name'] == name: return header['value'] return None def get_message_body(payload): """Helper function to extract the message body from the email payload.""" if 'body' in payload and 'data' in payload['body']: return payload['body']['data'] elif 'parts' in payload: for part in payload['parts']: if part['mimeType'] == 'text/plain': return part['body']['data'] return '' def save_attachment(filename, data): """Helper function to save the attachment to the local filesystem.""" folder_path = 'files' # Create a folder for attachments if it doesn't exist if not os.path.exists(folder_path): os.makedirs(folder_path) # Save the file in the 'attachments' folder file_path = os.path.join(folder_path, filename) with open(file_path, 'wb') as f: f.write(data) print(f"Attachment saved: {file_path}") return file_path model = genai.GenerativeModel(model_name="gemini-1.5-pro") # JSON file path json_file_path = "user_profile.json" # Initialize or load the JSON file def load_json(): with open(json_file_path, 'r') as f: return json.load(f) def create_draft(output, completed_file): """Create and send a draft with an attachment.""" # Create the MIME message message = MIMEMultipart() message['to'] = output['From'] # Send it back to the original sender message['from'] = output['To'] # The authorized user (you) [truncated — 1634 more characters] ``` ### easy_pdf/list_missing_fields.py ```python from PyPDF2 import PdfReader def list_widgets(pdf_path): """Lists all the widget names in a PDF file.""" try: with open(pdf_path, "rb") as file: reader = PdfReader(file) fields = reader.get_fields() return fields except FileNotFoundError: print(f"The file {pdf_path} does not exist.") except Exception as e: print(f"An error occurred: {e}") ``` ### pdf/to_json.py ```python # Step 3: Fill the Fields Using AI (Example: using a pre-trained GPT model from OpenAI API) import json def to_json(input_string, output_file_path): # Remove the markdown code block syntax json_string = input_string.replace('```json\n', '').replace('\n```', '') # Parse the JSON string data = json.loads(json_string) # Write the JSON data to a file with open(output_file_path, 'w') as json_file: json.dump(data, json_file, indent=4) print("JSON file created successfully!") ``` ### pdf/export_missing_fields.py ```python import os import json from pdf.read_pdf import read_pdf_informations from pdf.missing_field import identify_missing_fields from pdf.to_json import to_json pdf_file = "./tests/test_pdf/test/test_1.pdf" json_file = "json_file.json" def export_missing_fields(pdf_file): file, size, metadata = read_pdf_informations(pdf_file) json_str = identify_missing_fields(file, size, metadata) to_json(json_str, json_file) with open(json_file, 'r') as file: data = json.load(file) # Check if the file exists, then delete it if os.path.exists(json_file): os.remove(json_file) print(f"{json_file} has been deleted.") else: print(f"{json_file} does not exist.") return [item['field'] for item in data] ``` ### easy_pdf/fill_missing_files.py ```python import fitz # PyMuPDF from pathlib import Path import json # Import the json module output_file_name = 'files/output.pdf' # Assuming the output file will also be in the current directory def fill_missing_fields(pdf_path: str, user_profile: dict, output_file_name: str = output_file_name): # Load the data from the JSON file entry_data = user_profile # Open the PDF document with fitz.open(pdf_path) as doc: target_page = doc[0] # Assuming we want to edit the first page # Iterate over each form field in the PDF and fill it with the data from JSON for field in target_page.widgets(): field_name = field.field_name # Get the field name if field.field_type == fitz.PDF_WIDGET_TYPE_TEXT: # If the field is a text field field.field_value = entry_data.get(field_name, '') # Get the corresponding value from JSON field.update() # Update the field in the PDF elif field.field_type == fitz.PDF_WIDGET_TYPE_CHECKBOX: # If the field is a checkbox field.field_value = entry_data.get(field_name, False) # Get the checkbox value (True/False) field.update() # Update the field in the PDF doc.save(output_file_name) # Save the updated PDF print(f"Form filled and saved to {output_file_name}") return output_file_name ``` ### pdf/read_pdf.py ```python # Step 1: Reading the PDF import google.generativeai as genai import PyPDF2 from dotenv import load_dotenv import os load_dotenv() genai.configure(api_key=os.getenv("GEMINI_API_KEY")) page_number = 1 def read_file(file_path): sample_file = genai.upload_file(path=file_path, display_name="Gemini 1.5 PDF") print(f"Uploaded file '{sample_file.display_name}' as: {sample_file.uri}") file = genai.get_file(name=sample_file.name) print(f"Retrieved file '{file.display_name}' as: {sample_file.uri}") return file def get_page_size(file_path, page_number=1): # Open the PDF file with open(file_path, 'rb') as file: # Initialize the PDF reader reader = PyPDF2.PdfReader(file) # Get the specified page page = reader.pages[page_number - 1] # Page numbers are 0-indexed # Get the media box (page dimensions) media_box = page.mediabox # Extract the width and height width = media_box.width height = media_box.height return [width, height] def get_pdf_metadata(file_path): # Open the PDF file with open(file_path, 'rb') as file: # Initialize the PDF reader reader = PyPDF2.PdfReader(file) # Get the document information (metadata) metadata = reader.metadata return metadata def read_pdf_informations(file_path, page_number=1): # Open the PDF file file = read_file(file_path) size = get_page_size(file_path, page_number) metadata = get_pdf_metadata(file_path) return file, size, metadata ``` [4 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]