# Project export: BAS Climate Action Matcher This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: A tool to match companies to relevant climate actions. Embedding search to find climate initiatives and agentic workflows+tool usage to discover corporate climate actions from sustainability reports - Devpost: https://devpost.com/software/bas-climate-action-matcher - GitHub: https://github.com/Suzehva/bas_labs - Video: https://www.youtube.com/embed/rUhAa9_z8zY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Sustainability Prize: Best prototyping process ($800 Cash); Intersystems: Best Use of GenAI using InterSystems IRIS Vector Search ($2k Cash [1st] & $1.5k Cash [2nd] & 1k Cash [3rd])) - Team: 2 GitHub contributor(s) — Suzehva (8 commits), Bubble Yu (8 commits) ## Devpost submission (written by the team) ### Inspiration Mitigating climate change is only possible through a patchwork of collective action. The future of our planet will be determined by our ability to change previous destructive corporate processes radically. However, companies often struggle to find concrete solutions and actions to reduce their carbon footprint while advancing their business goals. We partner with Race to Zero to create a climate action matcher. Through a user-friendly interface and an agentic workflow grounded in tool usage, we match THE USER COMPANY with relevant UN-catalogued Cooperative Climate Initiatives and provide them with sustainability reports from similar companies and peer corporate actions to inspire concrete action. Ultimately, we want to transform how we approach climate change – from apathy to collective action. By connecting companies and their initiatives, we want to show that action is possible, popular, and influential – especially when done together as an industry, nation, and planet. ### What it does Our tool matches companies to relevant climate initiatives. We provide an agentic system with tools like RAG embedding search on a custom database of company sustainability reports, and UN-cataloged Cooperative Climate Initiatives, web scraping on websites like https://zerotracker.net/ and https://nzdpu.com/home, and more to discover corporate climate actions. ### How we built it DAIN Butterfly: We use the DAIN Butterfly agentic workflow with tool usage as our central orchestrator for user interface interactions. We built custom tools that find companies similar to THE USER COMPANY based on industry sector and country, match companies to UN-catalogued Cooperative Climate Initiatives, and find relevant climate actions from hundreds of sustainability reports from a custom database we built. We provide the agent with the initial context of its foal (e.g., it is trying to write a report that should reference its sources). Still, the agent can choose which tools to use and autonomously decide its tool strategy depending on the outcomes of previous actions and details specified by THE USER COMPANY. To ensure responsible usage, we instruct the agent to include sources to its information (which it can do as our tools return the href links they got their information from) in its findings, allowing THE USER COMPANY to confirm and dive deeper into the sources. We used the DAIN UI components to format the responses engagingly and professionally. InterSystems Embedding Database: We use InterSystems as our database. We collected and embedded 172 UN-catalogued Cooperative Climate Initiatives with descriptions and over 17,000 paragraphs from scraped sustainability reports. NVIDIA Llama Embeddings: We use Llama-3.2-nv-embedqa-1b-v2 embeddings for our embedding database and query embedding in our RAG vector search. LangChain: We use LangChain to load sustainability PDF reports directly from the web and recursively split the text for subsequent chunk embeddings. Google Gemini Scoring and Classification: We implement company sector classification using Gemini Flash Experimental 2.0. Moreover, we use Gemini to score corporate actions based on their reproducibility and return on investment for action ranking and matching. Scrapybara: We implement an agent to find concrete PDF links on corporate websites that may be deep in the link structure of the page. Selenium Web Browser: We implement web scraping using Selenium. ### Challenges we ran into Finding relevant climate actions first proved tricky since sustainability reports can be pretty vague, and embedding similarity search works best if we try to match the target report structure as closely as possible. We solved the problem by having the DAIN agent brainstorm climate initiatives the company could be doing and then verify these ideas by finding actual climate actions by companies in their sustainability reports. Another challenge was to have the agent perform enough actions to take advantage of all our tools. We ended up spending some time on prompt engineering and writing clearer tool descriptions which had a clear boost in performance. ### Accomplishments we're proud of Created an end-to-end pipeline to match companies with sustainability efforts. Created an embedding vector database with hundreds of sustainability reports to be open-sourced to the broader community after the event. Developed core technical skills in web scraping, database manipulation, embedding models, document parsing, and tool creation. Built our understanding of sustainability reporting and found many avenues for continued work. What We Learned We learned a ton during the hackathon! On the technical side, we learned web scraping, document embeddings, how to work with Docker containers, and connecting Python and Typescript! On the environmental side, we opened the door to the vast world of sustainability reporting and tracking. Seeing all the initiatives already underway was inspiring, and we are incredibly excited to keep pushing for more action. ### What's next Extend the initiatives into a dynamic knowledge graph to track the impacts of climate actions. Extend scoring to include nature-based solutions, collaborations, estimated impact, and cost. Create a dashboard to standardize climate reporting for easier comparison. ## README (from the GitHub repository) ## BAS Buddy: AI-Agent for Climate Initiative Matching An agentic system leveraging custom database to connect companies with climate initiatives. Bas Labs in Action [Watch our video here](https://share.descript.com/view/JjI5tob8La9) ## Inspiration Mitigating climate change is only possible through a patchwork of collective action. The future of our planet will be determined by our ability to radically change previous destructive corporate processes. However, companies can often struggle to find concrete solutions and actions to reduce their carbon footprint while advancing their business goals. We partner with Race to Zero to create a **climate action matcher**. Through a user-friendly interface and an agentic workflow grounded in tool-usage, we match THE USER COMPANY with relevant UN-catalogued initiatives and provide them with sustainability reports from similar companies and peer corporate actions to inspire concrete action. Ultimately, we want to transform the way we approach climate change – from apathy to collective action. By making the connections between companies and their initiatives, we want to show that action is possible, popular, and powerful – especially when done together as an industry, nation, and planet. ## What it does Our tool matches companies to relevant climate initiatives. We provide an agentic system with tools like RAG embedding search on a custom database of company sustainability reports and UN-catalogued initiatives, web scraping on websites like https://zerotracker.net/ and https://nzdpu.com/home and more to discover corporate climate actions. ## How we built it DAIN Butterfly: We use the DAIN Butterfly agentic workflow + tool usage as our central orchestrator and user interface interactions. We built custom tools that find similar companies based on industry sector, match companies to UN-catalogued collective corporate initiatives, and find relevant climate actions from hundreds of sustainability reports from a database we built. We used the DAIN UI components to format the responses in an engaging and clear way. InterSystems Embedding Database: We use InterSystems as our database. We collected and embedded 172 UN-catalogued initiatives with descriptions and over 17,000 paragraphs from scraped sustainability reports. NVIDIA Llama Embeddings: We use Llama-3.2-nv-embedqa-1b-v2 embeddings for our embedding database and query embedding in our RAG vector search. LangChain: We use lang chain to load sustainability PDF reports directly from the web and recursively split the text for subsequent chunk embeddings. Google Gemini Scoring and Classification: We implement company sector classification using Gemini Flash Experimental 2.0. Moreover, we use Gemini to score corporate actions based on their reproducibility, and return on investment for action ranking and matching. Scrapybara: We implement an agent to find concrete PDF links on corporate websites that may be deep in the link structure of the page. Selenium Web Browser: We implement web scraping using Selenium. ## Challenges we ran into Finding relevant climate actions first proved tricky since sustainability reports can be quite vague and embedding similarity search works best if we try to match the target report structure as closely as possible. We ended up solving the problem by having the DAIN agent brainstorm climate initiatives the company *could be doing* and then **verify** these ideas by finding **real climate actions by companies** in their sustainability reports. Another challenge was to have the agent perform enough actions to take advantage of all our tools. We ended up spending some time on prompt engineering and writing clearer tool descriptions which had a clear boost in performance. ## Accomplishments that we're proud of Created an end-to-end pipeline for matching companies with sustainability efforts. Created an embedding vector database with hundreds of sustainability reports to be open-sourced to the broader community after the event. Developed core technical skills in web scraping, database manipulation, embedding models, document parsing, and tool creations. Built our understanding of sustainability reporting and found many avenues for continued work. ## What We Learned We learned a ton during the hackathon! On the technical side, we learned web scraping, document embeddings, how to work with Docker containers, and connecting python and typescript! On the environmental side we opened the door into the vast world of sustainability reporting and tracking. It was very inspiring to see all the initiatives already underway, and we are incredibly excited to keep pushing for more action. ## What's next for BAS Climate Action Matcher Extend the initiatives into a dynamic knowledge graph to track the impacts of climate actions. Extend scoring to include nature based solutions, collaborations, estimated impact, and cost. Create a dashboard to standardize climate reporting for easier comparison. ## Detected evidence (automated analysis) Indexed codebase: 21 recognized source files, 93 KB. - C (language) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - LangChain (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (116 of 116) ``` .env.development .gitattributes .gitignore dain.json downloads/json (1).txt downloads/json (10).txt downloads/json (11).txt downloads/json (2).txt downloads/json (3).txt downloads/json (4).txt downloads/json (5).txt downloads/json (6).txt downloads/json (7).txt downloads/json (8).txt downloads/json (9).txt downloads/json.txt experiments/01_Vizualize/01_graph.ipynb experiments/01_Vizualize/interactive.html experiments/01_Vizualize/list_of_nodes_with_color.html experiments/01_Vizualize/nodes.html get_links.py install_intersystems/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-macosx_10_9_universal2.whl install_intersystems/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl install_intersystems/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl install_intersystems/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-win_amd64.whl install_intersystems/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-win32.whl iris-env/bin/activate iris-env/bin/activate.csh iris-env/bin/activate.fish iris-env/bin/Activate.ps1 iris-env/bin/coloredlogs iris-env/bin/debugpy iris-env/bin/distro iris-env/bin/dotenv iris-env/bin/f2py iris-env/bin/httpx iris-env/bin/huggingface-cli iris-env/bin/humanfriendly iris-env/bin/ipython iris-env/bin/ipython3 iris-env/bin/isympy iris-env/bin/jsondiff iris-env/bin/jsonpatch iris-env/bin/jsonpointer iris-env/bin/jupyter iris-env/bin/jupyter-kernel iris-env/bin/jupyter-kernelspec iris-env/bin/jupyter-migrate iris-env/bin/jupyter-run iris-env/bin/jupyter-troubleshoot iris-env/bin/langchain-server iris-env/bin/llamaindex-cli iris-env/bin/llamaindex-legacy-cli iris-env/bin/nltk iris-env/bin/normalizer iris-env/bin/onnxruntime_test iris-env/bin/openai iris-env/bin/pip iris-env/bin/pip3 iris-env/bin/pip3.12 iris-env/bin/pygmentize iris-env/bin/python iris-env/bin/python3 iris-env/bin/python3.12 iris-env/bin/striprtf iris-env/bin/torchfrtrace iris-env/bin/torchrun iris-env/bin/tqdm iris-env/bin/transformers-cli iris-env/include/site/python3.12/greenlet/greenlet.h iris-env/pyvenv.cfg iris-env/share/jupyter/kernels/python3/kernel.json iris-env/share/man/man1/ipython.1 iris-env/share/man/man1/isympy.1 package.json process_document.py README.md requirements.txt score_paragraph.py scored_paragraphs.csv spreadsheet/clean_total.csv spreadsheet/climate_arc_matches.csv spreadsheet/emissions.csv spreadsheet/member_reports.csv spreadsheet/nzdpu_matches.csv spreadsheet/nzt_matches.csv spreadsheet/pdf_links.txt spreadsheet/readme.csv spreadsheet/rtz_partners.csv spreadsheet/total_v3.csv src/BAS_Database.py src/BAS_Table.py src/call_rag_reports.py src/call_rag_UN.py src/index.ts src/scrape_netzero.py src/scrape_nzdpu.py tsconfig.json vectorbase/BASIRIS.py vectorbase/create_report_embed.ipynb vectorbase/create_report_embed.py vectorbase/create_UN_vectordb.py vectorbase/data/mock_climate_action.json vectorbase/data/nzdpu_companies.csv vectorbase/data/nzdpu/nzdpu_companies (1).csv vectorbase/data/reports_links.csv vectorbase/data/reports.csv vectorbase/data/sheetreports.csv vectorbase/data/snapshot_2025-02-16_13-11-05 - NZT DATA.csv vectorbase/data/tech_reports_links.json vectorbase/data/tech_reports.csv vectorbase/data/unfccc_initiatives.csv vectorbase/format_csv.ipynb vectorbase/streamlit_demo.py vectorbase/tools.py vectorbase/utils.py ``` ### Dependencies - package.json: @dainprotocol/cli@^1.0.31, @dainprotocol/service-sdk@^1.0.93, @dainprotocol/utils@^0.0.48, @types/express@^4.17.13, @types/node@^22.5.4, axios@^1.7.5, dotenv@^16.4.7, hono@^4.6.3, python-shell@^5.0.0, ts-node@^10.4.0, typescript@^5.5.4, zod@^3.23.8 - requirements.txt: fastembed, google-generativeai, ipykernel, langchain, langchain-anthropic@>=0.0.1, langchain-community, langchain-iris, langchain-openai, langgraph@>=0.0.15, llama-index-legacy, llama-iris, numpy@>=1.24.0, openai, pandas@>=2.0.0, pypdf, pypdf2, python-dotenv, requests@>=2.31.0, scikit-learn@>=1.3.0, selenium, sentence-transformers@>=2.2.2, setuptools, testcontainers-iris, tiktoken, tqdm@>=4.66.1, transformers, typing-extensions@>=4.5.0 ### Recent commits (newest first) - Update README.md - Update README.md - Add files via upload - Update README.md - Update README.md - Create README.md - Merge branch 'main' of https://github.com/Suzehva/bas_labs - add nzdpu - add rag reports - scraping nzdpu - update scrape - add ui list - add climate RAG pipeline - updates to DAIN - Merge branch 'vector' of github.com:Suzehva/bas_labs - progress on DAIN - rag system - committed reference_text.pdf (bytes) - completed implementation of scoring for a list of paragraphs - extract pdf links into pdf_links.txt, fixed process_document to get ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "BAS", "version": "1.0.0", "description": "A Dain Protocol project", "main": "src/index.ts", "scripts": { "start": "ts-node src/index.ts", "dev": "dain dev", "build": "dain build", "deploy": "dain deploy" }, "dependencies": { "@dainprotocol/cli": "^1.0.31", "@dainprotocol/service-sdk": "^1.0.93", "@dainprotocol/utils": "^0.0.48", "@types/express": "^4.17.13", "@types/node": "^22.5.4", "axios": "^1.7.5", "dotenv": "^16.4.7", "hono": "^4.6.3", "python-shell": "^5.0.0", "ts-node": "^10.4.0", "typescript": "^5.5.4", "zod": "^3.23.8" } } ``` ### requirements.txt ``` <<<<<<< HEAD ======= google-generativeai >>>>>>> 4672d5dc84887d30bd3c96db5be486d8374707c6 langgraph>=0.0.15 langchain>=0.1.0 langchain-anthropic>=0.0.1 typing-extensions>=4.5.0 python-dotenv>=1.0.0 langchain-community langchain-openai pypdf pandas>=2.0.0 langchain-iris testcontainers-iris llama-iris llama-index-legacy sentence-transformers langchain fastembed openai tiktoken python-dotenv pandas ipykernel setuptools pypdf2 pandas>=2.0.0 sentence-transformers>=2.2.2 <<<<<<< HEAD requests>=2.31.0 tqdm>=4.66.1 ======= selenium requests>=2.31.0 tqdm>=4.66.1 transformers >>>>>>> 4672d5dc84887d30bd3c96db5be486d8374707c6 numpy>=1.24.0 scikit-learn>=1.3.0 ``` ### src/index.ts ```typescript import { arrayOutputType, z } from "zod"; import axios from "axios"; import { PythonShell } from 'python-shell'; import * as path from 'path'; // require('dotenv').config({ path: '.env.treehacks' }); // const apiKey = process.env.DAIN_API_KEY; // console.log(apiKey); // this.pythonShell = new PythonShell('worker.py', { // mode: 'text', // pythonPath: 'python3', // pythonOptions: ['-u'], // scriptPath: './python_scripts' // }); type PythonShellOptions = { args: string[]; }; import { defineDAINService, ToolConfig, } from "@dainprotocol/service-sdk"; import { DainResponse, CardUIBuilder, TableUIBuilder, MapUIBuilder, LayoutUIBuilder, CardListUIBuilder } from "@dainprotocol/utils"; import { AgentInfo } from "@dainprotocol/service-sdk"; const find_company_climate_initiatives: ToolConfig = { id: "net_zero_tracker", name: "Find climate initiatives", description: "Finds the climate initiatives a company is doing", input: z .object({ companyName: z.string().describe("Company name"), }) .describe("Input parameters for the company climate initiative request. Make sure to provide the full company name and try different acronyms of the company name if the information returned is for the wrong company."), output: z .object({ company_climate_information: z.string().describe("Company information on climate initiatives"), }), pricing: { pricePerUse: 0, currency: "USD" }, handler: async ({ companyName }, agentInfo, context) => { console.log( `User / Agent ${agentInfo.id} requested information from net_zero_tracker for company ${companyName}` ); function scrape_netzero(companyName: string): Promise { const options: PythonShellOptions = { args: [companyName], }; const scriptPath = path.join(__dirname, 'scrape_netzero.py'); return PythonShell.run(scriptPath, options) // Using Promise-based API .then((result) => { // Since result is an array of strings (one for each printed line), // we join them together return result.join('\n'); }) .catch((err) => { console.error("Error executing Python script:", err); throw err; 2 }); } let company_info: string; try { company_info = await scrape_netzero(companyName); console.log("Python script result:", company_info); } catch (error) { console.error("Error:", error); } // const tableUI = new TableUIBuilder() // .addColumns([ // { key: "name", header: "Name", type: "text" }, // { key: "value", header: "Value", type: "text" } // ]) // .rows(company_info) // .build(); const summary = new CardUIBuilder() .setRenderMode("page") .title(`Climate Initiatives for ${companyName}`) .content(company_info) .build(); const one_climate_initiative = new CardUIBuilder() .setRenderMode("page") .title(`Climate Initiatives for ${companyName}`) .content(company_info) .build(); const fullUI = new CardUIBuilder() .setRenderMode("page") .title(`Climate Initiatives for ${companyName}`) .content(company_info) .addChild(summary).content(company_info) .addChild(one_climate_initiative).content(company_info) .addChild(one_climate_initiative).content(company_info) .addChild(one_climate_initiative).content(company_info) //.addChild(tableUI) .build(); // const gridLayout = new LayoutUIBuilder() // .setLayoutType("grid") // .setColumns(3) // .setGap(24) // .setMargin("32px") // .setBackgroundColor("#f5f5f5") // .build(); const super_basic_UI = new CardUIBuilder() //.title(`Climate Initiatives for ${companyName}`) //.content(company_info) .build(); return new DainResponse({ text: `This response includes data from net zero tracker, which tracks a company's climate's initiatives, for the company ${companyName}`, data: { company_climate_information: company_info, }, ui: super_basic_UI, }); }, }; const find_similar_companies: ToolConfig = { id: "nzdpu", name: "Find similar companies", description: "Finds companies similar to the company we are researching by using information about the sector and country the company is in. Use in combination with find_company_climate_initiatives to find what companies similar to the one we are researching are doing for their climate initiatives.", input: z .object({ sector: z.string().describe("Sector the company is in"), country: z.string().describe("Country the company is located in"), }) .describe("Input parameters for the similar companies request. If a company is located in multiple countries, ask user to provide only one"), output: z .object({ similar_companies: z.array(z.string()).describe("A list of companies similar to the one we are researching. Use find_company_climate_initiatives to find what climate initiatives each company does"), }), pricing: { pricePerUse: 0, currency: "USD" }, handler: async ({ sector, country }, agentInfo, context) => { console.log( `User / Agent ${agentInfo.id} requested information from nzdpu to find similar compabies` ); function scrape_nzdpu(sector: string, country: string): Promise> { const options: PythonShellOptions = { args: [sector, country], }; const scriptPath = path.join(__dirname, 'scrape_nzdpu.py'); return PythonShell.run(scriptPath, options) // Using Promise-based API .then((result) => { // Since result is an array of strings (one for each printed line), // we join them together return JSON.parse(result.join('')); }) .catch((err) => { console.error("Error executing Python script:", err); throw err; 2 }); } [truncated — 10168 more characters] ``` ### get_links.py ```python import pandas as pd import re import os import glob from typing import Set from urllib.parse import unquote class LinkExtractor: def __init__(self): self.base_directory = os.path.abspath(".") self.csv_directory = os.path.join(self.base_directory, "spreadsheet") self.url_pattern = re.compile( r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+' ) def is_pdf_link(self, url: str) -> bool: decoded_url = unquote(url) return decoded_url.lower().endswith('.pdf') def extract_links_from_csv(self, file_path: str) -> Set[str]: df = pd.read_csv(file_path) links = set() for column in df.columns: df[column] = df[column].astype(str) for column in df.columns: cell_links = df[column].str.findall(self.url_pattern) for cell_link_list in cell_links: pdf_links = {link for link in cell_link_list if self.is_pdf_link(link)} links.update(pdf_links) return links def extract_all_links(self) -> Set[str]: all_links = set() csv_files = glob.glob(os.path.join(self.csv_directory, "*.csv")) for csv_path in csv_files: try: file_links = self.extract_links_from_csv(csv_path) all_links.update(file_links) except Exception as e: print(f"Error processing {csv_path}: {str(e)}") output_file = os.path.join(self.csv_directory, "pdf_links.txt") with open(output_file, 'w', encoding='utf-8') as f: for link in sorted(all_links): f.write(f"{link}\n") print(f"Found {len(all_links)} unique PDF links") print(f"Links saved to {output_file}") return all_links def main(): extractor = LinkExtractor() extractor.extract_all_links() if __name__ == "__main__": main() ``` ### score_paragraph.py ```python # implements relevance scoring for paragraphs with reference text -> replace paragraphs (under main) with actual paragraphs import pandas as pd import hashlib import requests import numpy as np from PyPDF2 import PdfReader from sentence_transformers import SentenceTransformer, util model = SentenceTransformer("all-MiniLM-L6-v2") def extract_text_from_pdf(pdf_url): response = requests.get(pdf_url) response.raise_for_status() with open("reference_text.pdf", "wb") as file: file.write(response.content) reader = PdfReader("reference_text.pdf") text = "\n".join([page.extract_text() for page in reader.pages if page.extract_text()]) return text def calculate_relevance(paragraph: str, reference_text: str) -> float: reference_embedding = model.encode(reference_text, convert_to_tensor=True) paragraph_embedding = model.encode(paragraph, convert_to_tensor=True) similarity = util.pytorch_cos_sim(paragraph_embedding, reference_embedding).item() log_score = np.log1p(similarity + 1e-10) return log_score * 10 def score_paragraphs(paragraphs, reference_text): scored_data = [] for paragraph in paragraphs: relevance = calculate_relevance(paragraph, reference_text) paragraph_id = hashlib.md5(paragraph.encode()).hexdigest()[:8] scored_data.append({ "id": paragraph_id, "paragraph": paragraph, "score": relevance }) return pd.DataFrame(scored_data) if __name__ == "__main__": pdf_url = "https://www.oecd.org/content/dam/oecd/en/publications/reports/2024/11/responsible-business-conduct-for-climate-action_b9b43c9c/d098b352-en.pdf" reference_text = extract_text_from_pdf(pdf_url) paragraphs = [ "Investing in renewable energy sources such as solar and wind can significantly reduce a company's carbon footprint.", "Reducing emissions is important for sustainability.", "Companies should consider regulatory compliance to avoid fines and penalties in the future.", "Advancing carbon capture technology could be a game changer in industrial sustainability.", "Water conservation strategies should be considered for long-term environmental responsibility." ] df = score_paragraphs(paragraphs, reference_text) df.to_csv("scored_paragraphs.csv", index=False) print(df) ``` ### process_document.py ```python from langchain_community.document_loaders import PyPDFLoader, CSVLoader from langchain.text_splitter import RecursiveCharacterTextSplitter from sentence_transformers import SentenceTransformer from dotenv import load_dotenv from typing import List, Dict, Set import glob import pandas as pd import re import os import numpy as np import json import requests import tempfile from tqdm import tqdm from urllib.parse import unquote import io load_dotenv() class DocumentProcessor: def __init__(self): self.base_directory = os.path.abspath(".") self.csv_directory = os.path.join(self.base_directory, "spreadsheet") self.pdf_cache_dir = os.path.join(self.base_directory, "pdf_cache") os.makedirs(self.pdf_cache_dir, exist_ok=True) self.text_splitter = RecursiveCharacterTextSplitter( separators=["\n\n", "\n", ". ", "? ", "! "], chunk_size=2000, chunk_overlap=0, length_function=len, is_separator_regex=False ) self.model = SentenceTransformer("all-MiniLM-L6-v2") def clean_text(self, text: str) -> str: text = re.sub(r'\s+', ' ', text) text = text.strip() return text def download_pdf(self, url: str) -> str: try: safe_filename = re.sub(r'[^a-zA-Z0-9]', '_', url) + '.pdf' cache_path = os.path.join(self.pdf_cache_dir, safe_filename) if os.path.exists(cache_path): return cache_path # Download first response = requests.get(url, stream=True) response.raise_for_status() # Cache it with open(cache_path, 'wb') as f: for chunk in response.iter_content(chunk_size=8192): if chunk: f.write(chunk) return cache_path except Exception as e: print(f"Error downloading PDF from {url}: {str(e)}") return None def process_pdf_url(self, url: str) -> List[Dict]: pdf_path = self.download_pdf(url) if not pdf_path: return [] try: loader = PyPDFLoader(pdf_path) pages = loader.load() chunks = [] for page in pages: text = page.page_content if not text.strip(): continue chunks.append({ "content": self.clean_text(text), "metadata": { "source_url": url, "source_type": "pdf", "page": page.metadata.get("page", 0) } }) return chunks except Exception as e: print(f"Error processing PDF {url}: {str(e)}") return [] def process_csv(self, file_path: str) -> List[Dict]: df = pd.read_csv(file_path) chunks = [] pdf_urls = set() for idx, row in df.iterrows(): row_text = [] for col, val in row.items(): if pd.notna(val): row_text.append(f"{col}: {val}") if isinstance(val, str) and val.lower().endswith('.pdf'): pdf_urls.add(val) content = " | ".join(row_text) chunks.append({ "content": content, "metadata": { "source_file": os.path.basename(file_path), "source_type": "csv", "row_index": idx, "columns": list(df.columns) } }) print(f"\nProcessing {len(pdf_urls)} PDFs from {os.path.basename(file_path)}...") for url in tqdm(pdf_urls): pdf_chunks = self.process_pdf_url(url) chunks.extend(pdf_chunks) if pdf_chunks: print(f"Added {len(pdf_chunks)} chunks from {url}") return chunks def load_csvs(self) -> List[Dict]: all_chunks = [] csv_files = glob.glob(os.path.join(self.csv_directory, "*.csv")) for csv_path in csv_files: try: chunks = self.process_csv(csv_path) all_chunks.extend(chunks) print(f"Processed {os.path.basename(csv_path)}: {len(chunks)} total chunks") except Exception as e: print(f"Error processing {csv_path}: {str(e)}") return all_chunks def create_embeddings(self, chunks: List[Dict]) -> List[Dict]: embeddings_list = [] total = len(chunks) texts = [chunk["content"] if isinstance(chunk, dict) else chunk.page_content for chunk in chunks] print(f"Creating embeddings for {total} chunks...") embeddings = self.model.encode(texts, show_progress_bar=True) for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)): content = chunk["content"] if isinstance(chunk, dict) else chunk.page_content metadata = chunk["metadata"] if isinstance(chunk, dict) else chunk.metadata embeddings_list.append({ "content": content, "embedding": embedding.tolist(), "metadata": metadata }) return embeddings_list def save_embeddings(self, embeddings_data: List[Dict], output_dir: str = "embeddings"): os.makedirs(output_dir, exist_ok=True) embeddings_by_source = {} for item in embeddings_data: source = item["metadata"].get("source_url", item["metadata"].get("source_file")) if source not in embeddings_by_source: embeddings_by_source[source] = [] embeddings_by_source[source].append(item) for source, items in embeddings [truncated — 919 more characters] ``` ### src/call_rag_UN.py ```python import os import sys import json from BAS_Database import BAS_Database # Hello I am an agroforestry service based in the Netherlands and we would like some UN climate initiative to join def call_rag_UN(search_query: str) -> list[dict]: """ Perform RAG embeddings for UN initative data. """ db = BAS_Database() result = db.get_un_initative(search_query) return result if __name__ == "__main__": # Get company name from command line arguments if len(sys.argv) > 1: search_query = sys.argv[1] try: content = call_rag_UN(search_query) print(json.dumps(content)) except Exception as e: print(f"Error: {str(e)}", file=sys.stderr) sys.exit(1) else: print("Error: Not enough arguments provided", file=sys.stderr) sys.exit(1) ``` ### src/call_rag_reports.py ```python import os import sys import json from BAS_Database import BAS_Database # Hello I am an agroforestry service based in the Netherlands and we would like some UN climate initiative to join def get_report_section(search_query: str) -> list[dict]: """ Perform RAG search for report data. """ db = BAS_Database() result = db.get_report_section(search_query) return result if __name__ == "__main__": # Get company name from command line arguments if len(sys.argv) > 1: search_query = sys.argv[1] try: content = get_report_section(search_query) print(json.dumps(content)) except Exception as e: print(f"Error: {str(e)}", file=sys.stderr) sys.exit(1) else: print("Error: Not enough arguments provided", file=sys.stderr) sys.exit(1) ``` ### vectorbase/streamlit_demo.py ```python import streamlit as st from BASIRIS import BASDatabase import os ### Streamlit page bas_database = BASDatabase() st.title("BASLABS Climate Action") company_name = st.text_input("Company Name") company_description = st.text_area("Company Description") sectors = [ "Apparel", "Biotech", "health care & pharma", "Chemicals", "Fodd, beverage & agriculture", "Fossil Fuels", "Hospitality", "Infrastructure", "Manufacturing", "Power Generation", "Retail", "Services", "Transportation services", "NA", ] company_sector = st.selectbox("Company Sector", sectors) # chat_input = st.text_input("Ask me anything about climate action...") @st.cache_data() def get_results(query): return bas_database.search_UN(query) if st.button("Submit"): st.write( f""" Company Name: {company_name} Company Description: {company_description} Company Sector: {company_sector} """ ) results = get_results(company_description) results ``` ### vectorbase/tools.py ```python import iris import time import os import pandas as pd from sqlalchemy import create_engine class BAS_DATABASE: def __init__(self): ### Vector Database Setup username = "demo" password = "demo" hostname = os.getenv("IRIS_HOSTNAME", "localhost") port = "1972" namespace = "USER" CONNECTION_STRING = f"{hostname}:{port}/{namespace}" print(CONNECTION_STRING) conn = iris.connect(CONNECTION_STRING, username, password) self.cursor = conn.cursor() def download_table(table_name, output_path): pass # src_engine = create_engine("iris://_SYSTEM:SYS@localhost:1972/USER") # def RAG_report(query_text): # embedTableName = "BASLABS.ClimateReportsEmbed" # def export_table(): # searchVector = self.model.encode( # searchPhrase, normalize_embeddings=True # ).tolist() # sql = f""" # SELECT TOP ? title, summary, description # FROM {self.tableName} # ORDER BY VECTOR_DOT_PRODUCT(description_vector, TO_VECTOR(?)) DESC # """ # self.cursor.execute(sql, [numberOfResults, str(searchVector)]) # results = self.cursor.fetchall() # return [ # dict(title=row[0], summary=row[1], description=row[2]) for row in results # ] ``` [12 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]