# Project export: Broker AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Commercial real estate data is scattered across the web. We bring it together. - Devpost: https://devpost.com/software/broker-ai - GitHub: https://github.com/b0kch01/market-crawler - Team: 3 GitHub contributor(s) — nathan (33 commits), Joshua (14 commits), Haoze Chen (8 commits) ## Devpost submission (written by the team) ### Inspiration "There's not a single database for food halls in the United States" —VP Real Estate Development, ocV!BE The issue we are addressing today involves real estate, specifically the reliance on static, non-customizable, and expensive databases prevalent in the American market. Currently, real estate developers rely on data-driven decisions for land development, utilizing databases like CoStar, which sells data on apartments and homes to aid in making informed decisions. However, a significant gap exists: there are no current databases for food and market halls in the United States. A decade ago, food halls numbered around 100 locations nationwide. This figure has since surged by 400%, with over 500 food halls nationwide. Yet, real estate developers lack information on these properties due to the absence of a comprehensive tracking system. This information is sparse on the internet, and no active database collects it. Given the rapidly changing nature of food halls and trends, we cannot rely on existing databases, which are often outdated, static, and prone to bias. Currently, real estate developers have two options: (1) purchase data from a data broker at an exorbitant cost, or (2) conduct the research themselves. Even when opting for the first choice, the data aggregation process by brokers is very manual and time-consuming. This data is often biased because it is derived from past projects, resulting in a non-comprehensive view and inaccurate data. Our inspiration for this project came from Joshua's experience working with a real estate firm that is developing a $4 billion development project in the SoCal Anaheim area. A major challenge Joshua’s colleagues—with over 20 years of experience—face is the lack of customizable data and an accurate pulse on market trends for food, market halls, and other commercial real estate sectors. Even brokers from Cushman & Wakefield couldn't find any databases on food halls. Our goal with Broker AI is to bridge this gap, moving away from costly manual research to employing smart web crawling tools. Introducing Broker AI Our solution is an AI-powered web crawler and scraper that continuously searches the internet data on food halls, recording what characteristics lead to successful food halls. This approach allows us to bypass the traditional methods real estate researchers are forced to choose from: either relying on brokers, which often results in biased, non-customizable, and expensive data or conducting research independently—a time-consuming process that can waste significant human capital. Our smart web crawlers leverage AI and large language models to navigate the web, analyzing and extracting accurate information about food halls. This enables us to feed our databases in real-time, offering the most comprehensive and up-to-date information available, thus providing real estate developers with the best insight into the commercial real estate market for food halls. This untapped territory, now accessible through our Broker AI, represents a significant opportunity. By creating a database akin to what CoStar offers for homes and apartments, but focused on food halls, we can capitalize on this growing market segment. For context, I am a consultant for a real estate development firm currently working on a $4 billion project in the SoCal Anaheim area. A major challenge we—and many other developers with over 20 years of experience—face is the lack of customizable data and an accurate pulse on market trends for food, market halls, and other commercial real estate sectors. Our goal is to bridge this gap, moving away from costly independent research to employing smart AI crawling tools. ### How we built it At the heart of our innovation are the AI Agents, specifically designed for web crawling the food hall market. These aren't your run-of-the-mill LLM bots; they're equipped with advanced algorithms that enable them to navigate complex website structures intelligently. By chaining a team of agents together for a specific aspect of market research (data entry, navigation, synthesis, retrieval, etc) these agents can achieve impeccable levels of online ability, ensuring a thorough report on each food hall. To tackle the challenge of web scraping at scale, we've also employed multithreading techniques. We enable multiple agents to operate concurrently on a single report, drastically reducing the time required. It's akin to having several skilled data miners working simultaneously, each extracting valuable insights without stepping on each other's toes. As soon as new data is available, the user is notified immediately. Once the data is collected, the real magic begins. We use a sophisticated data pipeline to process and analyze the information, preparing it for integration. This is where Convex comes into play. Our choice of Convex as the database solution is strategic; it excels in handling real-time data updates. By leveraging Convex's capabilities, we ensure that the data not only feeds into our database seamlessly but also triggers updates to our React-based frontend in real time. This ensures that users get to see the latest trends in the food hall real estate market without any delay. The front end, crafted with React, is designed to be as dynamic as the data it displays. It's built to handle real-time updates efficiently, providing a smooth and responsive user experience. This level of interactivity is crucial for engaging users and offering them up-to-the-minute insights. ### Challenges we ran into Building our do-it-all web crawler was no walk in the park. We hit every bump in the road, from sneaky edge cases that tried to block our data scraping to the tricky task of making our crawler smart enough to think on its feet—or wheels? Anyway, we gave it some cool moves like website-hopping or doing a graceful exit when things got tough. Throwing Convex into the mix for our real-time database was like adding a puzzle to our puzzle, especially with all those threads running around. And let's not even start on the front end – we're backend heroes, so you can imagine the adventures (and misadventures) we had there. Lots of "Oops" and "Aha" moments, but hey, we made it through with a bunch of laughs and a whole lot of learning. ### Accomplishments we're proud of We built the first-ever self-growing food hall database in the country, hands-off from researchers! Tough project, and lots of late nights, but we finished strong and proud. ### What we learned Oh man, this hackathon was a wild ride. We thought our project was going to be pretty straightforward, but boy, were we wrong. Ended up pulling an all-nighter trying to fix this one bug that just wouldn't go away. It was like every time we thought we got it, something else popped up. Learned a lot though, especially about how important it is to be ready for anything and that sometimes, things are way harder than they look. Plus, we got to see how teamwork kicks in when you're all equally confused and tired but still determined to figure it out. Definitely a memorable experience, but next time, I'm bringing more snacks and maybe a pillow. ### What's next Next up for Broker AI? Big plans! We're turning this real estate game-changer into a commercial powerhouse. After getting thumbs-up from some big-shot VPs, we're all in on making research a breeze—not just for properties, but for anything. Want the lowdown on medical breakthroughs? Or which fish tank is trending? Our super scraper's got you. We're here to track all the trends, making sure you're always in the know, no matter the topic. Stay tuned; it's going to be exciting! ## README (from the GitHub repository) # market-crawler TreeHacks 2024 Project ## Getting Started ### 0. Install npm ```bash npm install ``` ### 1. Run Convex ```bash npx convex dev ``` ### 2. Build the Front-End ```bash bun run dev ``` ## Detected evidence (automated analysis) Indexed codebase: 82 recognized source files, 193 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (94 of 94) ``` .gitignore agents/start.py backend/.env.local backend/.gitignore backend/bun.lockb backend/convex/_generated/api.d.ts backend/convex/_generated/api.js backend/convex/_generated/dataModel.d.ts backend/convex/_generated/server.d.ts backend/convex/_generated/server.js backend/convex/findings.js backend/models/FoodHall.py backend/models/MongoFoodHall.py backend/package.json backend/routes.py backend/webcrawler/__init__.py backend/webcrawler/crawler.py backend/webcrawler/CrawlerTools.py backend/webcrawler/ResearchHall.py front-end/.eslintrc.json front-end/.gitignore front-end/components.json front-end/convex/_generated/api.d.ts front-end/convex/_generated/api.js front-end/convex/_generated/dataModel.d.ts front-end/convex/_generated/server.d.ts front-end/convex/_generated/server.js front-end/convex/findings.js front-end/jsconfig.json front-end/next.config.mjs front-end/package.json front-end/postcss.config.js front-end/README.md front-end/src/app/demo/page.jsx front-end/src/app/globals.css front-end/src/app/layout.js front-end/src/app/page.js front-end/src/components/carousel.jsx front-end/src/components/insightCard.jsx front-end/src/components/navbar.jsx front-end/src/components/searchbar.jsx front-end/src/components/ui/accordion.jsx front-end/src/components/ui/alert-dialog.jsx front-end/src/components/ui/alert.jsx front-end/src/components/ui/aspect-ratio.jsx front-end/src/components/ui/avatar.jsx front-end/src/components/ui/badge.jsx front-end/src/components/ui/button.jsx front-end/src/components/ui/calendar.jsx front-end/src/components/ui/card.jsx front-end/src/components/ui/carousel.jsx front-end/src/components/ui/checkbox.jsx front-end/src/components/ui/collapsible.jsx front-end/src/components/ui/command.jsx front-end/src/components/ui/context-menu.jsx front-end/src/components/ui/dialog.jsx front-end/src/components/ui/drawer.jsx front-end/src/components/ui/dropdown-menu.jsx front-end/src/components/ui/form.jsx front-end/src/components/ui/hover-card.jsx front-end/src/components/ui/input.jsx front-end/src/components/ui/label.jsx front-end/src/components/ui/menubar.jsx front-end/src/components/ui/navigation-menu.jsx front-end/src/components/ui/pagination.jsx front-end/src/components/ui/popover.jsx front-end/src/components/ui/progress.jsx front-end/src/components/ui/radio-group.jsx front-end/src/components/ui/resizable.jsx front-end/src/components/ui/scroll-area.jsx front-end/src/components/ui/select.jsx front-end/src/components/ui/separator.jsx front-end/src/components/ui/sheet.jsx front-end/src/components/ui/skeleton.jsx front-end/src/components/ui/slider.jsx front-end/src/components/ui/sonner.jsx front-end/src/components/ui/switch.jsx front-end/src/components/ui/table.jsx front-end/src/components/ui/tabs.jsx front-end/src/components/ui/textarea.jsx front-end/src/components/ui/toast.jsx front-end/src/components/ui/toaster.jsx front-end/src/components/ui/toggle-group.jsx front-end/src/components/ui/toggle.jsx front-end/src/components/ui/tooltip.jsx front-end/src/components/ui/use-toast.js front-end/src/components/views/datatable-demo.jsx front-end/src/components/views/neighborhoods-info.jsx front-end/src/contexts/data-context.js front-end/src/lib/utils.js front-end/tailwind.config.js he.csv README.md test.py ``` ### Dependencies - backend/package.json: convex@^1.10.0, dotenv@^16.4.5, lucide@^0.331.0 - front-end/package.json: @hookform/resolvers@^3.3.4, @radix-ui/react-accordion@^1.1.2, @radix-ui/react-alert-dialog@^1.0.5, @radix-ui/react-aspect-ratio@^1.0.3, @radix-ui/react-avatar@^1.0.4, @radix-ui/react-checkbox@^1.0.4, @radix-ui/react-collapsible@^1.0.3, @radix-ui/react-context-menu@^2.1.5, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-hover-card@^1.0.7, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.0.2, @radix-ui/react-menubar@^1.0.4, @radix-ui/react-navigation-menu@^1.1.4, @radix-ui/react-popover@^1.0.7, @radix-ui/react-progress@^1.0.3, @radix-ui/react-radio-group@^1.1.3, @radix-ui/react-scroll-area@^1.0.5, @radix-ui/react-select@^2.0.0, @radix-ui/react-separator@^1.0.3, @radix-ui/react-slider@^1.1.2, @radix-ui/react-slot@^1.0.2, @radix-ui/react-switch@^1.0.3, @radix-ui/react-tabs@^1.0.4, @radix-ui/react-toast@^1.1.5, @radix-ui/react-toggle@^1.0.3, @radix-ui/react-toggle-group@^1.0.4, @radix-ui/react-tooltip@^1.0.7, @tanstack/react-table@^8.12.0, @types/node@20.11.19, @types/react@18.2.56, autoprefixer@^10.0.1, babel@^6.23.0, class-variance-authority@^0.7.0, clsx@^2.1.0, cmdk@^0.2.1, convex@^1.9.0, date-fns@^3.3.1, embla-carousel-react@^8.0.0-rc22, lucide@^0.331.0, lucide-react@^0.331.0, next@14.1.0, next-themes@^0.2.1, postcss@^8, react@^18, react-day-picker@^8.10.0, react-dom@^18, react-hook-form@^7.50.1, react-resizable-panels@^2.0.9, sonner@^1.4.0, tailwind-merge@^2.2.1, tailwindcss@^3.3.0, tailwindcss-animate@^1.0.7, typescript@5.3.3, vaul@^0.9.0, zod@^3.22.4 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/b0kch01/market-crawler - google alerts link - finial commit (first outside of hackathon) - more pushes - frontend lol - thing - thingy - asjdfkl - stuff - weewoo - table - stuff - Finish neighborhoods-info's UI - zoo wee mama - wooooo - push convex fullstack - t - stuff on optimization - photos - add eager scraping and formatting ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/package.json ``` {"dependencies":{"convex":"^1.10.0","dotenv":"^16.4.5","lucide":"^0.331.0"}} ``` ### front-end/package.json ``` { "name": "front-end", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@hookform/resolvers": "^3.3.4", "@radix-ui/react-accordion": "^1.1.2", "@radix-ui/react-alert-dialog": "^1.0.5", "@radix-ui/react-aspect-ratio": "^1.0.3", "@radix-ui/react-avatar": "^1.0.4", "@radix-ui/react-checkbox": "^1.0.4", "@radix-ui/react-collapsible": "^1.0.3", "@radix-ui/react-context-menu": "^2.1.5", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-hover-card": "^1.0.7", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-menubar": "^1.0.4", "@radix-ui/react-navigation-menu": "^1.1.4", "@radix-ui/react-popover": "^1.0.7", "@radix-ui/react-progress": "^1.0.3", "@radix-ui/react-radio-group": "^1.1.3", "@radix-ui/react-scroll-area": "^1.0.5", "@radix-ui/react-select": "^2.0.0", "@radix-ui/react-separator": "^1.0.3", "@radix-ui/react-slider": "^1.1.2", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-switch": "^1.0.3", "@radix-ui/react-tabs": "^1.0.4", "@radix-ui/react-toast": "^1.1.5", "@radix-ui/react-toggle": "^1.0.3", "@radix-ui/react-toggle-group": "^1.0.4", "@radix-ui/react-tooltip": "^1.0.7", "@tanstack/react-table": "^8.12.0", "babel": "^6.23.0", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "cmdk": "^0.2.1", "convex": "^1.9.0", "date-fns": "^3.3.1", "embla-carousel-react": "^8.0.0-rc22", "lucide": "^0.331.0", "lucide-react": "^0.331.0", "next": "14.1.0", "next-themes": "^0.2.1", "react": "^18", "react-day-picker": "^8.10.0", "react-dom": "^18", "react-hook-form": "^7.50.1", "react-resizable-panels": "^2.0.9", "sonner": "^1.4.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "vaul": "^0.9.0", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "20.11.19", "@types/react": "18.2.56", "autoprefixer": "^10.0.1", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "5.3.3" } } ``` ### front-end/src/app/layout.js ```javascript import { Inter } from "next/font/google"; import "./globals.css"; import { DataContextProvider } from "../contexts/data-context"; const inter = Inter({ subsets: ["latin"] }); export const metadata = { title: "Market Crawler", description: "2024 TreeHacks project", }; export default function RootLayout({ children }) { return ( {children} ); } ``` ### front-end/src/app/page.js ```javascript 'use client' import { useQuery } from 'convex/react'; import { api } from '@/../convex/_generated/api'; export default function Home() { const halls = useQuery(api.findings.getAll); return (
{ halls == undefined ? Loading... : halls.map((obj) => { return ( { Object.keys(obj).map((key) => { return ( {key} {`${obj[key]}`} ) }) } ) }) }
); } ``` ### front-end/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.9.0. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### backend/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.10.0. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### front-end/src/app/demo/page.jsx ```javascript "use client"; import InsightCard from "@/components/insightCard"; import Navbar from "@/components/navbar"; import SearchBar from "@/components/searchbar"; import { DataTableDemo } from "@/components/views/datatable-demo"; export default function DemoPage() { return (
{ }} />
Insight Icon

AI Insights

Live Food Hall Database

); } ``` ### test.py ```python import pandas as pd import numpy as np import random import clipboard csv = pd.read_csv('he.csv', header=0) # iterate through the rows of the csv file output = "" for index, row in csv.iterrows(): output += "{\n" if row['Name'] and not pd.isna(row["Name"]): output += (f"\"name\": \"{row['Name']}\",\n") if row['City'] and not pd.isna(row["City"]): output += (f"\"city\": \"{row['City']}\",\n") if row['State'] and not pd.isna(row["State"]): output += (f"\"state\": \"{row['State']}\",\n") if row['# of food stalls'] and not pd.isna(row["# of food stalls"]): output += (f"\"food\": \"{row['# of food stalls']}\",\n") if row['Bars'] and not pd.isna(row["Bars"]): output += (f"\"bars\": \"{row['Bars']}\",\n") if row['Retail'] and not pd.isna(row["Retail"]): output += (f"\"retail\": \"{row['Retail']}\",\n") if row["Year Opened"] and not pd.isna(row["Year Opened"]): output += (f"\"year_established\": \"{row['Year Opened']}\",\n") output += (f"\"median_income\": \"${random.randint(30000, 100000):,}\",\n") output += ( f"\"population_density\": \"{random.randint(50, 200)}/sq.miles\",\n") output += ("},") # copy clipboard.copy(output) ``` ### front-end/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### agents/start.py ```python from uagents import Agent, Context, Bureau agent_multiply = Agent(name="multiply", seed="_multiply_recovery") agent_base = Agent(name="base", seed="_base_recovery") @agent_base.on_event("startup") async def startup(ctx: Context): ctx.logger.info("Agent base started") @agent_multiply.on_event("startup") async def startup(ctx: Context): ctx.logger.info("Agent multiply started") if __name__ == "__main__": bureau = Bureau() bureau.add(agent_base) bureau.add(agent_multiply) bureau.run() ``` [70 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]