# Project export: Blacksmith This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Blacksmith: From Prompt to Performance - Automated Text Model Fine-tuning Made Simple & Accessible for Everyone - Devpost: https://devpost.com/software/blacksmtih - GitHub: https://github.com/tejasprabhune/blacksmith - Video: https://www.youtube.com/embed/d9Wftder1oU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Tejas Prabhune (7 commits), Adi (5 commits) ## Devpost submission (written by the team) ### Inspiration First, we witnessed how Webflow revolutionized web development by abstracting complex coding into intuitive visual interfaces, empowering non-technical creators to build sophisticated websites. This democratization of web development sparked our vision to do the same for machine learning - creating a platform that removes technical barriers and allows users to express their AI needs without dealing with the underlying complexity. Second, Databricks' AutoML platform showed us the power of automated machine learning for classical ML techniques like XGBoost. While their platform serves enterprise users and data scientists effectively, we saw an opportunity to push this concept further. Our vision extends beyond traditional ML by incorporating fine-tuned LLMs and making the entire process more accessible through intelligent agents that handle web scraping and data collection completely automatically, without the user having to lift a finger. Finally, we were captivated by the potential of complex multi-agent workflows to build smaller, more specialized models. The idea of multiple AI agents working in concert - each handling specific tasks like data collection, preprocessing, and optimization - inspired us to bring this vision to life during TreeHacks. This approach not only makes model development more efficient but also opens up new possibilities for creating highly specialized AI models. ### What it does Blacksmith is a no-code, text-only ML training and deployment automation platform. Buzzwords aside, all this means is you put in text, you get out ML models :) With just simple prompts like “Create an ML model based on George Washington”, you’ll get a personalized, historically accurate agent for the first president. Not only do you immediately have access to the chatbot just 10 minutes after you request your model, you also have an API endpoint that can then be as part of your larger multi-agent workflow. Traditionally, this process is extremely tedious and impossible for someone who is just getting into machine learning. For finetuning models, data has to be manually scraped and collected, converted into a Q/A format, labeled, uploaded, and finetuned. For training models from scratch, it’s even worse! You need to set up the machine learning codebase yourself and be very very careful when training these sensitive models. We want machine learning to be accessible to everyone, with the power of personalized agents in every person and organization’s hands. Blacksmith makes this possible. Given the text prompt, it first figures out the type of model and data that should be used. Then we use our extremely fast, custom, multi-agent Computer Use system to jump through webpages and extract raw data for our model. The data is automatically organized and labeled, then GPT and/or Mistral is used for the finetuning process. Once the model is fully trained, you have access to the API endpoint and the chatbot, making the world your oyster! ### How we built it How we built it To create an end-to-end process that transforms a simple user prompt into a fully functional ML model, we developed a sophisticated multi-agent workflow system operating in three distinct phases: Phase 1: Prompt Analysis Agent: Our initial agent performs deep analysis of the user's prompt through analysing three key aspects of what is required next in the workflow. Model type selection: This agent determines whether to use OpenAI or Mistral's base models based on the specific use case requirements and performance characteristics. We use NLP techniques to parse user intent and map it to specific model architectures. For instance, if a user wants to create a customer service bot, the first thing this agent decides is its model. Model type selection: This agent determines whether to use OpenAI or Mistral's base models based on the specific use case requirements and performance characteristics. We use NLP techniques to parse user intent and map it to specific model architectures. For instance, if a user wants to create a customer service bot, the first thing this agent decides is its model. Data type requirements: Simple selection of what needs datatypes need to be collected in order to train/finetune a machine learning model that suits our users needs; especially useful as we plan to incorporate vision and audio in future in addition to text Data type requirements: Simple selection of what needs datatypes need to be collected in order to train/finetune a machine learning model that suits our users needs; especially useful as we plan to incorporate vision and audio in future in addition to text Web scraping prompt formulation: Transforms user requirements into sophisticated search strategies that combine domain-specific keywords with contextual parameters. For example, "customer service interactions" might be expanded to include "support tickets", "FAQ responses", and "resolution examples". Web scraping prompt formulation: Transforms user requirements into sophisticated search strategies that combine domain-specific keywords with contextual parameters. For example, "customer service interactions" might be expanded to include "support tickets", "FAQ responses", and "resolution examples". Phase 2: Intelligent Web Scraping Agents - Our dual-LLM architecture orchestrates sophisticated web scraping: Thinking LLM: Functions as a strategic orchestrator for the scraping, maintaining a comprehensive understanding of the scraping mission's progress and goals. Continuously evaluating webpage content against the target data requirements, making real-time decisions about which content to extract and where to navigate next with state management. Incorporating learned patterns from successful data extraction attempts and adapting to different website structures. Thinking LLM: Functions as a strategic orchestrator for the scraping, maintaining a comprehensive understanding of the scraping mission's progress and goals. Continuously evaluating webpage content against the target data requirements, making real-time decisions about which content to extract and where to navigate next with state management. Incorporating learned patterns from successful data extraction attempts and adapting to different website structures. Selenium Commander LLM: Translates high-level directives from the thinking LLM into precise, executable Selenium commands, handling complex scenarios, and optimizing scraping execution speed. These commands are directly executed using Seleniums computer use and are used to navigate the web and scrape useful data. Selenium Commander LLM: Translates high-level directives from the thinking LLM into precise, executable Selenium commands, handling complex scenarios, and optimizing scraping execution speed. These commands are directly executed using Seleniums computer use and are used to navigate the web and scrape useful data. Phase 3: Model Fine-tuning & Deployment Agent Data Processing & Formatting: Implements cleaning algorithms that normalize the scraped content, remove irrelevant information, and ensure consistency across different sources. Generates high-quality question-answer pairs from raw scraped data, ensuring each pair contributes meaningful finetuning signals to the model. Data Processing & Formatting: Implements cleaning algorithms that normalize the scraped content, remove irrelevant information, and ensure consistency across different sources. Generates high-quality question-answer pairs from raw scraped data, ensuring each pair contributes meaningful finetuning signals to the model. Model Training Integration: Manages the entire fine-tuning pipeline through direct integration with OpenAI and Mistral's APIs, handling authentication, data upload, and training monitoring. Model Training Integration: Manages the entire fine-tuning pipeline through direct integration with OpenAI and Mistral's APIs, handling authentication, data upload, and training monitoring. Deployment Architecture: Implements a responsive chat interface that handles real-time model inference while maintaining low latency and high availability. Deployment Architecture: Implements a responsive chat interface that handles real-time model inference while maintaining low latency and high availability. The result is a fully automated pipeline that transforms a simple user prompt into a production-ready, domain-specialized LLM, making advanced AI development accessible to non-technical users while maintaining enterprise-grade quality and reliability. ### Challenges we ran into The most difficult part of this process was the web scraping section. Given the unstructured nature of the web, it became near impossible for us to extract data using classical techniques, so we turned to state-of-the-art models like Computer Use for scraping. However, since most of these models use manual mouse and keyboard inputs, the time for webscraping simply was taking way too long. Given this, we decided to build our own Computer Use system that was much, much faster, but also more deterministic. Using a novel multi-agent optimization algorithm integrating Selenium, we enable step-by-step navigation on the web with source code reflection. A Thinker and a Worker work in tandem to create and run Selenium commands on the server, with constant summaries being returned to the frontend. With this, we cut down our time for data collection by multiple orders of magnitude! ### Accomplishments we're proud of Our biggest accomplishment was our novel autonomous web scraping technology described in the challenges section :) We’re also really proud of how integrated and end-to-end our solution is. The first time we were able to get the full system from prompt to chatbot running perfectly was a huge milestone for us, and we want to share the same delights we have training machine learning models with everyone in the world, not just with experts in the field. ### What we learned We learned so much about the finetuning process and how all of these APIs work together. The orchestration process between multiple agents depending on each other was a huge learning curve at first, since it feels a little like managing a distributed system of really smart toddlers LOL. But this whole process was super fun and the integration with traditional tools like Selenium was a big part of our learning process. Outside of this, our background is largely in machine learning itself and not too much in the web domain, so building a web workflow that seamlessly integrates with our frontend was a big learning process for us! ### What's next We have so much planned for Blacksmith! The very first thing we want to work on is diversifying all of the models that we support. Right now, we support text models from OpenAI and Mistral, but we really want to integrate something like the RoboFlow universe into our system, where you can work on large amounts of computer vision models and train them from scratch based purely on a text prompt. For bigger enterprise use cases, we also want users and organizations to allow for training with their own unstructured data that we will automatically label and organize for them. A big pain point in finetuning is the data formatting part, and Blacksmith’s custom Computer Use is really really good at figuring out what data is important and how to format it best for finetuning purposes. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 22 recognized source files, 54 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Mistral AI (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (31 of 31) ``` .DS_Store .gitignore .python-version app.py blacksmith/__init__.py blacksmith/backend/__init__.py blacksmith/backend/server.py blacksmith/deploy/__init__.py blacksmith/parse/__init__.py blacksmith/parse/parsing.py blacksmith/scrape/__init__.py blacksmith/scrape/scraping.py blacksmith/scrape/scraping2.py blacksmith/train/__init__.py blacksmith/train/finetune.py blacksmith/train/generate_jsonl.txt pyproject.toml README.md ui/next-env.d.ts ui/package.json ui/postcss.config.js ui/public/fonts/TobiasTRIAL-SemiBold.otf ui/src/app/globals.css ui/src/app/layout.tsx ui/src/app/page.tsx ui/src/components/AnimatedResponse.tsx ui/src/components/TypeWriter.tsx ui/src/services/api.ts ui/tailwind.config.js ui/tsconfig.json uv.lock ``` ### Dependencies - pyproject.toml: bs4@>=0.0.2, fastapi[standard]@>=0.115.8, mistralai@>=1.5.0, openai@>=1.63.0, pydantic@>=2.10.6, scrapybara@>=2.2.5, selenium@>=4.28.1, together@>=1.4.1 - ui/package.json: @types/node@^20.11.5, @types/react@^18.2.48, @types/react-dom@^18.2.18, autoprefixer@^10.4.17, next@^14.2.24, postcss@^8.4.33, react@^18.2.0, react-dom@^18.2.0, tailwindcss@^3.4.1, typescript@^5.3.3 ### Recent commits (newest first) - devpost version - merge backend - working ui - full running app.py - clean root - parse and scraping - parse and scraping - parse and scraping - add finetuning - add enum lol - layout - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### pyproject.toml ``` [project] name = "blacksmith" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.10" dependencies = [ "bs4>=0.0.2", "fastapi[standard]>=0.115.8", "mistralai>=1.5.0", "openai>=1.63.0", "pydantic>=2.10.6", "scrapybara>=2.2.5", "selenium>=4.28.1", "together>=1.4.1", ] ``` ### ui/package.json ``` { "name": "blacksmith-ui", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@types/node": "^20.11.5", "@types/react": "^18.2.48", "@types/react-dom": "^18.2.18", "autoprefixer": "^10.4.17", "next": "^14.2.24", "postcss": "^8.4.33", "react": "^18.2.0", "react-dom": "^18.2.0", "tailwindcss": "^3.4.1", "typescript": "^5.3.3" } } ``` ### app.py ```python import asyncio import time import requests from blacksmith.parse import Prompt, MlModel from blacksmith.scrape import Scraper # host = "https://4017-2001-5a8-450b-4900-9555-6b3a-f17f-92e.ngrok-free.app" # # x = requests.post(host + "/request_model/", params={"request": "Make a model that will speak like Sherlock Holmes"}) # # print("Prompt: ", x.text) # # for i in range(100): # time.sleep(3) # # y = requests.get("https://4017-2001-5a8-450b-4900-9555-6b3a-f17f-92e.ngrok-free.app/request_stage/") # # print(y.text) #$ async def main(): #$ prompt = Prompt(webscraping_prompt="Extract source code for the language Zig", model_type="gpt-4o-mini-2024-07-18", data_type="text") #$ scraper = Scraper() #$ await scraper.scrape_content(prompt) #$ print(scraper.state.body_content) #$ scraper.close() #$ #$ asyncio.run(main()) host = "https://4017-2001-5a8-450b-4900-9555-6b3a-f17f-92e.ngrok-free.app" x = requests.post(host + "/completions/", params={"request": "Who are you?"}) print("Prompt: ", x.text) ``` ### blacksmith/backend/server.py ```python from ..parse import PromptParser, Prompt, MlModel from ..scrape import Scraper, AutomationState from ..train import SmithModel import asyncio from typing import Literal from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from openai import AsyncOpenAI app = FastAPI() origins = ["*"] app.add_middleware( CORSMiddleware, allow_origins=origins, allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) stage: Literal["parsing", "scraping", "finetuning", "deploying", "not_ready", "deployed"] = "not_ready" scraper = Scraper() model = SmithModel("gpt") ft_model = "ft:gpt-4o-mini-2024-07-18:monet::B1IBTo3q" @app.post("/request_model/") async def request_model(request: str): print(request) prompt_parser = PromptParser() prompt = prompt_parser.analyze_request(request) asyncio.create_task(handle_model_request(request, prompt)) return prompt.to_json() @app.post("/completions/") async def completions(request: str): print(request) print(model.system_prompt) client = AsyncOpenAI() if model.system_prompt is None: model.system_prompt = "You are Sherlock Holmes." response = await client.chat.completions.create( model=ft_model, messages=[ {"role": "system", "content": model.system_prompt}, {"role": "user", "content": request}, ]) return response.choices[0].message.content @app.get("/request_stage/") async def request_stage(): global ft_model if not scraper.complete: stage = "scraping" summary = await scraper.state.summarize() elif not model.complete: stage = "finetuning" summary = await model.summarize() else: stage = "deployed" summary = "Deployed with name: " + model.model.ft_name ft_model = model.model.ft_name return {"stage": stage, "summary": summary} async def handle_model_request(request: str, prompt: Prompt): print("Handling model request") data = await scrape_data(prompt) model = await finetune_model(request, prompt.webscraping_prompt, data) return model async def scrape_data(prompt: Prompt) -> AutomationState: print("Scraping data") await scraper.scrape_content(prompt) print("Scraping complete") return scraper.state async def finetune_model(model_query: str, data_query: str, data: str): await model.finetune_text_model(model_query, data_query, data) ``` ### ui/src/app/layout.tsx ```typescript import './globals.css' import type { Metadata } from 'next' export const metadata: Metadata = { title: 'Blacksmith UI', description: 'UI Interface for Blacksmith Python Package', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return ( {children} ) } ``` ### ui/src/app/page.tsx ```typescript 'use client' import { useState, useEffect, useRef } from 'react' import TypeWriter from '@/components/TypeWriter' import AnimatedResponse from '@/components/AnimatedResponse' // Define our responses const base_url = "https://4017-2001-5a8-450b-4900-9555-6b3a-f17f-92e.ngrok-free.app" const responses = [ { header: "Deciding your model", content: "" }, { header: "Scraping data for your model", content: "" }, { header: "Creating your model", content: "" }, { header: "Deploying your model", content: "" } ]; export default function Home() { const [input, setInput] = useState('') const [messages, setMessages] = useState([]) const [isMinimized, setIsMinimized] = useState(false) const [messageVisible, setMessageVisible] = useState([]) const [hideInput, setHideInput] = useState(false) const [currentResponseIndex, setCurrentResponseIndex] = useState(-1) const [responseVisible, setResponseVisible] = useState([false, false, false, false]) const inputRef = useRef(null) useEffect(() => { if (inputRef.current) { inputRef.current.focus() } }, []) useEffect(() => { if (messages.length > messageVisible.length) { setTimeout(() => { setMessageVisible([...messageVisible, true]) }, 100) } }, [messages]) const resetToInitialState = () => { setInput('') setMessages([]) setIsMinimized(false) setMessageVisible([]) setHideInput(false) setCurrentResponseIndex(-1) // Focus the input after a short delay to ensure DOM is ready setTimeout(() => { if (inputRef.current) { inputRef.current.focus() } }, 100) } const handleKeyPress = async (e: React.KeyboardEvent) => { if (e.key === 'Enter' && !e.shiftKey) { e.preventDefault() if (input.trim()) { setMessages([...messages, input]) setInput('') setIsMinimized(true) setTimeout(() => { setHideInput(true) }, 200) await requestMlModel(input.trim()) //added this await here } } } const requestMlModel = async (prompt: string) => { // POST request to /request_model/ that will send request: prompt // Receives back the prompt that is generated as the search prompt console.log("Sending request with prompt:", prompt); const response = await fetch(base_url + '/request_model/?request=' + prompt, { method: 'POST', }) let response_json = JSON.parse(await response.json()) console.log("Response json:", response_json) responses[0].content = response_json.webscraping_prompt setCurrentResponseIndex(0); // Set this first to ensure component is visible setResponseVisible(prev => { const newVisible = [...prev]; newVisible[0] = true; return newVisible; }) console.log("Received search prompt:", responses[0].content) // Wait for the first animation to complete before polling // Animation takes ~30ms per 2 chars + 1000ms wait + 500ms collapse = ~2-3s total const animationTime = (responses[0].content.length / 2) * 30 + 1500; await new Promise(resolve => setTimeout(resolve, animationTime)); await pollStage() } type StageResponse = { stage: keyof typeof stages summary: string } //write a hashmap with stage as key and then index of response as value const stages = { "parsing": 0, "scraping": 1, "finetuning": 2, "deploying": 3 } as const; //now based on what the stage is, in pollstage we should manipulate that and set it to whatever the corresponding index is const pollStage = async () => { console.log("waiting 5 seconds") return new Promise((resolve) => { setTimeout(async () => { console.log("waited 5 seconds, fetching") const response = await fetch(base_url + "/request_stage/", { method: 'GET', }) let response_json = await response.json() as StageResponse let stage = response_json.stage let summary = response_json.summary responses[stages[stage]].content = summary console.log("Responses: " + responses[0].content + responses[1].content + responses[2].content + responses[3].content) // First set visibility to false to trigger re-render setResponseVisible(prev => { const newVisible = [...prev]; newVisible[stages[stage]] = false; console.log("Response visible: " + newVisible); return newVisible; }) // Then set it back to true after a delay setTimeout(() => { setResponseVisible(prev => { const newVisible = [...prev]; newVisible[stages[stage]] = true; console.log("Response visible: " + newVisible); return newVisible; }) }, 500) setCurrentResponseIndex(prev => stages[stage]) resolve(response_json) }, 5000) }) } const handleResponseComplete = () => { // Move to next response after a small delay setTimeout(async () => { const result = await pollStage() if (!result) { //if our polling is resolving to null then we can try again await pollStage() } }, 500) } const handleInput = (e: React.ChangeEvent) => { const textarea = e.target setInput(textarea.value) textarea.style.height = 'auto' textarea.style.height = `${textarea.scrollHeight}px` } return (
{/* Title */}