# Project export: Grandkid This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Catching the Modern-Day Fish - Devpost: https://devpost.com/software/grandkid - GitHub: https://github.com/TylerKerch/TreeHacks2024 - Video: https://www.youtube.com/embed/ZT_F5yFrbFc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Charan Sriram (52 commits), YuanSamuel (39 commits), PhoenixPhighter (33 commits), Tyler Kerch (27 commits) ## Devpost submission (written by the team) ### Inspiration As computer science kids, being home for the holidays doesn't mean being free from work: it means we become 24/7 tech support! As we brainstormed for TreeHacks, we thought a lot about accessibility and how much we take for granted; we all shared experiences of helping our grandparents navigate the challenges of modern technology. With recent incredible jumps in computer vision, low-latency computing, and generative AI, we figured there must be a way to make modern technology work for our golden-age friends and family, instead of against them. We noticed current accessibility services (like Apple's Screen Reader) are useful utilities for those with severe visual impairments, but for the vast majority of elderly folks, confusion over navigation, functionality, and communication through software is much, much more common. We decided to create "Grandkid", a powerful tool that doesn't just help the elderly troubleshoot tumultuous technology, but helps teach them fundamentals in computer literacy. ### What it does At its core, Grandkid is about answering questions. Users can speak queries aloud while pressing a button, and we generate answers to help guide them through software processes. For example, Sarvesh's grandpa runs a prayer group and is always printing, or rather, trying to print PDFs, excerpts from books, and Google Docs. He sometimes gets confused by the different options on the screen and has trouble when errors we think are trivial arise. With Grandkid, he could simply ask, while on the Google Docs page, "How do I print this?". Using a combination of computer vision and generative AI, we would break the process down into simple, single-action steps and walk through it with him. Additionally, we can highlight relevant components, which will help him form strong associations with common workflows, improving his e-literacy not only on Google Docs but the web/computer in general. ### How we built it There are four sections to our architecture. 1) We use a Swift AppKit frontend which handles speech input and speech rendering and makes necessary socket pack sends to interact with our backend 2) We built a Go web socket server that takes images and queries from the frontend and constructs embeddings on relevant interactable components in parallel 3) Our machine-learning layer involves several models - CLIP is utilized both for checking image differences to decide when to dispatch API requests and for allowing us to take subsections of images and help decide which interactable portion we direct the user towards - OpenAI's GPT-4 Vision Preview model is used for taking a multimodal context of an image and query string to broadly understand which step in the process a user is currently at, and what actions are feasible - Gogosseract, a Go port of the Tesseract OCR model to aid with text recognition of individual interactable components - Roboflow's UI Detection Model, a supervised computer vision model that bounds different interactable UI components to support positional highlighting 4) A lightweight JS server with ngrok to reduce token sizes for GPT by switching away from base64 image encodings to storing images on disk (shop local!) ### Challenges we ran into This was an incredibly technically complex challenge, as we had to grapple with a lot of moving parts. There are multiple languages and frameworks at play, there are multiple models that need to produce coherent results, and there was a good deal of systems-level concurrency that we needed to get latency as low as possible, which of course introduced a multitude of race conditions. In 36 hours, trying to get goroutines to play nice with constant packet sends over a web socket from Swift code that also had to be dynamic enough to deal with erratic user scrolling and navigation, along with lowering the time it took to run several models, was a tall order. ### Accomplishments we're proud of 1) Fixing most (?) of the race conditions related to multithreading and channel communication in the Go code and finally got Grandkid to guide us through each step of a workflow smoothly and quickly 2) Getting the web socket code working with different models publishing messages and sending our first packets between terminals. Also getting those performance benefits by using realtime polling instead of a standard HTTP server 3) Making the logo into a gif that would play whenever we recorded audio (c'mon, it's cute) ### What we learned While we learned umpteen different things grinding the past day and a half, the main lesson we learned was that this technology is truly visible on the horizon. Amazing leaps in accessibility are possible with the generalizability of AI. If we, goofy college students who occasionally took coding breaks to play goldfish basketball (don't ask - rules are self-explanatory), could build this strong of a product with a wide scope of use cases, then in a few years, after dedicated efforts from talented coders in industry and academia, we will see services that beneficially transform the way even the least tech-literate people live. ### What's next Call it the sunk-cost fallacy, but after spending so many hours working on an idea we're truly passionate about, we're not ready to call it quits just because the hackathon's over. There are a lot of cool features that we could add, such as doing click automation to make workflows even easier (something we thought of from the very start), letting users control the level of given information or quiz themselves, and even the ability to fine-tune a personal instance of Grandkid (imagine Grandpa/Grandma's face when they see that!). ## README (from the GitHub repository) ## Inspiration As computer science kids, being home for the holidays doesn't mean being free from work: it means we become 24/7 tech support! As we brainstormed for TreeHacks, we thought a lot about accessibility and how much we take for granted; we all shared experiences of helping our grandparents navigate the challenges of modern technology. With recent incredible jumps in computer vision, low-latency computing, and generative AI, we figured there must be a way to make modern technology work *for* our golden-age friends and family, instead of *against* them. We noticed current accessibility services (like Apple's Screen Reader) are useful utilities for those with severe visual impairments, but for the vast majority of elderly folks, confusion over navigation, functionality, and communication through software is much, *much* more common. We decided to create "Grandkid", a powerful tool that doesn't just help the elderly troubleshoot tumultuous technology, but helps teach them fundamentals in computer literacy. ## What it does At its core, Grandkid is about answering questions. Users can speak queries aloud while pressing a button, and we generate answers to help guide them through software processes. For example, Sarvesh's grandpa runs a prayer group and is always printing, or rather, *trying* to print PDFs, excerpts from books, and Google Docs. He sometimes gets confused by the different options on the screen and has trouble when errors we think are trivial arise. With Grandkid, he could simply ask, while on the Google Docs page, "How do I print this?". Using a combination of computer vision and generative AI, we would break the process down into simple, single-action steps and walk through it with him. Additionally, we can highlight relevant components, which will help him form strong associations with common workflows, improving his e-literacy not only on Google Docs but the web/computer in general. ## How we built it There are four sections to our architecture. 1) We use a Swift AppKit frontend which handles speech input and speech rendering and makes necessary socket pack sends to interact with our backend 2) We built a Go web socket server that takes images and queries from the frontend and constructs embeddings on relevant interactable components in parallel 3) Our machine-learning layer involves several models - `CLIP` is utilized both for checking image differences to decide when to dispatch API requests and for allowing us to take subsections of images and help decide which interactable portion we direct the user towards - `OpenAI's GPT-4 Vision Preview` model is used for taking a multimodal context of an image and query string to broadly understand which step in the process a user is currently at, and what actions are feasible - `Gogosseract`, a Go port of the Tesseract OCR model to aid with text recognition of individual interactable components - `Roboflow's UI Detection Model`, a supervised computer vision model that bounds different interactable UI components to support positional highlighting 4) A lightweight JS server with `ngrok` to reduce token sizes for GPT by switching away from base64 image encodings to storing images on disk (shop local!) ## Challenges we ran into This was an incredibly technically complex challenge, as we had to grapple with a lot of moving parts. There are multiple languages and frameworks at play, there are multiple models that need to produce coherent results, and there was a good deal of systems-level concurrency that we needed to get latency as low as possible, which of course introduced a multitude of race conditions. In 36 hours, trying to get goroutines to play nice with constant packet sends over a web socket from Swift code that also had to be dynamic enough to deal with erratic user scrolling and navigation, *along* with lowering the time it took to run several models, was a tall order. ## Accomplishments that we're proud of 1) Fixing most (?) of the race conditions related to multithreading and channel communication in the Go code and finally got Grandkid to guide us through each step of a workflow smoothly and quickly 2) Getting the web socket code working with different models publishing messages and sending our first packets between terminals. Also getting those performance benefits by using realtime polling instead of a standard HTTP server 3) Making the logo into a gif that would play whenever we recorded audio (c'mon, it's cute) ## What we learned While we learned umpteen different things grinding the past day and a half, the main lesson we learned was that this technology is truly visible on the horizon. Amazing leaps in accessibility are possible with the generalizability of AI. If we, goofy college students who occasionally took coding breaks to play goldfish basketball (don't ask - rules are self-explanatory), could build this strong of a product with a wide scope of use cases, then in a few years, after dedicated efforts from talented coders in industry and academia, we will see services that beneficially transform the way even the least tech-literate people live. ## What's next for Grandkid Call it the sunk-cost fallacy, but after spending so many hours working on an idea we're truly passionate about, we're not ready to call it quits just because the hackathon's over. There are a lot of cool features that we could add, such as doing click automation to make workflows even easier (something we thought of from the very start), letting users control the level of given information or quiz themselves, and even the ability to fine-tune a personal instance of Grandkid (imagine Grandpa/Grandma's face when they see that!). ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 72 KB. - Express (technology) — detected in the code - Go (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - Swift (language) — detected in the code ## Codebase structure (from repository index) ### Files (57 of 57) ``` .DS_Store .gitignore backend/backend backend/base64.zig backend/boxSelect.go backend/clip-server/server.py backend/clip-server/Untitled.ipynb backend/constants/constants.go backend/eng.traineddata backend/go.mod backend/go.sum backend/image_segment.go backend/image.go backend/main.go backend/ocr.go backend/query.go backend/README.md backend/s3_utils.go backend/vector_processing.go file_server/package.json file_server/server.js frontend/.DS_Store frontend/frontend.xcodeproj/project.pbxproj frontend/frontend.xcodeproj/project.xcworkspace/contents.xcworkspacedata frontend/frontend.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist frontend/frontend.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved frontend/frontend.xcodeproj/project.xcworkspace/xcuserdata/samuel.xcuserdatad/UserInterfaceState.xcuserstate frontend/frontend.xcodeproj/project.xcworkspace/xcuserdata/sarveshphoenix.xcuserdatad/UserInterfaceState.xcuserstate frontend/frontend.xcodeproj/xcshareddata/xcschemes/frontend.xcscheme frontend/frontend.xcodeproj/xcuserdata/samuel.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist frontend/frontend.xcodeproj/xcuserdata/samuel.xcuserdatad/xcschemes/xcschememanagement.plist frontend/frontend.xcodeproj/xcuserdata/sarveshphoenix.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist frontend/frontend.xcodeproj/xcuserdata/sarveshphoenix.xcuserdatad/xcschemes/xcschememanagement.plist frontend/frontend/.DS_Store frontend/frontend/AppDelegate.swift frontend/frontend/Assets.xcassets/.DS_Store frontend/frontend/Assets.xcassets/AccentColor.colorset/Contents.json frontend/frontend/Assets.xcassets/AppIcon.appiconset/Contents.json frontend/frontend/Assets.xcassets/AudioInput.imageset/Contents.json frontend/frontend/Assets.xcassets/Contents.json frontend/frontend/Assets.xcassets/MenuIcon.imageset/Contents.json frontend/frontend/AudioControl/AudioInputModal.swift frontend/frontend/AudioControl/NSImageView.swift frontend/frontend/AudioControl/TextSpeaker.swift frontend/frontend/AudioControl/VoiceRecorder.swift frontend/frontend/Base.lproj/Main.storyboard frontend/frontend/CursorControl/CursorContents.swift frontend/frontend/CursorControl/CursorController.swift frontend/frontend/frontend.entitlements frontend/frontend/Info.plist frontend/frontend/MenuBar/MenuBarController.swift frontend/frontend/ScreenReader/ScreenPainter.swift frontend/frontend/ScreenReader/ScreenReader.swift frontend/frontend/ServerConnection/ClientSocket.swift frontend/frontend/ServerConnection/SocketModels.swift frontend/frontend/ViewController.swift README.md ``` ### Dependencies - backend/go.mod: git.sr.ht/~sbinet/gg@v0.5.0, github.com/ajstarks/svgo@v0.0.0-20211024235047-1546f124cd8b, github.com/aws/aws-sdk-go@v1.50.20, github.com/campoy/embedmd@v1.0.0, github.com/danlock/gogosseract@v0.0.11-0ad3421, github.com/danlock/pkg@v0.0.17-a9828f2, github.com/disintegration/imaging@v1.6.2, github.com/go-fonts/liberation@v0.3.2, github.com/go-latex/latex@v0.0.0-20231108140139-5c1ce85aa4ea, github.com/go-pdf/fpdf@v0.9.0, github.com/goccmack/gocc@v0.0.0-20230228185258-2292f9e40198, github.com/golang/freetype@v0.0.0-20170609003504-e2365dfdc4a0, github.com/google/uuid@v1.6.0, github.com/gorilla/websocket@v1.5.1, github.com/jerbob92/wazero-emscripten-embind@v1.3.0, github.com/jmespath/go-jmespath@v0.4.0, github.com/lpernett/godotenv@v0.0.0-20230527005122-0de1d4c5ef5e, github.com/otiai10/gosseract/v2@v2.4.1, github.com/pmezard/go-difflib@v1.0.0, github.com/tetratelabs/wazero@v1.5.0, golang.org/x/exp@v0.0.0-20240213143201-ec583247a57a, golang.org/x/image@v0.15.0, golang.org/x/mod@v0.15.0, golang.org/x/net@v0.21.0, golang.org/x/text@v0.14.0, golang.org/x/tools@v0.18.0, gonum.org/v1/gonum@v0.14.0, gonum.org/v1/plot@v0.14.0 - file_server/package.json: express@^4.18.2 ### Recent commits (newest first) - boxes - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - fixes - Hover complete - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - readme updated - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - Hovering Cursor - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - fixes - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - Fix bounding box error - try this one - reverted - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - here you go - Merge branch 'main' of https://github.com/TylerKerch/TreeHacks2024 - Image description - here you go - removed mentions about screenshots ## Key source files (fetched from GitHub, selected and truncated for size) ### file_server/package.json ``` { "name": "file_server", "version": "1.0.0", "description": "", "main": "server.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "start": "node server.js" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "express": "^4.18.2" } } ``` ### backend/go.mod ``` module treehacks/backend go 1.21.1 require ( git.sr.ht/~sbinet/gg v0.5.0 // indirect github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b // indirect github.com/aws/aws-sdk-go v1.50.20 // indirect github.com/campoy/embedmd v1.0.0 // indirect github.com/danlock/gogosseract v0.0.11-0ad3421 // indirect github.com/danlock/pkg v0.0.17-a9828f2 // indirect github.com/disintegration/imaging v1.6.2 // indirect github.com/go-fonts/liberation v0.3.2 // indirect github.com/go-latex/latex v0.0.0-20231108140139-5c1ce85aa4ea // indirect github.com/go-pdf/fpdf v0.9.0 // indirect github.com/goccmack/gocc v0.0.0-20230228185258-2292f9e40198 // indirect github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect github.com/google/uuid v1.6.0 // indirect github.com/gorilla/websocket v1.5.1 // indirect github.com/jerbob92/wazero-emscripten-embind v1.3.0 // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/lpernett/godotenv v0.0.0-20230527005122-0de1d4c5ef5e // indirect github.com/otiai10/gosseract/v2 v2.4.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/tetratelabs/wazero v1.5.0 // indirect golang.org/x/exp v0.0.0-20240213143201-ec583247a57a // indirect golang.org/x/image v0.15.0 // indirect golang.org/x/mod v0.15.0 // indirect golang.org/x/net v0.21.0 // indirect golang.org/x/text v0.14.0 // indirect golang.org/x/tools v0.18.0 // indirect gonum.org/v1/gonum v0.14.0 // indirect gonum.org/v1/plot v0.14.0 // indirect ) ``` ### file_server/server.js ```javascript const express = require('express'); const fs = require('fs'); const path = require('path'); const app = express(); const port = 3000; // Middleware to parse JSON bodies app.use(express.json({ limit: '50mb' })); // Adjust the limit as needed app.use(express.urlencoded({ limit: '50mb', extended: true })); app.use('/images', express.static('images')); // POST endpoint to upload a base64 encoded image app.post('/upload', (req, res) => { const { imageBase64, filename } = req.body; // Check if the imageBase64 and filename are provided if (!imageBase64 || !filename) { return res.status(400).send('Missing imageBase64 or filename in the request body.'); } // Decode the base64 image const imageBuffer = Buffer.from(imageBase64, 'base64'); // Define the path for the saved image const imagePath = path.join(__dirname, 'images', filename); // Save the image to the disk fs.writeFile(imagePath, imageBuffer, (err) => { if (err) { console.error('Failed to save the image:', err); return res.status(500).send('Failed to save the image.'); } res.send(`https://real-bug-pet.ngrok-free.app/images/${filename}`); }); }); // Start the server app.listen(port, () => { console.log(`Server listening at http://localhost:${port}`); }); ``` ### backend/main.go ```go package main import ( "context" "encoding/base64" "encoding/json" "errors" "fmt" "log" "net/http" "os" "strconv" "strings" "time" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/sagemakerruntime" "github.com/gorilla/websocket" "github.com/lpernett/godotenv" ) var upgrader = websocket.Upgrader{ ReadBufferSize: 1024, WriteBufferSize: 1024, CheckOrigin: func(r *http.Request) bool { return true }, } const PORT = 8080 type MessageContents struct { Type string `json:"type"` Payload string `json:"payload"` } const ( SCREENSHOT = "IMAGE" QUERY = "QUERY" CLEAR_BOUNDING_BOXES = "CLEAR" VOICE_OVER = "SPEAK" SELECT_BOX = "SELECT" BOUNDING_BOXES = "BOXES" HOVER = "HOVER" // Internal REINDEX = "REI" NOTHING = "NONE" // Image Description GPT4V_MODEL_ENGINE = "gpt-4-vision-preview" GPT4V_OPENAI_URL = "https://api.openai.com/v1/chat/completions" // Starting context window FRESH_CONTEXT_WINDOW = "You are a tool that aides the elderly in navigating their computers by helping them fulfill a goal (like 'watching a video about cats') by suggesting a next step. Your goal is to only output the next step towards reaching the final screen. Currently, your goal is to assist the user with this query that they've provided: GLOBAL_QUERY. If you have reached the final screen (that is, there isn't an action the user needs to take), say 'LAST STEP'. Below is the context of the task including steps that have been taken. CONTEXT: " ) var sagemaker_client *sagemakerruntime.SageMakerRuntime var previous_embedding []float64 = nil var previous_image []byte = nil var conn *websocket.Conn var current_screen_image string var current_global_query string // reset me on new query var step_channel chan (bool) = nil // reset me on new query var difference_detected chan (bool) = make(chan bool, 1) var current_step_count = 0 // reset me on new query var current_context_window string = "" // reset me on new query func UpdateContextWindow(global_query string) { current_context_window = strings.Replace(FRESH_CONTEXT_WINDOW, "GLOBAL_QUERY", global_query, 1) } func writeBack(messageType string, payload string) { err := conn.WriteJSON(MessageContents{ Type: messageType, Payload: payload, }) if err != nil { log.Println(err) } } func processMessage() error { wsMessageType, message, err := conn.ReadMessage() // Read a message from the WebSocket. if err != nil { return err } var incomingMessage MessageContents if wsMessageType == websocket.TextMessage { err := json.Unmarshal(message, &incomingMessage) if err != nil { return err } } else { return errors.New("WS message was not in a JSON form") } switch incomingMessage.Type { case SCREENSHOT: log.Print("Received screenshot") current_screen_image = incomingMessage.Payload decodedBytes, err := base64.StdEncoding.DecodeString(incomingMessage.Payload) if err != nil { return err } startTime := time.Now() result, err := sagemaker_client.InvokeEndpoint(&sagemakerruntime.InvokeEndpointInput{ Body: decodedBytes, EndpointName: aws.String("clip-image-model-2023-02-11-06-16-48-670"), ContentType: aws.String("application/x-image"), }) if err != nil { return errors.New("failed to call Sagemaker (CLIP) endpoint") } elapsedTime := time.Since(startTime) fmt.Printf("CLIP took %s to execute.\n", elapsedTime) embedding, err := ConvertBodyToVector(result.Body) if err != nil { return errors.New("failed to convert body to vector from (CLIP) model") } embedding = Normalize(embedding) current_image := decodedBytes fmt.Println("Normalized embedding") next_action := VOICE_OVER if previous_embedding != nil { next_action = CompareVectors(previous_embedding, embedding, previous_image, current_image) } previous_image = current_image previous_embedding = embedding fmt.Printf("Next action: %s\n", next_action) // If we're waiting for a subquery, we can't do anything else. if next_action != NOTHING { select { case difference_detected <- true: default: } } switch next_action { case NOTHING: go writeBack(NOTHING, "") return nil case REINDEX: go ReindexImage(incomingMessage.Payload) return nil case VOICE_OVER: go ReindexImage(incomingMessage.Payload) if current_step_count == 0 { voiceMessage := ImageDescription(incomingMessage.Payload) go writeBack(VOICE_OVER, voiceMessage) } return nil } case QUERY: if step_channel != nil { step_channel <- true } current_global_query = incomingMessage.Payload step_channel = make(chan bool) current_step_count = 0 UpdateContextWindow(current_global_query) go func() { for { select { case <-step_channel: log.Println("Finished this query. Giving up now.") return default: fmt.Println("Waiting for difference...") <-difference_detected fmt.Println("Difference detected!") // Event loop nextStep := GetQueryNextStep(QueryNextStepContext{ CurrentStep: current_step_count, CurrentScreenImage: current_screen_image, CurrentContextWindow: current_context_window, GlobalQuery: current_global_query, }) text := nextStep.Text // We're done. if text == "LAST STEP" || current_step_count > 10 { log.Println("Query finished.") step_channel <- true continue } closestBox := getClosestBox(current_screen_image, text) boxJSON, err := json.Marshal(closestBox) if err != nil { log.Println(err) } writeBack(SELECT_BOX, string(boxJSON)) writeBack(VOICE_OVER, nextStep.Audio) current_context_window += "\n" + nextStep.Text current_step_count++ // log.Println(current_context_window) } } }() case HOVER: dims := [truncated — 2349 more characters] ``` ### backend/clip-server/server.py ```python from flask import Flask, request, jsonify import torch from transformers import CLIPProcessor, CLIPModel from PIL import Image import base64 import io import json # Initialize Flask app app = Flask(__name__) # Initialize CLIP model_name = "openai/clip-vit-base-patch32" model = CLIPModel.from_pretrained(model_name) processor = CLIPProcessor.from_pretrained(model_name) @app.route('/tag-image', methods=['POST']) def tag_image(): data = request.json if 'image_base64' not in data: return "No image provided", 400 image_base64 = data['image_base64'] try: # Decode the Base64 encoded image image_data = base64.b64decode(image_base64) image = Image.open(io.BytesIO(image_data)) # Process the image for CLIP inputs = processor(images=image, return_tensors="pt") # Generate the embedding with torch.no_grad(): embeddings = model.get_image_features(**inputs) # Convert embedding tensor to list for JSON serialization embeddings_list = embeddings.tolist() # Return the embeddings as JSON print(embeddings_list) return jsonify({"embedding": embeddings_list}) except Exception as e: return str(e), 500 @app.route('/process-image', methods=['POST']) def process_image(): data = request.json print('here') if 'image_base64' not in data: return "No image provided", 400 if 'text_query' not in data: return "No text query provided", 400 if 'predictions' not in data: return "No predictions provided", 400 image_base64 = data['image_base64'] text_query = data['text_query'] predictions = data['predictions'] print(text_query) print(predictions) try: # Decode the Base64 encoded image image_data = base64.b64decode(image_base64) image = Image.open(io.BytesIO(image_data)) # Process the text query text_input = processor(text=text_query, return_tensors="pt", padding=True) text_features = model.get_text_features(**text_input) batch_size = 32 sub_image_batches = [] batch_predictions = [] for detection_id, prediction in enumerate(predictions): x, y, width, height = prediction['x'], prediction['y'], prediction['width'], prediction['height'] print(x,y,width,height) sub_image = image.crop((max(0,x-width/2), max(0,y-height/2), min(image.width, x + width/2), min(image.height, y + height/2))) sub_image_batches.append(sub_image) prediction['detection_id'] = detection_id batch_predictions.append(prediction) if len(sub_image_batches) == batch_size or detection_id == len(predictions) - 1: sub_image_inputs = processor(images=sub_image_batches, return_tensors="pt", padding=True) with torch.no_grad(): image_features = model.get_image_features(**sub_image_inputs) similarities = torch.nn.functional.cosine_similarity(image_features, text_features) for prediction, similarity in zip(batch_predictions, similarities): prediction['similarity'] = similarity.item() sub_image_batches = [] batch_predictions = [] sorted_predictions = sorted(predictions, key=lambda x: x['similarity'], reverse=True) print(sorted_predictions) return jsonify({'predictions': sorted_predictions}) except Exception as e: return str(e), 500 if __name__ == '__main__': app.run(port=8081, debug=True) ``` ### backend/s3_utils.go ```go package main import ( "bytes" "fmt" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/s3" ) // Bucket: pictag-public-testing-bucket func UploadObject(s3client *s3.S3, bucket string, key string, content []byte) (string, error) { input := &s3.PutObjectInput{ Bucket: aws.String(bucket), Key: aws.String(key), Body: bytes.NewReader(content), } _, err := s3client.PutObject(input) if err != nil { return "", err } return fmt.Sprintf("https://%s.s3.amazonaws.com/%s", bucket, key), nil } ``` ### backend/ocr.go ```go package main import ( "bytes" "context" "encoding/base64" "fmt" "os" "strings" "github.com/danlock/gogosseract" "golang.org/x/net/html" ) var pool *gogosseract.Pool // extractTextNodes traverses the DOM, extracts text nodes, and appends them to a slice. func extractTextNodes(n *html.Node, texts *[]string) { if n.Type == html.TextNode { // Trim the node data to remove extra spaces and newlines text := strings.TrimSpace(n.Data) if text != "hOCR text" { *texts = append(*texts, text) if len(*texts) > 2 && (*texts)[len(*texts)-1] == "" && (*texts)[len(*texts)-2] == "" && (*texts)[len(*texts)-3] == "" { *texts = append(*texts, "\n") } } } for c := n.FirstChild; c != nil; c = c.NextSibling { extractTextNodes(c, texts) } } // hOCRToText takes hOCR data as input and returns cleaned plain text. func hOCRToText(hocrData string) (string, error) { doc, err := html.Parse(strings.NewReader(hocrData)) if err != nil { return "", err } var texts []string extractTextNodes(doc, &texts) return strings.Join(texts, " "), nil } func textRecognition(ctx context.Context, base64String string) string { data, _ := base64.StdEncoding.DecodeString(base64String) reader := bytes.NewReader(data) hocr, _ := pool.ParseImage(ctx, reader, gogosseract.ParseImageOptions{ IsHOCR: true, }) text, _ := hOCRToText(hocr) fmt.Println(text) return text } func ocrSetUp(ctx context.Context) { trainingDataFile, _ := os.Open("eng.traineddata") cfg := gogosseract.Config{ Language: "eng", TrainingData: trainingDataFile, } // Create 10 Tesseract instances that can process image requests concurrently. pool, _ = gogosseract.NewPool(ctx, 10, gogosseract.PoolConfig{Config: cfg}) } func ocrClosePool() { pool.Close() } ``` ### backend/vector_processing.go ```go package main import ( "encoding/json" "fmt" "gonum.org/v1/gonum/mat" "image" _ "image/jpeg" _ "image/png" "log" // "math" "bytes" ) func ConvertBodyToVector(body []byte) ([]float64, error) { var result [][]float64 if err := json.Unmarshal(body, &result); err != nil { return nil, err } return result[0], nil } func Normalize(v []float64) []float64 { vec := mat.NewVecDense(len(v), v) // Compute the l2 norm (Euclidean norm) norm := mat.Norm(vec, 2) // Normalize the vector if norm != 0 { vec.ScaleVec(1/norm, vec) } return vec.RawVector().Data } // func dotProduct(vectorA, vectorB []float64) float64 { // var sum float64 // for i := range vectorA { // sum += vectorA[i] * vectorB[i] // } // return sum // } // // Function to calculate the magnitude (or norm) of a vector // func magnitude(vector []float64) float64 { // var sum float64 // for _, v := range vector { // sum += v * v // } // return math.Sqrt(sum) // } // // Function to calculate cosine similarity between two vectors // func cosineSimilarity(vectorA, vectorB []float64) float64 { // dot := dotProduct(vectorA, vectorB) // magA := magnitude(vectorA) // magB := magnitude(vectorB) // return dot / (magA * magB) // } // bytesToImage converts a byte slice into an image.Image. func bytesToImage(b []byte) (image.Image, error) { reader := bytes.NewReader(b) img, _, err := image.Decode(reader) if err != nil { return nil, err } return img, nil } // compareImages compares two images pixel by pixel and returns a score based on the similarity. func compareVectors(imgData1 []byte, imgData2 []byte) float64 { img1, err := bytesToImage(imgData1) if err != nil { log.Fatalf("Failed to convert bytes to image for imgData1: %v", err) } img2, err := bytesToImage(imgData2) if err != nil { log.Fatalf("Failed to convert bytes to image for imgData2: %v", err) } bounds1 := img1.Bounds() bounds2 := img2.Bounds() if bounds1.Dx() != bounds2.Dx() || bounds1.Dy() != bounds2.Dy() { log.Fatalf("Images are of different sizes: %v vs %v", bounds1.Size(), bounds2.Size()) } var similarPixels int totalPixels := bounds1.Dx() * bounds1.Dy() for y := bounds1.Min.Y; y < bounds1.Max.Y; y++ { for x := bounds1.Min.X; x < bounds1.Max.X; x++ { r1, g1, b1, a1 := img1.At(x, y).RGBA() r2, g2, b2, a2 := img2.At(x, y).RGBA() if r1 == r2 && g1 == g2 && b1 == b2 && a1 == a2 { similarPixels++ } } } score := (float64(similarPixels) / float64(totalPixels)) * 100 return score } func CompareVectors(v1 []float64, v2 []float64, b1 []byte, b2 []byte) string { // similarity := cosineSimilarity(v1, v2) similarity := compareVectors(b1, b2) fmt.Println(similarity) if similarity < 85 { return VOICE_OVER } if similarity < 95 { return REINDEX } return NOTHING } ``` ### backend/image_segment.go ```go package main import ( "bytes" "encoding/json" "errors" "fmt" "io" "net/http" // "image/jpeg" // "log" // "sync" ) type TagBoxesPayload struct { ImageBase64 string `json:"image_base64"` TextQuery string `json:"text_query"` Predictions []BoundingBox `json:"predictions"` } type TagBoxesResponse struct { Predictions []CLIPPrediction `json:"predictions"` } type CLIPPrediction struct { X float64 `json:"x"` Y float64 `json:"y"` Width float64 `json:"width"` Height float64 `json:"height"` Class string `json:"class"` DetectionId string `json:"detection_id"` Similarity float64 `json:"similarity"` } func tagImageBoxes(b64image string, textQuery string) ([]CLIPPrediction, error) { // Construct the payload payload := TagBoxesPayload{ ImageBase64: b64image, TextQuery: textQuery, Predictions: boundingBoxes, } // Marshal the payload into JSON jsonData, err := json.Marshal(payload) if err != nil { fmt.Println("Error marshaling JSON:", err) return nil, errors.New(err.Error()) } // Make the HTTP POST request resp, err := http.Post("http://localhost:8081/process-image", "application/json", bytes.NewBuffer(jsonData)) if err != nil { fmt.Println("Error making request:", err) return nil, errors.New(err.Error()) } defer resp.Body.Close() // Read the response body body, err := io.ReadAll(resp.Body) if err != nil { fmt.Println("Error reading response body:", err) return nil, errors.New(err.Error()) } var tags TagBoxesResponse err = json.Unmarshal(body, &tags) if err != nil { fmt.Println("Error: ", err) return nil, errors.New(err.Error()) } fmt.Println("Response:", len(tags.Predictions)) return tags.Predictions, nil // Decode the base64 image // img, err := decodeBase64Image(b64image) // if err != nil { // log.Fatalf("Failed to decode base64 image: %v", err) // } // var wg sync.WaitGroup // embeddings := make([][]float64, len(predictions)) // for i, prediction := range predictions { // wg.Add(1) // Increment the WaitGroup counter // go func(i int, prediction Prediction) { // defer wg.Done() // Decrement the counter when the goroutine completes // boundingBox := image.Rect(int(prediction.X-prediction.Width/2), int(prediction.Y-prediction.Height/2), int(prediction.X+prediction.Width/2), int(prediction.Y+prediction.Height/2)) // // Crop the image using the bounding box // croppedImg, err := cropImage(img, boundingBox) // if err != nil { // log.Fatalf("Failed to crop image: %v", err) // } // // Encode the cropped image to a format (e.g., JPEG) and save it to a file // var buf bytes.Buffer // if err := jpeg.Encode(&buf, croppedImg, nil); err != nil { // log.Fatalf("Failed to encode cropped image: %v", err) // } // croppedImageBytes := buf.Bytes() // embedding, err := tagImage(croppedImageBytes) // if err != nil { // log.Fatalf("Failed to tag cropped image: %v", err) // } // embeddings[i] = embedding // }(i, prediction) // Pass the current loop variables as arguments to the goroutine // } // wg.Wait() // Wait for all goroutines to complete // print(len(embeddings), embeddings[0][0]) } ``` ### backend/query.go ```go package main import ( "bytes" "encoding/json" "errors" "fmt" "io" "log" "net/http" "os" ) type QueryStep struct { Text string `json:"step"` Audio string `json:"audio"` Err error `json:"error"` } type QueryNextStepContext struct { GlobalQuery string `json:"global_query"` CurrentStep int `json:"current_step"` CurrentScreenImage string `json:"current_screen_image"` CurrentContextWindow string `json:"current_context_window"` } func GetQueryNextStep(args QueryNextStepContext) QueryStep { context := args.CurrentContextWindow current_step := args.CurrentStep current_screen_image := args.CurrentScreenImage current_global_query := args.GlobalQuery prompt := fmt.Sprintf("I am on the following page. I want to explain to a friend '%s'. Tell me just the first step to achieve this, mentioning a button that may complete the action if it exists. Be brief. If there's a good chance the action we take will finish the task, say 'LAST STEP' with no other text, but otherwise do not.", current_global_query) if current_step != 0 { prompt = fmt.Sprintf("I am on the following page. I want to explain to a friend '%s'. Tell me just the first step to achieve this and get to the next step, mentioning a button that may complete the action if it exists. Be brief. If there's a good chance the action we take will finish the task, say 'LAST STEP' with no other text, but otherwise do not.", current_global_query) } maxTokens := 2048 var headers = map[string]string{ "Authorization": "Bearer " + os.Getenv("OPEN_AI_API_KEY"), "Content-Type": "application/json", } image_url := UploadBase64Image(current_screen_image) data := map[string]interface{}{ "model": GPT4V_MODEL_ENGINE, "messages": []map[string]interface{}{ {"role": "system", "content": context}, { "role": "user", "content": []map[string]string{ {"type": "text", "text": prompt}, {"type": "image_url", "image_url": image_url}, }, }, }, "max_tokens": maxTokens, } jsonData, err := json.Marshal(data) if err != nil { return QueryStep{Err: errors.New("error marshaling struct to pass to GPT4")} } req, err := http.NewRequest("POST", GPT4V_OPENAI_URL, bytes.NewBuffer(jsonData)) if err != nil { return QueryStep{Err: errors.New("error creating request")} } log.Println("Called OpenAI") for key, value := range headers { req.Header.Add(key, value) } client := &http.Client{} resp, err := client.Do(req) if err != nil { return QueryStep{Err: errors.New("error actually sending request")} } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { return QueryStep{Err: errors.New("error reading from IO")} } log.Println(string(body)) type ApiResponse struct { Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` } `json:"choices"` Index int `json:"index"` } var apiResponse ApiResponse if err := json.Unmarshal(body, &apiResponse); err != nil { return QueryStep{Err: errors.New("error unmarshaling response body for query")} } if len(apiResponse.Choices) < 1 { return QueryStep{Err: errors.New("error with OpenAI API: " + string(body))} } content := apiResponse.Choices[0].Message.Content return QueryStep{Text: content, Audio: content, Err: nil} } ``` [16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]