# Project export: UrbanQuest This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Urban Quest: Explore the world through interactive storytelling and image-based discovery. - Devpost: https://devpost.com/software/urbanquest - GitHub: https://github.com/arricsekhon/urban-Quest - Team: 1 GitHub contributor(s) — Arric Sekhon (2 commits) ## Devpost submission (written by the team) ### Inspiration The inspiration behind Urban Quest came from the desire to blend everyday exploration with the power of AI. We wanted to create an interactive platform where users could snap photos of their surroundings and instantly learn more about what they captured. Whether it’s an interesting landmark, an object, or a question about the environment, Urban Quest makes discovery and learning fun and accessible. We aimed to bring adventure and curiosity into the palms of our users. ### What it does Urban Quest allows users to take photos and ask questions about the content in the images. Using AI, the app processes the photos, analyzes the visual elements, and provides insightful responses. Whether it's identifying objects, explaining a scene, or answering specific questions related to the captured image, Urban Quest brings context and understanding to the world around the user. It’s a blend of exploration and knowledge sharing, designed for curious minds. ### How we built it We built Urban Quest using Swift and integrated advanced AI technologies like Google Vision and Gemini API for image recognition and content generation. The app's core functionality revolves around the seamless interaction between the camera, the AI model, and the user interface. We used cloud-based AI models to handle the heavy lifting of image analysis and natural language processing, ensuring fast and accurate responses. The app’s design focuses on simplicity, allowing users to easily capture images, ask questions, and receive answers in real-time. ### Challenges we ran into One of the biggest challenges we faced was integrating AI models to effectively understand and process the images, especially in real-time. Ensuring smooth performance while handling large images and making API calls was difficult. Another challenge was creating a user-friendly experience that balanced the power of AI with simplicity. We also had to carefully manage user privacy, particularly when accessing the camera and photo library, while adhering to strict data security protocols. ### Accomplishments we're proud of We’re proud of successfully combining image capture, AI-based analysis, and interactive storytelling in a single, cohesive app. The smooth integration of the camera functionality with real-time AI processing was a significant achievement. We also take pride in making a highly interactive and educational tool that can engage users in a fun and meaningful way. Creating an intuitive user interface that can bring such advanced technology to the everyday user is another highlight. ### What we learned Throughout the development process, we learned a lot about AI integration, especially in mobile applications. We deepened our understanding of how to process visual data and convert it into meaningful responses using AI models. Managing real-time API calls, handling image data, and optimizing app performance across different devices provided valuable insights. We also learned how important user experience is, especially when dealing with complex technology—keeping things simple and easy to use was key. ### What's next for Urban Quest Moving forward, we plan to expand Urban Quest’s capabilities by adding more advanced image recognition features, such as real-time object tracking and expanded visual analysis. We also aim to incorporate social features where users can share their discoveries and learn from others. Additionally, we want to explore gamification elements to make the learning process even more engaging. Lastly, we’re looking to enhance the AI model's ability to understand even more complex queries and provide richer, more detailed responses. ## README (from the GitHub repository) # Urban Quest **Urban Quest** is an interactive mobile app that allows users to capture images and ask questions about the content. With real-time AI-generated insights, Urban Quest makes everyday exploration fun and educational. ## Features - **Capture Images**: Use your camera to capture moments or objects. - **AI-Powered Responses**: Ask questions about the images and get detailed AI-generated responses. - **Seamless Integration**: Effortlessly switch between capturing photos and receiving educational insights. ## How to Use - **Capture an Image**: Open the app and use the camera feature to capture an image. - **Ask a Question**: Type your question related to the captured image in the text field. - **Get Responses**: The AI model will analyze the image and provide a meaningful response. ## Technologies Used - SwiftUI for the app interface. - Google Cloud Vision for image recognition. - Google Generative AI for text-based responses. ## Detected evidence (automated analysis) Indexed codebase: 8 recognized source files, 18 KB. - Swift (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (19 of 19) ``` README.md urban-Quest.xcodeproj/project.pbxproj urban-Quest.xcodeproj/project.xcworkspace/contents.xcworkspacedata urban-Quest.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved urban-Quest.xcodeproj/xcuserdata/arricsekhon.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist urban-Quest.xcodeproj/xcuserdata/arricsekhon.xcuserdatad/xcschemes/xcschememanagement.plist urban-Quest/APIkey.swift urban-Quest/Assets.xcassets/AccentColor.colorset/Contents.json urban-Quest/Assets.xcassets/AppIcon.appiconset/Contents.json urban-Quest/Assets.xcassets/Contents.json urban-Quest/ContentView.swift urban-Quest/GenerativeAI-Info.plist urban-Quest/Info.plist urban-Quest/MenuView.swift urban-Quest/Preview Content/Preview Assets.xcassets/Contents.json urban-Quest/urban_QuestApp.swift urban-QuestTests/urban_QuestTests.swift urban-QuestUITests/urban_QuestUITests.swift urban-QuestUITests/urban_QuestUITestsLaunchTests.swift ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Create README.md - Initial Commit ## Key source files (fetched from GitHub, selected and truncated for size) ### urban-Quest/urban_QuestApp.swift ```swift // // urban_QuestApp.swift // urban-Quest // // Created by Arric Sekhon on 19/10/24. // import SwiftUI @main struct urban_QuestApp: App { var body: some Scene { WindowGroup { ContentView() } } } ``` ### urban-QuestTests/urban_QuestTests.swift ```swift // // urban_QuestTests.swift // urban-QuestTests // // Created by Arric Sekhon on 19/10/24. // import Testing @testable import urban_Quest struct urban_QuestTests { @Test func example() async throws { // Write your test here and use APIs like `#expect(...)` to check expected conditions. } } ``` ### urban-Quest/APIkey.swift ```swift import Foundation enum APIKey { // Fetch the API key from `GenerativeAI-Info.plist` static var `default`: String { guard let filePath = Bundle.main.path(forResource: "GenerativeAI-Info", ofType: "plist") else { fatalError("Couldn't find file 'GenerativeAI-Info.plist'.") } let plist = NSDictionary(contentsOfFile: filePath) guard let value = plist?.object(forKey: "API_KEY") as? String else { fatalError("Couldn't find key 'API_KEY' in 'GenerativeAI-Info.plist'.") } if value.starts(with: "_") { fatalError( "Follow the instructions at https://ai.google.dev/tutorials/setup to get an API key." ) } return value } } ``` ### urban-QuestUITests/urban_QuestUITestsLaunchTests.swift ```swift // // urban_QuestUITestsLaunchTests.swift // urban-QuestUITests // // Created by Arric Sekhon on 19/10/24. // import XCTest final class urban_QuestUITestsLaunchTests: XCTestCase { override class var runsForEachTargetApplicationUIConfiguration: Bool { true } override func setUpWithError() throws { continueAfterFailure = false } @MainActor func testLaunch() throws { let app = XCUIApplication() app.launch() // Insert steps here to perform after app launch but before taking a screenshot, // such as logging into a test account or navigating somewhere in the app let attachment = XCTAttachment(screenshot: app.screenshot()) attachment.name = "Launch Screen" attachment.lifetime = .keepAlways add(attachment) } } ``` ### urban-Quest/MenuView.swift ```swift // // MenuView.swift // urban-Quest // // Created by Arric Sekhon on 19/10/24. // import SwiftUI struct SideMenu: View { var body: some View { VStack(alignment: .leading) { // Add items in your side menu Text("Menu Item 1") .padding(.top, 20) .font(.headline) Text("Menu Item 2") .padding(.top, 20) Text("Menu Item 3") .padding(.top, 20) Spacer() } .frame(width: 250) // Adjust the width of the side menu .padding(.leading, 30) .background(Color.white) .cornerRadius(20) // Optional: To round the corners of the menu .shadow(radius: 5) // Optional: Add shadow for a nice effect } } struct SideMenu_Previews: PreviewProvider { static var previews: some View { SideMenu() } } ``` ### urban-QuestUITests/urban_QuestUITests.swift ```swift // // urban_QuestUITests.swift // urban-QuestUITests // // Created by Arric Sekhon on 19/10/24. // import XCTest final class urban_QuestUITests: XCTestCase { override func setUpWithError() throws { // Put setup code here. This method is called before the invocation of each test method in the class. // In UI tests it is usually best to stop immediately when a failure occurs. continueAfterFailure = false // In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this. } override func tearDownWithError() throws { // Put teardown code here. This method is called after the invocation of each test method in the class. } @MainActor func testExample() throws { // UI tests must launch the application that they test. let app = XCUIApplication() app.launch() // Use XCTAssert and related functions to verify your tests produce the correct results. } @MainActor func testLaunchPerformance() throws { if #available(macOS 10.15, iOS 13.0, tvOS 13.0, watchOS 7.0, *) { // This measures how long it takes to launch your application. measure(metrics: [XCTApplicationLaunchMetric()]) { XCUIApplication().launch() } } } } ``` ### urban-Quest/ContentView.swift ```swift import SwiftUI import GoogleGenerativeAI import UIKit import AVFoundation import Photos import GoogleCloudVision // A struct for managing the image picker struct ImagePicker: UIViewControllerRepresentable { @Binding var selectedImage: UIImage? @Binding var isPickerPresented: Bool class Coordinator: NSObject, UINavigationControllerDelegate, UIImagePickerControllerDelegate { let parent: ImagePicker init(parent: ImagePicker) { self.parent = parent } func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) { if let image = info[.originalImage] as? UIImage { parent.selectedImage = image } parent.isPickerPresented = false } func imagePickerControllerDidCancel(_ picker: UIImagePickerController) { parent.isPickerPresented = false } } func makeCoordinator() -> Coordinator { Coordinator(parent: self) } func makeUIViewController(context: Context) -> UIImagePickerController { let picker = UIImagePickerController() picker.delegate = context.coordinator picker.sourceType = .camera return picker } func updateUIViewController(_ uiViewController: UIImagePickerController, context: Context) { } } struct ContentView: View { let model = GenerativeModel(name: "gemini-1.5-flash-002", apiKey: APIKey.default) @State private var isMenuOpen = false @State var userPrompt = "" @State var isLoading = false @State var isFirstLoad = true @State var uQresponse = false @State var isPickerPresented = false @State var selectedImage: UIImage? // Single array to store pairs of user inputs and responses @State var chatHistory: [(input: String, response: String, image: UIImage?)] = [] var body: some View { ZStack { // Main Content with Background Color Color.white.opacity(0.2) .edgesIgnoringSafeArea(.all) .onTapGesture { if isMenuOpen { withAnimation { isMenuOpen = false } } } VStack { // Menu and Header HStack { Button(action: { withAnimation { self.isMenuOpen.toggle() } }) { Image(systemName: "line.horizontal.3") .resizable() .frame(width: 30, height: 20) .padding() .foregroundColor(.black) } Spacer() Text("UQ") .font(.title) .fontWeight(.bold) .foregroundColor(.black) Spacer() Button(action: { // Action for new quest }) { Image(systemName: "pencil") .resizable() .frame(width: 20, height: 20) .foregroundColor(.black) } } .padding(.horizontal) Spacer() // Scrollable list for displaying chat history if isFirstLoad { VStack { Spacer() Text("Urban Quest") .font(.largeTitle) .fontWeight(.bold) .multilineTextAlignment(.center) Text("Explore the world") .font(.subheadline) .multilineTextAlignment(.center) Spacer() } .frame(maxWidth: .infinity, maxHeight: .infinity) } else { ScrollViewReader { scrollViewProxy in ScrollView { VStack(spacing: 15) { ForEach(0..