# Project export: WorldDex This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: WorldDex is a real-life Pokédex: on our mobile app, you can scan and capture any object, talk to your collection, and share your experiences (using Computer Vision, LLMs, Web3, and more)! - Devpost: https://devpost.com/software/worlddex-6diekg - GitHub: https://github.com/antqin/WorldDex - Video: https://www.youtube.com/embed/LRpubtPBChA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([MLH - Hedera] Best Blockchain Project Using Hedera; Best Web3 Hack) - Team: 4 GitHub contributor(s) — Anthony Qin (21 commits), Jaiden Reddy (19 commits), vivekvajipey (12 commits), Andrew Liu (4 commits) ## Devpost submission (written by the team) ### Inspiration As we go about our lives, all of us experience so much, witness so much, learn so much. When we were kids, exploration and excitement were a crucial part of every day, but as adults, responsibility can cause us to lose some of that magic. We wanted to remind each other that although we may not live in the Kanto region, the Earth still offers an enormous amount to see—and infinite amount to collect. So, we built WorldDex. WorldDex is a real-life PokeDex (and so much more) that lets you scan and save anything into an ever-growing collection, offering a portal for you to learn, share, enjoy, and remember. ### What it does WorldDex is primarily available through our mobile iOS application. Via this application, you can take a picture of any object and speak the name of what is being scanned into the interface. Be careful though! You will have a limited number of catch tries per day. Whether it's a simple lamp on your desk or a rare species of salamander, our app analyzes your image and spoken label, computing a catch probability which informs a roll of the dice. If you managed to catch your item, it is automatically added to your collection both in-app and on the blockchain. You also have the opportunity to record some information about where you were, who you were with, and any other details you're excited to include about your capture. WorldDex allows you to have a spoken or written conversation with your collection about the items you have caught. All you need is a microphone, a speaker, and an object; from there, our intelligent models can tell you all about your item. This puts the knowledge of the world's best LLMs at your fingertips, just one scan away from a detailed educational discussion about some of your favorite memories. Once you have completed a scan, WorldDex creates an NFT of your image. No need to panic—we are not crypto devs gearing for a rug-pull. Instead, the NFT means your ownership of that scan is provable and trade-able. Yes, someone can screenshot your NFT, but they will not be able to get it into their WorldDex! On the community page, you can check out other people's scans and take a look at what your family, friends, and acquaintances have been collecting—in real time! Your WorldDex account also comes with access to our web application. There, you can access a more dynamic view of your collection. Coming soon to the web version: detailed analytics, rarities, trading, and more! ### How we built it The workhorse of our mobile application is written in Swift, with iOS native speech-to-text translation and tons of custom styling and design. The app interacts with a variety of powerful back-ends: OBJECT DETECTION, CAPTURE, AND PROBABILITY are handled by an AWS EC2 instance with a Deep Learning AMI running PyTorch 2.0 with an NVIDIA T4 GPU. The instance runs inference on a state-of-the-art pre-trained zero-shot object detection model called GroundingDINO (Liu et al., 2023). Associated technology: GroundingDINO, AWS EC2, PyTorch, iOS speech-to-text, GPT-3.5 Turbo Instruct, GPT 4 A flask server runs on the EC2 instance. When the user takes a scan and speaks into their phone, iOS speech-to-text transcribes the user's message and sends both the image and the user's provided information to the back-end. A flask server runs on the EC2 instance. When the user takes a scan and speaks into their phone, iOS speech-to-text transcribes the user's message and sends both the image and the user's provided information to the back-end. We use GPT-3.5 Turbo Instruct (instruct-tuned version of GPT exceptionally good at executing novel, precise instructions) to extract the object label within the user's spoken text. With some careful prompt engineering and in-context learning, the label extraction works like a charm even when given complex and long audio transcriptions. We use GPT-3.5 Turbo Instruct (instruct-tuned version of GPT exceptionally good at executing novel, precise instructions) to extract the object label within the user's spoken text. With some careful prompt engineering and in-context learning, the label extraction works like a charm even when given complex and long audio transcriptions. GroundingDINO is a pre-trained transformer-based object detection model that can predict on any natural language label (rather than only on labels it was trained on), which enables WorldDex users to collect anything they can dream of! GroundingDINO is a pre-trained transformer-based object detection model that can predict on any natural language label (rather than only on labels it was trained on), which enables WorldDex users to collect anything they can dream of! GroundingDINO uses the label spoken by the user as a prompt to search for all instances of their desired object in the photo taken on their phone. It assigns these instances with a confidence score, and the instance with the max confidence score (logits) is bounded and scanned. In WorldDex, the success of a catch is a function of this confidence score. This bounding box is later used in the mobile app to crop a perfect thumbnail image for the object. GroundingDINO uses the label spoken by the user as a prompt to search for all instances of their desired object in the photo taken on their phone. It assigns these instances with a confidence score, and the instance with the max confidence score (logits) is bounded and scanned. In WorldDex, the success of a catch is a function of this confidence score. This bounding box is later used in the mobile app to crop a perfect thumbnail image for the object. DATA STORAGE AND BLOCKCHAIN INTERACTIONS are handled by an intricate JavaScript microservice architecture. This back-end uses Node.js and Express to route across a variety of on and off-chain services. Associated Technology: Hedera, CockroachDB, Express, web3.storage, Ethereum, IPFS Database information (user account info, friends, image metadata, images, etc) are stored in a server-less CockroachDB instance. Through our endpoints, the back-end writes SQL queries to properly interact with the DB. Database information (user account info, friends, image metadata, images, etc) are stored in a server-less CockroachDB instance. Through our endpoints, the back-end writes SQL queries to properly interact with the DB. When a catch is confirmed, the back-end uses the web3.storage framework to deploy the image onto IPFS for decentralized storage. Although a copy is maintained in our internal database for quick access, this grants the user the ability to share and interact with their image external to our centralized application. When a catch is confirmed, the back-end uses the web3.storage framework to deploy the image onto IPFS for decentralized storage. Although a copy is maintained in our internal database for quick access, this grants the user the ability to share and interact with their image external to our centralized application. After a catch has been hosted by IPFS, our custom WLDD Token programmatically deployed on the open-source PoS Hedera network's testnet is used to mint an original NFT based on the image. This NFT contains in on-chain metadata a pointer to the IPFS metadataCID. The Hedera network offers straightforward and highly cost-effective functionality. After a catch has been hosted by IPFS, our custom WLDD Token programmatically deployed on the open-source PoS Hedera network's testnet is used to mint an original NFT based on the image. This NFT contains in on-chain metadata a pointer to the IPFS metadataCID. The Hedera network offers straightforward and highly cost-effective functionality. Once the image has been stored on IPFS and an associated NFT has been minted, the unique ID we generate per image is granted to the user's collection on our custom Ethereum smart contract deployed on the Sepolia Testnet. Why the need for interactions with Ethereum AND Hedera? Our Ethereum smart contract allows for our own two-party trading functionality. However, we plan to deploy this EVM byte code to the Hedera network to simplify in the future. Once the image has been stored on IPFS and an associated NFT has been minted, the unique ID we generate per image is granted to the user's collection on our custom Ethereum smart contract deployed on the Sepolia Testnet. Why the need for interactions with Ethereum AND Hedera? Our Ethereum smart contract allows for our own two-party trading functionality. However, we plan to deploy this EVM byte code to the Hedera network to simplify in the future. CHATTING WITH COLLECTION is handled by another API hosted on the same GPU-accelerated EC2 instance used by the models. When the user records an audio message in the mobile app, we use iOS speech to text to transcribe the message and send it to this API. The API uses GPT-4 to determine the response, with additional context retrieved from the database's image labels and metadata. Then, it uses the state-of-the-art text-to-speech model from ElevenLabs (with streaming functionality for minimized latency) to read out the model's response and hold a conversation with the user. Associated Technology: GPT-4, ElevenLabs TTS, AWS EC2 COMMUNITY & PROFILE is built on the frontend using Swift and portrays your community's captures in real time. You can see people's captured items, their capture location, time of capture, probability of capture, and live audio transcription notes. Each user's captures are linked to their unique accounts, for which they can sign up for an account or log in on the profile page of the app. The frontend is linked to the CockroachDB database, making GET and POST requests to display community posts and manage creation of profiles. Associated Technology: Swift, XCode, CockroachDB THE WEB APPLICATION is built with Reflex and backed by the same JavaScript API and databases as the other front-end. Here, we display a more dynamic interface meant to be more nostalgic of opening the PokeDex upon logging in to your account. However, we also wanted to include a web app to give us flexibility for future extensions, with the ability to display detailed visualizations, analytics, trade, and more. Associated Technology: Reflex We know that's a lot of text and a lot of technologies. So: The TLDR is: Your image and label are sent to a Python back-end, which use a combination of models to determine if you actually correctly identified what you scanned. If you did (and get a bit lucky), your catch completes. Your image and label are sent to a Python back-end, which use a combination of models to determine if you actually correctly identified what you scanned. If you did (and get a bit lucky), your catch completes. At this point, we store a copy in a centralized CockroachDB server less database. We also store your image in a decentralized manner on IPFS, mint you an NFT on Hedera associated with the IPFS metadata, and register your ownership on Ethereum in case you want to start trading. At this point, we store a copy in a centralized CockroachDB server less database. We also store your image in a decentralized manner on IPFS, mint you an NFT on Hedera associated with the IPFS metadata, and register your ownership on Ethereum in case you want to start trading. The user can chat with an item from their collection. Their audio message is transcribed and sent to a back-end, which asks GPT-4 for a prompted response and plays this back with ElevenLabs Text-to-Speech for the user. The user can chat with an item from their collection. Their audio message is transcribed and sent to a back-end, which asks GPT-4 for a prompted response and plays this back with ElevenLabs Text-to-Speech for the user. Users can share their captures and see other's captures in the community page. Each user has a unique password-protected account with their captures that they can create/login to on the account page. Users can share their captures and see other's captures in the community page. Each user has a unique password-protected account with their captures that they can create/login to on the account page. There's also a Reflex web application, from which you can check out your collection upon logging in. In the future, it offers a great home for more complex analytics and displays. There's also a Reflex web application, from which you can check out your collection upon logging in. In the future, it offers a great home for more complex analytics and displays. DOUBLE TAP! Check out our Sepolia Ethereum Smart Contract (code in the GitHub Repo) governing trading. Check out what's being minted on our custom Hedera WLDD Token. Check out one of our favorite IPFS hosted images, which was recently collected and minted! ### Challenges we ran into HIGH LATENCY Our APIs are highly complex and interact with a variety of frameworks. This introduces latency and robustness issues. Specifically, on a successful catch, we have to: Perform speech-to-text with iOS built-in functionality Analyze text using fine-tuned GPT 3.5 Identify instances of an object type using GroundingDINO and rank them by probability, selecting the highest probability one and determining if a catch will happen Writing caught image and associated metadata to CockroachDB Deploying caught image to IPFS Minting NFT including IPFS metadata CID on Hedera Transferring NFT from treasury to user Logging catch onto Ethereum smart contract To solve this problem, we divided this functionality into two portions: low-latency required and high-latency allowed. The user needs to know if they caught their object, which requires steps 1-3 to happen rapidly. To adjust, we used built-in iOS speech-to-text instead of a more general-purpose transcription model like the one offered by AssemblyAI (we tested it, but latency was too high). We used a fine-tuned version of GPT 3.5 Turbo Instruct, which is geared towards novel instruction types to reduce the performance tradeoff you get from avoiding the cost and time required by GPT-4. We also hosted on an AWS EC2 instance using a T4 GPU to speed up GroundingDINO computation. Once the user has caught their item, we display a cached version of the image, allowing our back-end time to handle the higher-latency blockchain and database interactions. EXPLODING COMPLEXITY Our three back-ends threatened to create an exploding complexity problem which would have made debugging impossible. However, by siloing endpoints and routing related calls to more abstracted sections handling functionality, we were able to develop iteratively and minimize failure. We have: A set of independent endpoints handling Ethereum smart contract interactions A set of independent functions handling IPFS deployment A set of independent functions handling Hedera minting and transfer A set of endpoints wrapping IPFS deployment, Hedera interactions, and Ethereum calls into a single abstraction. A set of endpoints wrapping serverless CockroachDB functionality and caching. We used a similar approach to carefully create an effective back-end for our AWS EC2 instance. INTUITIVE UI Building simple, intuitive UI for a social app with so much functionality and unique technologies was a big challenge. From figuring out the best flow of user experience to adding the smallest things that provided users with more information about what's happening behind the scenes (e.g. loading v.s. empty screen, flashing dot for speaking AI), we invested a lot of time in optimizing the human-computer interaction (HCI) and designing a novel and forward-looking interface that leverages large language models (LLMs). Accomplishments that we are proud of User Interface and Design We had a vision for our application—to interweave the joy of exploration with the extraordinary nature of everyday life. Although our mobile app, back-ends, and models are far from perfect, we are incredibly excited at just how fun it is for the four of us to scan random objects, talk to our collection, and share what we have discovered. Integration of New Technologies We rapidly prototyped and iterated over a wide variety of new technologies. We knew just how extensible a real-life PokeDex could be; there were infinite directions to take! Using the sponsor list and our own field knowledge as inspiration, we tested out different APIs, frameworks, and models until we found the combination that matched our vision. Although this means we took an uncommon and often complex approach to our project, it has also resulted in some very cool functionality. Purposeful Incorporation of AI and Crypto We find using the blockchain or artificial intelligence simply because they exist to be a silly approach. Instead, as we read about different new technologies, we asked ourselves how their design aligns with our vision for WorldDex. We used a variety of state-of-the-art models for a plethora of functions, but nothing is casual; difference instances of GPT help to contextualize or explain, object-detection models... detect objects, text-to-speech models bring alive the PokeDex we remember from watching Cartoon Network as kids. Meanwhile, decentralization lets us put the collection in the hands of the users. If we could, the model would be running on the blockchain too (trust us - we looked into some ideas related to that). While NFTs and smart contracts allow for trading, monetization, and a market economy, they more fundamentally mean that when you collect a picture, it's verifiably yours. Hedera lets us do these mints at a remarkably low cost. ### What we learned It is hard to describe how much we learned in such a short period of time! We are frankly shocked at how many different technologies and functionalities we were able to weave into the same project in less than two days. Intuitive UI, UX, and HCI We experimented a lot on our UI through user testing and feedback, especially in the more novel aspects of the app (e.g. LLM and voice integration). We learned about the value of simplicity but also the value of information, finding a balance between keeping the experience enjoyable and intuitive while providing the user with all the information they needed to fully leverage the app. Model Comparison Shopping We learned a ton about different state-of-the-art models and their relative tradeoffs. We had to select the best models to our latency, accuracy, cost, and storage requirements. While we all know a lot about machine learning and artificial intelligence in the academic environment, watching the impact of different real-world models on these crucial in-production outcomes was fascinating. Crypto and Blockchain Integrations It is becoming increasingly clear that while adoption of crypto nosedives, technology and infrastructure for the field is exploding. We are ridiculously excited at how efficiently new crypto protocols and chains are offering high levels of programmability at low costs without sacrificing core blockchain principles. While we might still be most comfortable in Solidity, the once-painful smart contract programming process has gotten so much easier. Databases and Back-end Integrations It is remarkable how easy it is to get up and running on a database like CockroachDB, with direct SQL queries after minimal setup. High-performance back-ends interacting with a variety of different APIs and data storage solutions are increasingly viable every day. It was so much fun to construct one (or a couple) and watch them work in real-time. And finally, a lot of what we learned and are excited about happens in what we did not use. In the future, we want to spend more time exploring vector databases for RAG, increased privacy chains and crypto-native languages, and more when extending this project and others. ### What's next We want to expand on the community. We imagine planned events in which scavenger hunts and collection events ask users to capture a certain subset of items in a set period or within a set geographical location. Imagine: collect the SF 100 Set of Items in 24 hours to win a prize! We want to build out the social side of the application. Being able to interact with other users, trade more robustly, and create a more open marketplace will allow for tons of fun and has cool implications for rarity and more. We want to improve the educational features. While being able to talk with an object is a feature we are really happy about, augmenting GPT's ability to explain facts about an item with a vector database would supercharge our app's ability to teach. ## README (from the GitHub repository)

logo

## Inspiration As we go about our lives, all of us experience so much, witness so much, learn so much. When we were kids, exploration and excitement were a crucial part of every day, but as adults, responsibility can cause us to lose some of that magic. We wanted to remind each other that although we may not live in the Kanto region, the Earth still offers an enormous amount to see—and infinite amount to collect. So, we built WorldDex. WorldDex is a real-life PokeDex (and so much more) that lets you scan and save anything into an ever-growing collection, offering a portal for you to learn, share, enjoy, and remember. ## What it does WorldDex is primarily available through our mobile iOS application. Via this application, you can take a picture of any object and speak the name of what is being scanned into the interface. Be careful though! You will have a limited number of catch tries per day. Whether it's a simple lamp on your desk or a rare species of salamander, our app analyzes your image and spoken label, computing a catch probability which informs a roll of the dice. If you managed to catch your item, it is automatically added to your collection both in-app and on the blockchain. You also have the opportunity to record some information about where you were, who you were with, and any other details you're excited to include about your capture. WorldDex allows you to have a spoken or written conversation with your collection about the items you have caught. All you need is a microphone, a speaker, and an object; from there, our intelligent models can tell you all about your item. This puts the knowledge of the world's best LLMs at your fingertips, just one scan away from a detailed educational discussion about some of your favorite memories. Once you have completed a scan, WorldDex creates an NFT of your image. No need to panic—we are not crypto devs gearing for a rug-pull. Instead, the NFT means your ownership of that scan is provable and trade-able. Yes, someone can screenshot your NFT, but they will not be able to get it into their WorldDex! On the community page, you can check out other people's scans and take a look at what your family, friends, and acquaintances have been collecting—in real time! Your WorldDex account also comes with access to our web application. There, you can access a more dynamic view of your collection. Coming soon to the web version: detailed analytics, rarities, trading, and more! ## How we built it The workhorse of our mobile application is written in Swift, with iOS native speech-to-text translation and tons of custom styling and design. The app interacts with a variety of powerful back-ends: **OBJECT DETECTION, CAPTURE, AND PROBABILITY** are handled by an AWS EC2 instance with a Deep Learning AMI running PyTorch 2.0 with an NVIDIA T4 GPU. The instance runs inference on a state-of-the-art pre-trained zero-shot object detection model called [GroundingDINO](https://arxiv.org/abs/2303.05499) (Liu et al., 2023). _Associated technology: GroundingDINO, AWS EC2, PyTorch, iOS speech-to-text, GPT-3.5 Turbo Instruct, GPT 4_ - A flask server runs on the EC2 instance. When the user takes a scan and speaks into their phone, iOS speech-to-text transcribes the user's message and sends both the image and the user's provided information to the back-end. - We use GPT-3.5 Turbo Instruct (instruct-tuned version of GPT exceptionally good at executing novel, precise instructions) to extract the object label within the user's spoken text. With some careful prompt engineering and in-context learning, the label extraction works like a charm even when given complex and long audio transcriptions. - GroundingDINO is a pre-trained transformer-based object detection model that can predict on _any_ natural language label (rather than only on labels it was trained on), which enables WorldDex users to collect anything they can dream of! - GroundingDINO uses the label spoken by the user as a prompt to search for all instances of their desired object in the photo taken on their phone. It assigns these instances with a confidence score, and the instance with the max confidence score (logits) is bounded and scanned. In WorldDex, the success of a catch is a function of this confidence score. This bounding box is later used in the mobile app to crop a perfect thumbnail image for the object. **DATA STORAGE AND BLOCKCHAIN INTERACTIONS** are handled by an intricate JavaScript microservice architecture. This back-end uses Node.js and Express to route across a variety of on and off-chain services. _Associated Technology: Hedera, CockroachDB, Express, web3.storage, Ethereum, IPFS_ - Database information (user account info, friends, image metadata, images, etc) are stored in a server-less CockroachDB instance. Through our endpoints, the back-end writes SQL queries to properly interact with the DB. - When a catch is confirmed, the back-end uses the web3.storage framework to deploy the image onto IPFS for decentralized storage. Although a copy is maintained in our internal database for quick access, this grants the user the ability to share and interact with their image external to our centralized application. - After a catch has been hosted by IPFS, our [custom WLDD Token](https://hashscan.io/testnet/token/0.0.5787090?p=1&k=78) programmatically deployed on the open-source PoS Hedera network's testnet is used to mint an original NFT based on the image. This NFT contains in on-chain metadata a pointer to the IPFS metadataCID. The Hedera network offers straightforward and highly cost-effective functionality. - Once the image has been stored on IPFS and an associated NFT has been minted, the unique ID we generate per image is granted to the user's collection on our custom Ethereum smart contract deployed on the Sepolia Testnet. Why the need for interactions with Ethereum AND Hedera? Our Ethereum smart contract allows for our own two-party trading functionality. However, we plan to deploy this EVM byte code to the Hedera network to simplify in the future. **CHATTING WITH COLLECTION** is handled by another API hosted on the same GPU-accelerated EC2 instance used by the models. When the user records an audio message in the mobile app, we use iOS speech to text to transcribe the message and send it to this API. The API uses GPT-4 to determine the response, with additional context retrieved from the database's image labels and metadata. Then, it uses the state-of-the-art text-to-speech model from ElevenLabs (with streaming functionality for minimized latency) to read out the model's response and hold a conversation with the user. _Associated Technology: GPT-4, ElevenLabs TTS, AWS EC2_ **COMMUNITY & PROFILE** is built on the frontend using Swift and portrays your community's captures in real time. You can see people's captured items, their capture location, time of capture, probability of capture, and live audio transcription notes. Each user's captures are linked to their unique accounts, for which they can sign up for an account or log in on the profile page of the app. The frontend is linked to the CockroachDB database, making GET and POST requests to display community posts and manage creation of profiles. _Associated Technology: Swift, XCode, CockroachDB_ **THE WEB APPLICATION** is built with Reflex and backed by the same JavaScript API and databases as the other front-end. Here, we display a more dynamic interface meant to be more nostalgic of opening the PokeDex upon logging in to your account. However, we also wanted to include a web app to give us flexibility for future extensions, with the ability to display detailed visualizations, analytics, trade, and more. _Associated Technology: Reflex_ **We know that's a lot of text and a lot of technologies. So:** ##The TLDR is: - Your [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 109 recognized source files, 853 KB. - C (language) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Solidity (language) — detected in the code - Swift (language) — detected in the code - AWS (technology) — claimed on Devpost, not found in the code - Express (technology) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 182) ``` .DS_Store .gitignore Model API/.gitignore Model API/conversation_server.py Model API/Dockerfile Model API/gpu_test.py Model API/requirements.txt README.md rxconfig.py WorldDex App/.DS_Store WorldDex App/Podfile WorldDex App/Podfile.lock WorldDex App/Pods/.DS_Store WorldDex App/Pods/Kingfisher/LICENSE WorldDex App/Pods/Kingfisher/README.md WorldDex App/Pods/Kingfisher/Sources/Cache/CacheSerializer.swift WorldDex App/Pods/Kingfisher/Sources/Cache/DiskStorage.swift WorldDex App/Pods/Kingfisher/Sources/Cache/FormatIndicatedCacheSerializer.swift WorldDex App/Pods/Kingfisher/Sources/Cache/ImageCache.swift WorldDex App/Pods/Kingfisher/Sources/Cache/MemoryStorage.swift WorldDex App/Pods/Kingfisher/Sources/Cache/Storage.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/CPListItem+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/ImageView+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/NSButton+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/NSTextAttachment+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/TVMonogramView+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/UIButton+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/Extensions/WKInterfaceImage+Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/General/ImageSource/AVAssetImageDataProvider.swift WorldDex App/Pods/Kingfisher/Sources/General/ImageSource/ImageDataProvider.swift WorldDex App/Pods/Kingfisher/Sources/General/ImageSource/Resource.swift WorldDex App/Pods/Kingfisher/Sources/General/ImageSource/Source.swift WorldDex App/Pods/Kingfisher/Sources/General/KF.swift WorldDex App/Pods/Kingfisher/Sources/General/KFOptionsSetter.swift WorldDex App/Pods/Kingfisher/Sources/General/Kingfisher.swift WorldDex App/Pods/Kingfisher/Sources/General/KingfisherError.swift WorldDex App/Pods/Kingfisher/Sources/General/KingfisherManager.swift WorldDex App/Pods/Kingfisher/Sources/General/KingfisherOptionsInfo.swift WorldDex App/Pods/Kingfisher/Sources/Image/Filter.swift WorldDex App/Pods/Kingfisher/Sources/Image/GIFAnimatedImage.swift WorldDex App/Pods/Kingfisher/Sources/Image/GraphicsContext.swift WorldDex App/Pods/Kingfisher/Sources/Image/Image.swift WorldDex App/Pods/Kingfisher/Sources/Image/ImageDrawing.swift WorldDex App/Pods/Kingfisher/Sources/Image/ImageFormat.swift WorldDex App/Pods/Kingfisher/Sources/Image/ImageProcessor.swift WorldDex App/Pods/Kingfisher/Sources/Image/ImageProgressive.swift WorldDex App/Pods/Kingfisher/Sources/Image/ImageTransition.swift WorldDex App/Pods/Kingfisher/Sources/Image/Placeholder.swift WorldDex App/Pods/Kingfisher/Sources/Networking/AuthenticationChallengeResponsable.swift WorldDex App/Pods/Kingfisher/Sources/Networking/ImageDataProcessor.swift WorldDex App/Pods/Kingfisher/Sources/Networking/ImageDownloader.swift WorldDex App/Pods/Kingfisher/Sources/Networking/ImageDownloaderDelegate.swift WorldDex App/Pods/Kingfisher/Sources/Networking/ImageModifier.swift WorldDex App/Pods/Kingfisher/Sources/Networking/ImagePrefetcher.swift WorldDex App/Pods/Kingfisher/Sources/Networking/RedirectHandler.swift WorldDex App/Pods/Kingfisher/Sources/Networking/RequestModifier.swift WorldDex App/Pods/Kingfisher/Sources/Networking/RetryStrategy.swift WorldDex App/Pods/Kingfisher/Sources/Networking/SessionDataTask.swift WorldDex App/Pods/Kingfisher/Sources/Networking/SessionDelegate.swift WorldDex App/Pods/Kingfisher/Sources/PrivacyInfo.xcprivacy WorldDex App/Pods/Kingfisher/Sources/SwiftUI/ImageBinder.swift WorldDex App/Pods/Kingfisher/Sources/SwiftUI/ImageContext.swift WorldDex App/Pods/Kingfisher/Sources/SwiftUI/KFAnimatedImage.swift WorldDex App/Pods/Kingfisher/Sources/SwiftUI/KFImage.swift WorldDex App/Pods/Kingfisher/Sources/SwiftUI/KFImageOptions.swift WorldDex App/Pods/Kingfisher/Sources/SwiftUI/KFImageProtocol.swift WorldDex App/Pods/Kingfisher/Sources/SwiftUI/KFImageRenderer.swift WorldDex App/Pods/Kingfisher/Sources/Utility/Box.swift WorldDex App/Pods/Kingfisher/Sources/Utility/CallbackQueue.swift WorldDex App/Pods/Kingfisher/Sources/Utility/Delegate.swift WorldDex App/Pods/Kingfisher/Sources/Utility/ExtensionHelpers.swift WorldDex App/Pods/Kingfisher/Sources/Utility/Result.swift WorldDex App/Pods/Kingfisher/Sources/Utility/Runtime.swift WorldDex App/Pods/Kingfisher/Sources/Utility/SizeExtensions.swift WorldDex App/Pods/Kingfisher/Sources/Utility/String+MD5.swift WorldDex App/Pods/Kingfisher/Sources/Views/AnimatedImageView.swift WorldDex App/Pods/Kingfisher/Sources/Views/Indicator.swift WorldDex App/Pods/Manifest.lock WorldDex App/Pods/Pods.xcodeproj/project.pbxproj WorldDex App/Pods/Pods.xcodeproj/xcuserdata/anthonyqin.xcuserdatad/xcschemes/Pods-WorldDex.xcscheme WorldDex App/Pods/Pods.xcodeproj/xcuserdata/anthonyqin.xcuserdatad/xcschemes/xcschememanagement.plist WorldDex App/Pods/Pods.xcodeproj/xcuserdata/jaidenreddy.xcuserdatad/xcschemes/Kingfisher-Kingfisher.xcscheme WorldDex App/Pods/Pods.xcodeproj/xcuserdata/jaidenreddy.xcuserdatad/xcschemes/Kingfisher.xcscheme WorldDex App/Pods/Pods.xcodeproj/xcuserdata/jaidenreddy.xcuserdatad/xcschemes/Pods-WorldDex.xcscheme WorldDex App/Pods/Pods.xcodeproj/xcuserdata/jaidenreddy.xcuserdatad/xcschemes/xcschememanagement.plist WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher-dummy.m WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher-Info.plist WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher-prefix.pch WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher-umbrella.h WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher.debug.xcconfig WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher.modulemap WorldDex App/Pods/Target Support Files/Kingfisher/Kingfisher.release.xcconfig WorldDex App/Pods/Target Support Files/Kingfisher/ResourceBundle-Kingfisher-Kingfisher-Info.plist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-acknowledgements.markdown WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-acknowledgements.plist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-dummy.m WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-frameworks-Debug-input-files.xcfilelist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-frameworks-Debug-output-files.xcfilelist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-frameworks-Release-input-files.xcfilelist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-frameworks-Release-output-files.xcfilelist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-frameworks.sh WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-Info.plist WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex-umbrella.h WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex.debug.xcconfig WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex.modulemap WorldDex App/Pods/Target Support Files/Pods-WorldDex/Pods-WorldDex.release.xcconfig WorldDex App/WorldDex.xcodeproj/project.pbxproj WorldDex App/WorldDex.xcodeproj/project.xcworkspace/contents.xcworkspacedata WorldDex App/WorldDex.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist WorldDex App/WorldDex.xcodeproj/project.xcworkspace/xcuserdata/anthonyqin.xcuserdatad/UserInterfaceState.xcuserstate WorldDex App/WorldDex.xcodeproj/xcuserdata/anthonyqin.xcuserdatad/xcschemes/xcschememanagement.plist WorldDex App/WorldDex.xcodeproj/xcuserdata/jaidenreddy.xcuserdatad/xcschemes/xcschememanagement.plist WorldDex App/WorldDex.xcworkspace/contents.xcworkspacedata WorldDex App/WorldDex.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist WorldDex App/WorldDex.xcworkspace/xcshareddata/swiftpm/Package.resolved WorldDex App/WorldDex.xcworkspace/xcuserdata/anthonyqin.xcuserdatad/UserInterfaceState.xcuserstate WorldDex App/WorldDex.xcworkspace/xcuserdata/anthonyqin.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist WorldDex App/WorldDex.xcworkspace/xcuserdata/jaidenreddy.xcuserdatad/UserInterfaceState.xcuserstate WorldDex App/WorldDex/.DS_Store WorldDex App/WorldDex/Assets.xcassets/.DS_Store [62 more files omitted for size] ``` ### Dependencies - Model API/requirements.txt: elevenlabs, Flask, flask-cors, Flask-Session, numpy, openai, opencv-python, python-dotenv, torch, torchvision - WorldHost/requirements.txt: azure-data-tables, azure-storage-blob, coincurve, cryptography, eciespy, eth-account, eth-keys, eth-rlp, fastapi, httpx, json, python-dotenv, requests, web3 ### Recent commits (newest first) - Make detail view prettier - Back-end support for social page pagination - pagination for social page - add prefetching - remove unnecessary back-end code from previous iteration - CACHING WORKS - send URLs instead of images to backend - Update server.py - Merge remote-tracking branch 'origin/main' into main - Azure compatible + GPT4V functionality for AI servers - Fix time formatting - Fix bottom bar thing - Add loading animation to conversation AI - Fix timestamp formatting - Fix plist file - Remove WorldDex.sol file - working nft backend - Update info.plist file - support more of decentralize storage, still not ready - change urls ## Key source files (fetched from GitHub, selected and truncated for size) ### Model API/requirements.txt ``` Flask flask-cors numpy torch torchvision opencv-python openai python-dotenv Flask-Session elevenlabs ``` ### WorldHost/requirements.txt ``` fastapi httpx azure-storage-blob azure-data-tables eciespy cryptography web3 requests coincurve eth-account eth-keys eth-rlp python-dotenv json ``` ### Model API/Dockerfile ``` FROM python:3.8-slim-buster # Set working directory WORKDIR /workspace # Install git and clone GroundingDINO repository RUN apt-get update && apt-get install -y git && \ git clone https://github.com/IDEA-Research/GroundingDINO.git /workspace/GroundingDINO && \ apt-get clean && rm -rf /var/lib/apt/lists/* # Install torch and any other dependencies RUN pip install --no-cache-dir torch # Install GroundingDINO RUN pip install --no-cache-dir -e /workspace/GroundingDINO # Set default command CMD ["python"] ``` ### WorldHost/server.py ```python from fastapi import FastAPI, File, UploadFile, HTTPException, Form, status from web3 import Web3 from web3.middleware import geth_poa_middleware from typing import Optional from cryptography.fernet import Fernet import base64 import json import os from httpx import AsyncClient, TimeoutException from parse_public import eth_address_to_pub_key from ecies import encrypt, decrypt from io import BytesIO from azure.storage.blob import BlobServiceClient, ContainerClient, generate_blob_sas, BlobSasPermissions from azure.data.tables import TableServiceClient, TableEntity import threading from typing import List from urllib.parse import urlparse import hashlib from dotenv import load_dotenv import datetime import time import asyncio from starlette.concurrency import run_in_threadpool import pytz lock = threading.Lock() load_dotenv() app = FastAPI() # Set your NFT.storage API key in environment variable nft_storage_api_key = os.environ.get('NFT_STORAGE_API_KEY') # Set Etherscan API Key etherscan_api_key = os.environ.get('ETHERSCAN_API_KEY') # Set Web3 Provider web3provider = os.environ.get('WEB3_PROVIDER') # Azure storage account details account_name = "worlddexstorage2" blob_storage_name = "dex-storage" account_key = os.environ.get("BLOB_ACCOUNT_KEY") connection_string = f"DefaultEndpointsProtocol=https;AccountName={account_name};AccountKey={account_key};EndpointSuffix=core.windows.net" # Initialize Azure Blob Service Client blob_service_client = BlobServiceClient.from_connection_string(connection_string) # Initialize Azure Table Service Client table_service_client = TableServiceClient.from_connection_string(connection_string) table_client = table_service_client.get_table_client(table_name="dextablestorage") users_table_client = table_service_client.get_table_client(table_name="users") # Ethereum node connection WEB3_PROVIDER_URI = os.getenv("WEB3_PROVIDER") web3 = Web3(Web3.HTTPProvider(WEB3_PROVIDER_URI)) # Inject middleware for compatibility with networks like Binance Smart Chain web3.middleware_onion.inject(geth_poa_middleware, layer=0) # Contract details CONTRACT_ADDRESS = os.getenv("CONTRACT_ADDRESS") # Load the ABI from the file with open('contract_abi.txt', 'r') as abi_file: CONTRACT_ABI = json.load(abi_file) contract = web3.eth.contract(address=CONTRACT_ADDRESS, abi=CONTRACT_ABI) # Wallet details for the transaction sender WALLET_PRIVATE_KEY = os.getenv("WALLET_PRIVATE_KEY") WALLET_ADDRESS = os.getenv("WALLET_ADDRESS") @app.get("/userData") async def user_data(username: str): # try: # Query for the specific user filter_query = f"PartitionKey eq '{username}' and RowKey eq 'userinfo'" user_entities = list(users_table_client.query_entities(filter_query)) if not user_entities: raise HTTPException(status_code=404, detail="User not found") user_entity = user_entities[0] # Assuming 'Email' and 'EthereumAddress' are the field names in your table email = user_entity.get('Email', 'No email provided') ethereum_address = user_entity.get('EthereumAddress', 'No Ethereum address provided') return {"email": email, "ethereum_address": ethereum_address} # except Exception as e: # raise HTTPException(status_code=500, detail=str(e)) @app.post("/registerUser") async def register_user(username: str, password: str, email: str, ethereum_address: str = None): try: # Check if username already exists username_query = f"PartitionKey eq '{username}'" existing_users_by_username = list(users_table_client.query_entities(username_query)) if existing_users_by_username: raise HTTPException(status_code=400, detail="Username already exists") # Check if email already exists email_query = f"Email eq '{email}'" existing_users_by_email = list(users_table_client.query_entities(email_query)) if existing_users_by_email: raise HTTPException(status_code=400, detail="Email already exists") # Hash the password hashed_password = hash_password(password) # Create user entity user_entity = TableEntity() user_entity['PartitionKey'] = username user_entity['RowKey'] = 'userinfo' # Or some other suitable row key user_entity['Password'] = hashed_password user_entity['Email'] = email user_entity['EthereumAddress'] = ethereum_address # Add to table users_table_client.create_entity(user_entity) return {"message": "User registered successfully"} except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.post("/login") async def login(username: str = Form(...), password: str = Form(...)): filter_query = f"PartitionKey eq '{username}' and RowKey eq 'userinfo'" user_entities = list(users_table_client.query_entities(filter_query)) if not user_entities: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid username or password") user_entity = user_entities[0] hashed_password = hash_password(password) if user_entity['Password'] == hashed_password: return {"status": "success", "message": "Login successful"} else: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid username or password") @app.get("/getAllUsernames") async def get_all_usernames(): try: # Query to retrieve all user entities filter_query = "RowKey eq 'userinfo'" # Assuming 'userinfo' is the RowKey for all user entries user_entities = users_table_client.query_entities(filter_query) # Extracting usernames (PartitionKey) usernames = [entity['PartitionKey'] for entity in user_entities] return {"usernames": usernames} except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.get("/getUserImageUrls") async def get_user_image_urls(u [truncated — 13715 more characters] ``` ### worlddex/pages/index.py ```python """The home page of the app.""" from WorldDex import styles from WorldDex.templates import template import reflex as rx @template(route="/", title="Home", image="/github.svg") def index() -> rx.Component: """The home page. Returns: The UI for the home page. """ with open("README.md", encoding="utf-8") as readme: content = readme.read() return rx.markdown(content, component_map=styles.markdown_style) ``` ### rxconfig.py ```python import reflex as rx config = rx.Config( app_name="WorldDex", ) ``` ### worlddex/__init__.py ```python """Base template for Reflex.""" ``` ### Model API/gpu_test.py ```python import torch # Check if CUDA (GPU support) is available print("CUDA available:", torch.cuda.is_available()) # Set device to GPU (cuda) if available, else CPU device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print("Using device:", device) # Create a random tensor and move it to the chosen device x = torch.rand(5, 5).to(device) print("Random Tensor on GPU:", x) # Perform a simple computation y = x * x print("Squared Tensor:", y) ``` ### WorldHost/WorldNFT.sol ```solidity // SPDX-License-Identifier: MIT pragma solidity ^0.8.0; import "@openzeppelin/contracts/token/ERC721/ERC721.sol"; import "@openzeppelin/contracts/token/ERC721/extensions/ERC721URIStorage.sol"; import "@openzeppelin/contracts/access/Ownable.sol"; contract WorldNFT is ERC721URIStorage, Ownable { uint256 private _tokenIds; constructor() ERC721("ImageNFT", "IMG") {} function mintNFT(address recipient, string memory tokenURI) public onlyOwner returns (uint256) { _tokenIds++; uint256 newItemId = _tokenIds; _mint(recipient, newItemId); _setTokenURI(newItemId, tokenURI); return newItemId; } } ``` ### worlddex/styles.py ```python """Styles for the app.""" import reflex as rx border_radius = "0.375rem" box_shadow = "0px 0px 0px 1px rgba(84, 82, 95, 0.14)" border = "1px solid #F4F3F6" text_color = "black" accent_text_color = "#1A1060" accent_color = "#F5EFFE" hover_accent_color = {"_hover": {"color": accent_color}} hover_accent_bg = {"_hover": {"bg": accent_color}} content_width_vw = "90vw" sidebar_width = "20em" template_page_style = {"padding_top": "5em", "padding_x": ["auto", "2em"]} template_content_style = { "width": "100%", "align_items": "flex-start", "box_shadow": box_shadow, "border_radius": border_radius, "padding": "1em", "margin_bottom": "2em", } link_style = { "color": text_color, "text_decoration": "none", **hover_accent_color, } overlapping_button_style = { "background_color": "white", "border": border, "border_radius": border_radius, } base_style = { rx.MenuButton: { "width": "3em", "height": "3em", **overlapping_button_style, }, rx.MenuItem: hover_accent_bg, } markdown_style = { "code": lambda text: rx.code(text, color="#1F1944", bg="#EAE4FD"), "a": lambda text, **props: rx.link( text, **props, font_weight="bold", color="#03030B", text_decoration="underline", text_decoration_color="#AD9BF8", _hover={ "color": "#AD9BF8", "text_decoration": "underline", "text_decoration_color": "#03030B", }, ), } ``` [99 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]