# Project export: APIVerse This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: A platfotm revolutionizing API integration with LLMs, offering voice-guided recommendations for developers and end-users, ensuring a seamless and intuitive experience. - Devpost: https://devpost.com/software/apiverse - GitHub: https://github.com/liyufeng0802/cal_hack - Team: 3 GitHub contributor(s) — Jingchao Zhong (33 commits), liyufeng0802 (11 commits), a1exzhang0903 (1 commits) ## Devpost submission (written by the team) ### Inspiration The inspiration behind APIVerse stems from the common frustrations and time-consuming processes that developers and users alike face when integrating and utilizing APIs. Traditionally, users have to sift through extensive documentation to understand how to work with an API, figure out the required parameters, manage authentication tokens, and more. We wanted to create a platform that not only simplifies these steps but also leverages the power of Large Language Models (LLMs) to provide intuitive and conversational interactions with APIs. ### What it does APIVerse serves as a centralized platform that radically simplifies the API integration and utilization process. Users can effortlessly import APIs by uploading their documentation, while companies have the ability to publish and manage their APIs on the platform. With the assistance of LLMs, users can interact with APIs through natural language, asking for recommendations, and even sending requests verbally. Developers have the option to export configurations, making the platform versatile and adaptable to different needs. Essentially, APIVerse streamlines the entire API experience, making it more accessible, efficient, and user-friendly. ### How we built it APIVerse is built on a robust framework that integrates LLMs to interpret and process natural language inputs. The platform has a user-friendly interface for importing API documentation and managing API listings. On the backend, we developed algorithms to parse and interpret API documentation, extracting necessary information like endpoints, parameters, and authentication details. The LLMs are trained to understand API-related queries and provide accurate and helpful responses, recommendations, and assistance to the users. ### Challenges we ran into One of the major challenges was ensuring that the LLMs could accurately interpret and provide relevant information and recommendations based on varied and sometimes incomplete API documentation. Developing a system that could handle different documentation formats and standards required extensive testing and refinement. Additionally, ensuring a seamless and intuitive user experience while managing the complexities of API interactions posed significant design and development challenges. ### Accomplishments we're proud of We are incredibly proud of creating a platform that democratizes API interactions, making them accessible to a wider audience beyond just developers. APIVerse’s ability to provide conversational interactions with APIs, backed by the power of LLMs, represents a significant advancement in simplifying complex technical processes. The platform’s capacity to understand and process a variety of API documentation formats is another achievement that we are particularly proud of. ### What we learned Through the development of APIVerse, we gained valuable insights into natural language processing, machine learning, and the intricacies of API interactions. We learned about the challenges associated with interpreting diverse API documentation and the importance of a user-centric design to ensure an intuitive and efficient user experience. ### What's next The future of APIVerse is focused on continuous improvement and expansion. We plan to further enhance the LLMs’ capabilities, ensuring even more accurate and helpful interactions. We aim to expand the platform’s functionality to support more API types and documentation formats. Additionally, we are looking at integrating additional tools and resources to aid in API development and testing, creating a comprehensive ecosystem for API management and integration. ## README (from the GitHub repository) # APIVerse (Cal Hacks 10.0)
🪐 APIVerse's goal is to provide a more convenient approach to utilizing APIs. We are in the process of developing an API platform that enables users to import API documentation from various sources and offers a natural conversational interface for interacting with the APIs. **💡: Make it simply to interact with technoloy!** # Project Design ✨ Here is the high level design of the project architecture. (**React** + **Flask**) # Contribution **Author:** - Jingchao (Perry) Zhong - Jiaxiang (Calvin) Li - Junyi (Alex) Zhang - Yufeng (Jimmy) Li ## Detected evidence (automated analysis) Indexed codebase: 71 recognized source files, 138 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (88 of 88) ``` .gitignore action_module/arg_retriever.py action_module/caller.py action_module/questioner.py action_module/rag_gpt.py action_module/router.py app.py database/cache.json database/current_curl.json database/live.json database/phote.json frontend/.eslintrc.json frontend/.gitattributes frontend/.gitignore frontend/.npmrc frontend/CHANGELOG.md frontend/next.config.js frontend/package.json frontend/pages/_app.tsx frontend/pages/_document.tsx frontend/pages/index.tsx frontend/prettier.config.js frontend/public/fonts/dm-sans/OFL.txt frontend/public/manifest.json frontend/public/robots.txt frontend/README.md frontend/src/components/APIContext/APIContext.tsx frontend/src/components/apiModal/index.tsx frontend/src/components/card/Card.tsx frontend/src/components/CodeBlock.tsx frontend/src/components/fields/InputField.tsx frontend/src/components/footer/FooterAdmin.tsx frontend/src/components/icons/IconBox.tsx frontend/src/components/icons/Icons.tsx frontend/src/components/image/Avatar.tsx frontend/src/components/image/Image.tsx frontend/src/components/link/Link.tsx frontend/src/components/link/LinkButton.tsx frontend/src/components/link/NavLink.tsx frontend/src/components/MarkdownBlock.tsx frontend/src/components/MessageBox.tsx frontend/src/components/navbar/NavbarAdmin.tsx frontend/src/components/navbar/NavbarLinksAdmin.tsx frontend/src/components/navbar/searchBar/SearchBar.tsx frontend/src/components/scrollbar/Scrollbar.tsx frontend/src/components/search/index.tsx frontend/src/components/separator/Separator.tsx frontend/src/components/sidebar/components/Brand.tsx frontend/src/components/sidebar/components/Content.tsx frontend/src/components/sidebar/components/Links.tsx frontend/src/components/sidebar/components/SidebarCard.tsx frontend/src/components/sidebar/components/test.md frontend/src/components/sidebar/Sidebar.tsx frontend/src/contexts/SidebarContext.ts frontend/src/lib/chakra.tsx frontend/src/routes.tsx frontend/src/styles/App.css frontend/src/styles/Contact.css frontend/src/styles/MiniCalendar.css frontend/src/styles/Plugins.css frontend/src/theme/additions/card/card.ts frontend/src/theme/components/badge.ts frontend/src/theme/components/button.ts frontend/src/theme/components/input.ts frontend/src/theme/components/link.ts frontend/src/theme/components/progress.ts frontend/src/theme/components/switch.ts frontend/src/theme/components/textarea.ts frontend/src/theme/styles.ts frontend/src/theme/theme.tsx frontend/src/types/hui-types.d.ts frontend/src/types/images.d.ts frontend/src/types/navigation.d.ts frontend/src/types/react-table-config.d.ts frontend/src/types/stylis.d.ts frontend/src/types/types.ts frontend/src/utils/navigation.tsx frontend/tsconfig.json gpt_module.py import_module/import_module.py LICENSE Makefile README.md requirements.txt setup.py test_folder/test_curl.py test_folder/test_gpt.py test_folder/test_questioner.py ``` ### Dependencies - frontend/package.json: @chakra-ui/icons@^1.1.5, @chakra-ui/react@1.8.9, @chakra-ui/system@^1.12.1, @chakra-ui/theme-tools@^1.3.6, @codemirror/legacy-modes@^6.3.2, @emotion/react@^11.4.1, @emotion/styled@^11.3.0, @heroicons/react@^2.0.17, @material-tailwind/react@^1.4.2, @tanstack/react-table@^8.5.15, @testing-library/jest-dom@^5.14.1, @testing-library/react@^11.2.7, @testing-library/user-event@^12.8.3, @types/node@18.15.11, @types/react@18.0.31, @types/react-dom@18.0.11, @uiw/codemirror-theme-tokyo-night@^4.19.11, @uiw/react-codemirror@^4.19.11, autoprefixer@^10.4.14, axios@^1.6.0, babel-cli@^6.26.0, babel-preset-es2015@^6.24.1, babel-preset-react@^6.24.1, babel-register@^6.26.0, endent@^2.1.0, eslint@8.37.0, eslint-config-next@13.2.4, eventsource@^2.0.2, eventsource-parser@^1.0.0, framer-motion@^4.1.17, next@13.2.4, postcss@^8.4.21, prettier-plugin-tailwindcss@^0.2.6, react@18.2.0, react-build-sitemap@^0.2.2, react-custom-scrollbars-2@^4.2.1, react-dom@18.2.0, react-icons@^4.9.0, react-is@^18.0.0, react-markdown@^8.0.6, react-router-dom@^5.3.0, react-scripts@5.0.0, remark-gfm@^3.0.1, tailwindcss@^3.3.1, typescript@4.9.5, web-vitals@^1.1.2 - requirements.txt: flask, flask-cors, langchain, numpy, openai, pyyaml ### Recent commits (newest first) - Refactor modules, and update frontend layout - Update README and remove deprecated files - Add summary caching, and remove unnecessary files - Merge pull request #3 from liyufeng0802/frontend2 - Fix chat bug - Change frontend page rescoure - Update License, and git ignore - Update Frontend readme - Merge pull request #2 from liyufeng0802/frontend2 - Submission - All the updates so far - Frontend design fix - Backup - Backup - Backup - Backup frontend - Backup - Resolve merge conflict - Backiup - Add cache fix query logic ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/CHANGELOG.md ```markdown # Changelog ## [1.0.0] 2023-06-20 ### Official Release Added TypeScript & NextJS ``` ### frontend/src/components/sidebar/components/test.md ```markdown mkdir CenteredAuthLayout && mv CenteredAuthLayout.tsx CenteredAuthLayout/page.tsx && mkdir DefaultAuthLayout && mv DefaultAuthLayout.tsx DefaultAuthLayout/page.tsx && mkdir PricingAuthLayout && mv PricingAuthLayout.tsx PricingAuthLayout/page.tsx mv forgot-password ../../app/auth && mv lock ../../app/auth && mv main ../../app/auth && mv sign-in ../../app/auth && mv sign-up ../../app/auth && mv verification ../../app/auth mkdir centered && mv centered.tsx centered/page.tsx && mkdir default && mv default.tsx default/page.tsx ``` ### requirements.txt ``` flask openai pyyaml numpy langchain flask-cors ``` ### frontend/package.json ``` { "name": "API-Verse", "version": "1.0.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@chakra-ui/icons": "^1.1.5", "@chakra-ui/react": "1.8.9", "@chakra-ui/system": "^1.12.1", "@chakra-ui/theme-tools": "^1.3.6", "@codemirror/legacy-modes": "^6.3.2", "@emotion/react": "^11.4.1", "@emotion/styled": "^11.3.0", "@heroicons/react": "^2.0.17", "@material-tailwind/react": "^1.4.2", "@tanstack/react-table": "^8.5.15", "@testing-library/jest-dom": "^5.14.1", "@testing-library/react": "^11.2.7", "@testing-library/user-event": "^12.8.3", "@uiw/codemirror-theme-tokyo-night": "^4.19.11", "@uiw/react-codemirror": "^4.19.11", "axios": "^1.6.0", "babel-cli": "^6.26.0", "babel-preset-es2015": "^6.24.1", "babel-preset-react": "^6.24.1", "babel-register": "^6.26.0", "endent": "^2.1.0", "eventsource-parser": "^1.0.0", "framer-motion": "^4.1.17", "next": "13.2.4", "react": "18.2.0", "react-build-sitemap": "^0.2.2", "react-custom-scrollbars-2": "^4.2.1", "react-dom": "18.2.0", "react-icons": "^4.9.0", "react-is": "^18.0.0", "react-markdown": "^8.0.6", "react-router-dom": "^5.3.0", "react-scripts": "5.0.0", "remark-gfm": "^3.0.1", "typescript": "4.9.5", "web-vitals": "^1.1.2" }, "devDependencies": { "@types/node": "18.15.11", "@types/react": "18.0.31", "@types/react-dom": "18.0.11", "autoprefixer": "^10.4.14", "eslint": "8.37.0", "eslint-config-next": "13.2.4", "eventsource": "^2.0.2", "postcss": "^8.4.21", "prettier-plugin-tailwindcss": "^0.2.6", "tailwindcss": "^3.3.1" }, "resolutions": { "@types/react": "18.0.31", "@types/react-dom": "18.0.11" } } ``` ### app.py ```python from flask import Flask, request, jsonify from import_module.import_module import Import_module from action_module.caller import Caller from flask_cors import CORS app = Flask(__name__) CORS(app) # import url from the list @app.route('/import', methods=['GET']) def importNewDocument(): print("importing new document") args = request.args url = args.get("url") print(f"importing url: {url}") importer = Import_module() importer.addNewURL(url) jsonResponse, summary = importer.get_Response() data = { "listData" : jsonResponse, "summary" : summary, } # data = { # "listData" : [ # { # "title" : "title1", # "url" : "url1", # "summary" : "summary1", # }, # { # "title" : "title2", # "url" : "url2", # "summary" : "summary2", # }, # { # "title" : "title3", # "url" : "url3", # "summary" : "summary3", # }, # ], # "summary" : "summary 1231231231231", # } return jsonify(data) # chat windom input @app.route('/query', methods=['POST']) def readQuery(): # input conversation [] print("start processing query") j = request.json # print(j) clist = j["conversation"] print(f"conversation list: {clist}") user_input = clist[-1] print(f"user input: {user_input}") caller = Caller() summary = caller.process_user_request(user_input, clist) print(f"summary: {summary}") clist.append(summary) # Conversation List # API response summary data = { "conversation" : clist, "summary" : summary, } return jsonify(data) if __name__ == '__main__': importer = Import_module() importer.erase_current_curl() app.run(host='0.0.0.0', port=105) ``` ### frontend/pages/index.tsx ```typescript 'use client'; /*eslint-disable*/ import MessageBoxChat from '@/components/MessageBox'; import {Button, Flex, Icon, Img, Input, Text, useColorModeValue,} from '@chakra-ui/react'; import {useEffect, useRef, useState} from 'react'; import {MdAutoAwesome, MdPerson} from 'react-icons/md'; import Bg from '../public/img/chat/bg-image.png'; import axios from 'axios'; import APIModal from "@/components/apiModal"; type ChatMessage = { type: 'sent' | 'received'; message: string; }; type ChatHistory = ChatMessage[]; export default function Chat() { // Input States const [inputOnSubmit, setInputOnSubmit] = useState(''); const [inputCode, setInputCode] = useState(''); // Response message const [outputCode, setOutputCode] = useState(''); // Loading state const [loading, setLoading] = useState(false); const [apiDocURL, setURL] = useState(''); const [chatHistory, setChatHistory] = useState([]); const messagesEndRef = useRef(null); const scrollToBottom = () => { // @ts-ignore messagesEndRef.current?.scrollIntoView({behavior: "smooth"}); }; useEffect(() => { scrollToBottom(); }, [chatHistory]); useEffect(() => { if (apiDocURL === '') { return; } const msg = `Import documentation from ${apiDocURL}`; setInputCode(msg); }, [apiDocURL]); async function fetchAPIImport() { try { setLoading(true); const response = await axios.get('http://127.0.0.1:105/import', { params: {"url": 'https://icanhazdadjoke.com/api#endpoints'}, headers: { 'Content-Type': 'application/json', }, }); console.log("test0") // Accessing the response body let data = response.data; // Now you can use responseBody as needed console.log("123test") console.log(data); // @ts-ignore let summary = data['summary'] // summary = "123981239812938129389123" console.log(summary) const newChat: ChatMessage = { type: 'sent', message: `Import documentation from ${apiDocURL}` }; setChatHistory((prevChats) => [...prevChats, newChat].filter(chat => chat.message.trim() !== "")); setOutputCode(summary); const newReply: ChatMessage = {type: 'received', message: summary}; console.log(newReply) setChatHistory((prevChats) => [...prevChats, newReply].filter(chat => chat.message.trim() !== "")); setLoading(false); } catch (error) { console.error('Error fetching data: ', error); alert('Something went wrong when fetching from the API. Make sure to use a valid API key.'); } } async function fetchAPIQuery(conversation: string[]) { try { if (conversation.length === 0) { return; } console.log(`Final sent convesation ${conversation}`); const response = await axios.post('http://127.0.0.1:105/query', { conversation: conversation }, { headers: { 'Content-Type': 'application/json', } }); setLoading(true); console.log("Query Sent"); // Accessing the response body let data = response.data; console.log("Response Received"); console.log(data); let summary = data['summary']; console.log(`Here is the summary ${summary}`); // set new chat to history const newReply: ChatMessage = {type: 'received', message: summary}; const newHist = [...chatHistory, newReply].filter(chat => chat.message.trim() !== ""); console.log(`Test chat history: ${newHist}`); setChatHistory(newHist); setLoading(false); // Here, add your logic to handle the response as needed // For example, you can update the chat history, set output code, etc. } catch (error) { console.error('Error fetching data: ', error); alert('Something went wrong when fetching from the API. Please check the console for more details.'); } } function chatHistoryToConversation(chatHistory: ChatHistory): string[] { let conversation: string[] = []; for (let i = 0; i < chatHistory.length; i++) { conversation.push(chatHistory[i].message); } console.log(`Conversation transformed: ${conversation}`); return conversation; } const handleTranslate = async () => { setInputOnSubmit(inputCode); if (!inputCode) { alert('Please enter your message.'); return; } const newChat: ChatMessage = {type: 'sent', message: inputCode}; console.log(`Test new inserted question: ${newChat.message}`); let newHist = [...chatHistory, newChat].filter(chat => chat.message.trim() !== ""); console.log(`Test chat history: ${newHist}`); setOutputCode(' '); setLoading(true); // fetchData(); await setChatHistory(newHist); await fetchAPIQuery(chatHistoryToConversation(newHist)); setLoading(false); }; const borderColor = useColorModeValue('gray.200', 'whiteAlpha.200'); const inputColor = useColorModeValue('navy.700', 'white'); const brandColor = useColorModeValue('brand.500', 'white'); const textColor = useColorModeValue('navy.700', 'white'); const placeholderColor = useColorModeValue( {color: 'gray.500'}, {color: 'whiteAlpha.600'}, ); const handleChange = (Event: any) => { setInputCode(Event [truncated — 5554 more characters] ``` ### frontend/src/components/search/index.tsx ```typescript 'use client'; import { IconButton, Input, InputGroup, InputLeftElement, useColorModeValue, } from '@chakra-ui/react'; import { SearchIcon } from '@chakra-ui/icons'; export function SearchBar(props: { variant?: string; children?: JSX.Element; borderRadius?: string; value?: string | number; [x: string]: any; }) { // Pass the computed styles into the `__css` prop const { variant, children, borderRadius, ...rest } = props; // Chakra Color Mode const searchIconColor = useColorModeValue('gray.700', 'white'); const searchColor = useColorModeValue('gray.700', 'white'); const inputBg = useColorModeValue('transparent', 'navy.800'); return ( } /> ); } ``` ### frontend/src/components/apiModal/index.tsx ```typescript 'use client'; import Card from '@/components/card/Card'; import { Accordion, AccordionItem, AccordionButton, AccordionPanel, AccordionIcon, Box, Button, Flex, Icon, Input, Link, ListItem, UnorderedList, Modal, ModalBody, ModalCloseButton, ModalContent, ModalHeader, ModalOverlay, Text, useColorModeValue, useDisclosure, useToast, } from '@chakra-ui/react'; import { useState } from 'react'; import { MdLock } from 'react-icons/md'; function APIModal(props: { setApiKey: any; func_2_call?: any }) { const { setApiKey } = props; const { isOpen, onOpen, onClose } = useDisclosure(); const [inputCode, setInputCode] = useState(''); const textColor = useColorModeValue('navy.700', 'white'); const grayColor = useColorModeValue('gray.500', 'gray.500'); const inputBorder = useColorModeValue('gray.200', 'whiteAlpha.200'); const inputColor = useColorModeValue('navy.700', 'white'); const link = useColorModeValue('brand.500', 'white'); const navbarIcon = useColorModeValue('gray.500', 'white'); const toast = useToast(); const handleChange = (Event: any) => { setInputCode(Event.target.value); }; const handleApiKeyChange = (value: string) => { setApiKey(value); localStorage.setItem('apiKey', value); }; return ( <> Import available APIs by providing url to API Documentation Our LLM streamlines the API understanding process by automatically analyzing the content on the documentation and extract the the available APIs. ); } export default APIModal; ``` ### setup.py ```python from setuptools import setup, find_packages setup( name="APIVerse", version="0.0.1", description="Cal Hacks 10.0 Project", author="Jingchao (Perry) Zhong, Jiaxiang (Calvin) Li, Junyi (Alex) Zhang, Yufeng (Jimmy) Li", author_email="", license="Apache License, Version 2.0", packages=find_packages(exclude=("tests",)), python_requires='>=3.8', include_package_data=True, install_requires = open('requirements.txt').readlines(), ) ``` ### gpt_module.py ```python import os import yaml import openai class GPT_module: MODEL_NAME = "gpt-4" def __init__(self): openai.api_key = get_openai_api_key() def query(self, prompt): # answer = self.comp(prompt, MaxToken=30, outputs=1) if len(answer) > 0: print(answer) return answer[0]['message']['content'] else: print("[Error] No response from OpenAI model!") return None def comp(self, PROMPT, MaxToken=50, outputs=1): response = openai.ChatCompletion.create(model=self.MODEL_NAME, messages=[{"role": "system", "content": 'You are a AI that understand all API technical concept'}, {"role": "user", "content": PROMPT}]) return response['choices'] def query_with_conversation(self, prompt, conversation): """" GPT query in a conversation format conversation: [ {"role": "system", "content": "You are a coding tutor bot to help user write and optimize python code."}, {"role": "user", "content": "Brabra"} {"role": "assistant", "content": "Brabra"} ] (Note: remeber to keep increase the conversation) Output: reponse: the response for this prompt conversation: update conversation """ conversation.append({"role": "user", "content": prompt}) response = openai.ChatCompletion.create( model=self.MODEL_NAME, messages=conversation ) conversation.append({"role": "assistant", "content": response["choices"][0]["message"]['content']}) response_message = response["choices"][0]["message"] return response_message, conversation # Read OpenAI key from ~/.cal_hack/config.yaml def get_openai_api_key(): with open(os.path.expanduser("~/.cal_hack/config.yaml"), "r") as f: config = yaml.load(f, Loader=yaml.SafeLoader) if "openai_api_key" not in config: raise Exception("OpenAI API key not found in ~/.cal_hack/config.yaml," " please add 'openai_api_key: YOUR_KEY' in the file.") openai_api_key = config["openai_api_key"] return openai_api_key ``` [61 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]