# Project export: Semantic Drive

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Imagine if your file directory could give more than be a mess of month-old files. With the power of AI and ML using MindsDB, we provide invaluable semantic contexts for your personal files.
- Devpost: https://devpost.com/software/semantic-drive
- GitHub: https://github.com/sachiniyer/semantic-drive
- Team: 4 GitHub contributor(s) — Sachin Iyer (41 commits), Brayton-Lordianto (13 commits), aminoa (8 commits), Thaileaf (3 commits)

## Devpost submission (written by the team)

### Overview

💡

### Inspiration

💡 Navigating through a digital jungle of files and documents is a common struggle for students and professionals alike. The frustration of sifting through folders filled with cryptic file names like "syllabus_fall23 (5).pdf" or "README.txt" is something we all can relate to. The existing file management tools, like Google Drive, although efficient, often fall short in providing us with insightful, semantic information about our files' contents. This sparked our inspiration for a solution that could transform the way we organize and access our digital resources. ⚙️

### What it does

⚙️ We present Semantic Drive, a cloud-based file management platform, akin to Google Drive, but with a revolutionary twist. We've harnessed the power of machine learning technologies, seamlessly integrated with MindsDB, to provide summary data for a wide range of file types. Whether you're dealing with text documents or images, Semantic Drive doesn't just store your files; it comprehends them. AI Track In the AI Track, we've taken file management to a whole new level by infusing it with the intelligence of machine learning. This involves using MindDB's integration with ML models as well as Hugging Face to allow for more accurate semantic searching. 🏗️

### How we built it

🏗️ Our frontend was built with React and our backend was made using Flask. 🟣 CockroachDB Serverless 🟣 Our foundation is built upon CockroachDB Serverless, paired with Python, where we securely store your files and their associated metadata. This secure database forms the backbone of our system, ensuring your data is organized and accessible. We've created a user-friendly Flask backend to streamline the process of entering file data and metadata. This allows for efficient data management and retrieval. 🔴 MindsDB 🔴 MindsDB plays a pivotal role in the Semantic Drive ecosystem. As files are uploaded, MindsDB springs into action, automatically summarizing each file's content. What sets us apart is that this isn't limited to text documents alone; we've engineered a custom handler for MindsDB that can comprehend image data and other file formats, providing detailed, semantic descriptions. The semantic descriptions generated by MindsDB are seamlessly integrated into our Flask backend, enabling you to search for files using these intelligent descriptions, all while leveraging the power of Hugging Face. 🚩

### Challenges we ran into

Throughout our journey, we encountered challenges that pushed us beyond our comfort zones. Exploring new technologies, such as MindsDB and Intel Cloud, presented a learning curve. However, it was a rewarding challenge, as it allowed us to expand our horizons and deepen our understanding of cutting-edge tools and platforms. 🥇

### Accomplishments we're proud of

We've transformed the way files are managed and accessed. Semantic Drive's ability to understand and provide semantic descriptions of files is a significant achievement, making file searching more intuitive and efficient. 📚

### What we learned

Our journey taught us how to harness the power of new APIs and technologies, enriching our skills and broadening our capabilities. ⏳

### What's next

Semantic Drive is the proof that AI can enhance everyday tasks, like file searching, making them more intuitive and efficient. Our vision for the future includes expanding our capabilities further, removing the constraints of exact word searches and extending our semantic understanding to even more file formats, revolutionizing the way we interact with our digital resources. Semantic Drive is just the beginning of a smarter, more intuitive digital world. We hope that file managers like Google Drive will soon integrate a similar service.

## README (from the GitHub repository)

# Sembox

A drive with intelligent semantic searching.

For example, when you search for dog, it will bring up pictures, audios, texts... that contain dog.

Built (mostly) during Calhacks 10.0 (2023)

## Components

Look at each component for more information

- [Frontend](./semantic-drive-frontend)
- [API](./semantic-drive-backend)
- [DB](./semantic-drive-database)

## Potential Improvements

- better models
  - custom summary models (especially text)
    - Currently - size constraints for my hardware
    - Also can get faster inference times with gpu
  - better search model
    - Can be much better - currently, just cosine similarity
- better ui
  - light theme
  - better metadata display
  - lots of clean up (e.g. Loading)
  - User management


## Detected evidence (automated analysis)

Indexed codebase: 25 recognized source files, 71 KB.
- Flask (technology) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (38 of 38)

```
LICENSE.md
README.md
semantic-drive-backend/.dockerignore
semantic-drive-backend/.gitignore
semantic-drive-backend/app.py
semantic-drive-backend/db.py
semantic-drive-backend/Dockerfile
semantic-drive-backend/env.sample
semantic-drive-backend/README.md
semantic-drive-backend/requirements.txt
semantic-drive-backend/search.py
semantic-drive-backend/summary.py
semantic-drive-database/.gitignore
semantic-drive-database/Dockerfile
semantic-drive-database/env.sample
semantic-drive-database/init.sh
semantic-drive-database/README.md
semantic-drive-frontend/.dockerignore
semantic-drive-frontend/.gitignore
semantic-drive-frontend/Dockerfile
semantic-drive-frontend/init.sh
semantic-drive-frontend/jsconfig.json
semantic-drive-frontend/next.config.js
semantic-drive-frontend/package.json
semantic-drive-frontend/README.md
semantic-drive-frontend/src/app/layout.js
semantic-drive-frontend/src/app/page.js
semantic-drive-frontend/src/components/Consts.js
semantic-drive-frontend/src/components/Contexts.js
semantic-drive-frontend/src/components/FileGrid.js
semantic-drive-frontend/src/components/FileList.js
semantic-drive-frontend/src/components/LeftBar.js
semantic-drive-frontend/src/components/ListCard.js
semantic-drive-frontend/src/components/MediaCard.js
semantic-drive-frontend/src/components/SearchBar.js
semantic-drive-frontend/src/components/ThemeRegistry/EmotionCache.js
semantic-drive-frontend/src/components/ThemeRegistry/theme.js
semantic-drive-frontend/src/components/ThemeRegistry/ThemeRegistry.js
```

### Dependencies

- semantic-drive-backend/requirements.txt: flask, flask_cors, pillow, psycopg2-binary, python-dotenv, requests, sentence_transformers, transformers[torch]
- semantic-drive-frontend/package.json: @emotion/cache@latest, @emotion/react@latest, @emotion/styled@latest, @mui/icons-material@latest, @mui/material@latest, eslint@latest, eslint-config-next@latest, next@latest, react@latest, react-dom@latest

### Recent commits (newest first)

- transaction block error handling
- update readme one more time
- update readme
- remove .env
- remove .env
- remove todo
- finished project
- update main readme
- add readmes and small fixes
- just search now
- update todo
- everything except search
- update general readme
- update todo
- remove init.js random file
- everything is dockerized and backend is more concrete
- update env sample
- add custom db
- make list items clicky
- some small errors with the symlinks

## Key source files (fetched from GitHub, selected and truncated for size)

### LICENSE.md

```markdown
                    GNU GENERAL PUBLIC LICENSE
                       Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.

                            Preamble

The GNU General Public License is a free, copyleft license for
software and other kinds of works.

The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.

When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.

To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.

For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.

Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.

For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.

Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this p
[truncated — 31888 more characters]
```

### semantic-drive-backend/requirements.txt

```
flask
flask_cors
requests
psycopg2-binary
python-dotenv
transformers[torch]
pillow
sentence_transformers

```

### semantic-drive-database/Dockerfile

```
FROM postgres:latest

COPY init.sh /docker-entrypoint-initdb.d/
RUN chmod +x /docker-entrypoint-initdb.d/init.sh

EXPOSE 5432


```

### semantic-drive-frontend/Dockerfile

```
FROM node:20

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

RUN chmod +x ./init.sh

EXPOSE 3000
CMD ["./init.sh"]

```

### semantic-drive-backend/Dockerfile

```
FROM python:3.9-slim-buster

RUN apt-get update && apt-get install -y \
    pkg-config \
    cmake \
    ffmpeg

WORKDIR /app
RUN python -m pip install --upgrade pip
COPY requirements.txt /app

RUN pip install -r requirements.txt
COPY . /app
RUN mkdir /app/files

ENTRYPOINT ["python3"]
CMD ["app.py"]

```

### semantic-drive-frontend/package.json

```
{
  "name": "semantic-drive-frontend",
  "version": "5.0.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@emotion/cache": "latest",
    "@emotion/react": "latest",
    "@emotion/styled": "latest",
    "@mui/icons-material": "latest",
    "@mui/material": "latest",
    "eslint": "latest",
    "eslint-config-next": "latest",
    "next": "latest",
    "react": "latest",
    "react-dom": "latest"
  }
}

```

### semantic-drive-backend/app.py

```python
"""
Main file for the backend server.

It contains the following routes:
    - /: GET request for health check
    - /file: GET, POST, DELETE requests for file upload, retrieval, deletion
    - /filedata: GET request for retrieving a file from local storage
    - /files: GET, DELETE request for retrieving all file ids, deleting all
    - /search: GET request for searching for files
"""
# flake8: noqa E402
import base64
import json
import uuid
from flask_cors import cross_origin
from flask import send_from_directory, make_response
import flask
from dotenv import load_dotenv
import logging
import os

load_dotenv()
log_level = os.getenv("LOG_LEVEL", "INFO")
log_level = getattr(logging, log_level.upper())
logger = logging.getLogger()
logging.basicConfig(level=log_level)

from summary import summarize
from search import match
from db import (file_ids, insert_file, find_file, find_filename,
                delete_all, file_summaries_iter, delete_file)

app = flask.Flask(__name__)


def download_file(id, file, fileName):
    """
    Download a file to the local storage.

    Args:
        id (str): the id of the file
        file (bytes): the file to be downloaded
    Returns:
        filePath (str): the path to the downloaded file
    """
    extension = fileName.split('.')[-1]
    download_name = f"files/{id}"
    link = f"files/{id}.{extension}"

    with open(download_name, 'wb') as f:
        f.write(file)
    os.symlink(str(id), link)
    return link


@app.route('/', methods=['GET'])
@cross_origin()
def index():
    """
    Route for the backend server.

    GET request: 200 OK response
    Args:
        None
    Returns:
        str: 200 OK response
    """
    return "200 SERVER OK"


@app.route('/file', methods=['GET', 'POST', 'DELETE'])
@cross_origin()
def file():
    """
    Route for uploading and retrieving files.

    GET request: retrieve a file from the database
    Args:
        fileId (str): the id of the file to be retrieved
    Returns:
        json: the file data

    POST request: upload a file to the database
    Args:
        uploadTime (str): the time the file was uploaded
        fileType (str): the type of the file
        fileName (str): the name of the file
        file (bytes): the file to be uploaded
    Returns:
        json: the id of the newly added file

    DELETE request: delete a file from the database
    Args:
        fileId (str): the id of the file to be deleted
    Returns:
        str: 200 OK response
    """
    if flask.request.method == 'GET':
        response = find_file(flask.request.args.get('fileId'))

        return json.dumps({
            "uploadTime": response[1],
            "fileType": response[2],
            "fileName": response[3],
            "file": response[4],
            "summary": base64.b64decode(response[5]).decode('utf-8')})

    elif flask.request.method == 'POST':
        id = uuid.uuid4()
        uploadTime = flask.request.form['uploadTime']
        fileType = flask.request.form['fileType']
        fileName = flask.request.form['fileName']
        fileData = flask.request.files['file'].read()

        url = "/filedata?fileId=" + str(id)

        download_name = download_file(id, fileData, fileName)
        summary = base64.b64encode(str.encode(summarize(fileType, download_name)))
        entry = (id, uploadTime, fileType, fileName, url, summary)

        insert_file(entry)

        return json.dumps({"fileId": str(id)})

    elif flask.request.method == 'DELETE':
        id = flask.request.form['fileId']

        delete_file(id)

        return "200 OK"


@app.route('/filedata', methods=['GET'])
@cross_origin()
def filedata():
    """
    Route for retrieving a file from local storage.

    GET request: retrieve a file from local storage
    Args:
        fileId (str): the id of the file to be retrieved
    Returns:
        file: the file to be retrieved
    """
    id = flask.request.args.get('fileId')
    fileName = find_filename(id)
    file = send_from_directory('files', id, as_attachment=True, download_name=fileName)
    return file


@app.route('/files', methods=['GET', 'DELETE'])
@cross_origin()
def files():
    """
    Route for retrieving all file ids.

    GET request: retrieve all file ids
    Args:
        None
    Returns:
        json: the ids of all files

    DELETE request: delete all files from the database
    Args:
        None
    Returns:
        str: 200 OK response
    """
    if flask.request.method == 'GET':
        return file_ids()
    else:
        delete_all()
        os.system('rm -rf files/*')
        return "200 OK"


@app.route('/search', methods=['GET'])
@cross_origin()
def search():
    """
    Route for searching for files.

    GET request: search for files
    Args:
        terms (str): the search terms
    Returns:
        json: the ids of the files that match the search terms
    """
    terms = flask.request.args.get('terms')
    summaries = file_summaries_iter()
    results = match(summaries, terms)
    return json.dumps({"fileIds": results})


if __name__ == '__main__':
    app.run(host=os.getenv("BIND_IP", "127.0.0.1"),
            port=int(os.getenv("PORT", "8000")),
            debug=(log_level == logging.DEBUG))

```

### semantic-drive-frontend/src/app/layout.js

```javascript
import * as React from "react";
import Box from "@mui/material/Box";
import Drawer from "@mui/material/Drawer";
import Divider from "@mui/material/Divider";
import ThemeRegistry from "@/components/ThemeRegistry/ThemeRegistry";
import SearchBar from "@/components/SearchBar";
import { LayoutRegistry, FilesRegistry, FileIdRegistry } from "@/components/Contexts";
import LeftBar from "@/components/LeftBar";

export const metadata = {
  title: "Semantic Drive",
  description: "Semantic Drive",
};

export const DRAWER_WIDTH = 240;

export default function RootLayout({ children }) {
  return (
    <html lang="en">
      <body>
        <ThemeRegistry>
          <LayoutRegistry>
            <FilesRegistry>
              <FileIdRegistry>
                <Drawer
                  sx={{
                    width: DRAWER_WIDTH,
                    flexShrink: 0,
                    "& .MuiDrawer-paper": {
                      width: DRAWER_WIDTH,
                      boxSizing: "border-box",
                      top: ["48px", "56px", "64px"],
                      height: "auto",
                      bottom: 0,
                    },
                  }}
                  variant="permanent"
                  anchor="left"
                >
                  <SearchBar drawerWidth={DRAWER_WIDTH} />
                  <Divider />
                  <LeftBar />
                </Drawer>
                <Box
                  component="main"
                  sx={{
                    flexGrow: 1,
                    bgcolor: "background.default",
                    ml: `${DRAWER_WIDTH}px`,
                    mt: ["48px", "56px", "64px"],
                    p: 3,
                  }}
                >
                  {children}
                </Box>
              </FileIdRegistry>
            </FilesRegistry>
          </LayoutRegistry>
        </ThemeRegistry>
      </body>
    </html>
  );
}

```

### semantic-drive-frontend/src/app/page.js

```javascript
"use client";
import * as React from "react";
import { useContext, useEffect, useState } from "react";
import Box from "@mui/material/Box";
import Button from "@mui/material/Button";
import Drawer from "@mui/material/Drawer";
import Typography from "@mui/material/Typography";
import List from "@mui/material/List";
import ListItem from "@mui/material/ListItem";
import FileGrid from "@/components/FileGrid";
import FileList from "@/components/FileList";
import { LayoutContext, FilesContext, FileIdContext } from "@/components/Contexts";
import { API } from "@/components/Consts"

export default function HomePage() {
  let [layout, _setLayout] = useContext(LayoutContext);
  let [_files, setFiles] = useContext(FilesContext);
  let [_fileId, setFileId] = useContext(FileIdContext);
  let [newId, setNewId] = useState(0);
  let [displayFile, setDisplayFile] = useState(false);
  let [displayFileContent, setDisplayFileContent] = useState(<></>);

  useEffect(() => {
    fetch(`${API}/files`)
      .then(response => response.json())
      .then(data => {
        setFiles(data)
      });
  }, []);

  function closeDisplayFile() {
    setDisplayFile(false);
  }

  useEffect(() => {
    if (displayFile == false) {
      setDisplayFileContent(<></>);
      setFileId(0);
    } else {
      setFileId(newId);
      fetch(`${API}/file?fileId=${newId}`, {
        method: "GET",
      }).then(response => response.json())
        .then(data => {
          setDisplayFileContent(
            <Drawer
              sx={{
                width: 320,
                flexShrink: 0,
                "& .MuiDrawer-paper": {
                  width: 320,
                  boxSizing: "border-box",
                  top: ["48px", "56px", "64px"],
                  height: "auto",
                  bottom: 0,
                },
              }}
              elevation={0}
              open={true}
              variant="permanent"
              anchor="right"
            >
              <Button onClick={closeDisplayFile}>Close</Button>
              <List sx={{ px: 2 }}>
                <ListItem disablePadding>
                  <Typography variant="overline" sx={{ fontWeight: 500 }}>
                    <h3> Metadata: </h3>
                    <ul>
                      <li> File Name: {data.fileName} </li>
                      <li> File Type: {data.fileType} </li>
                      <li> Upload Time: {data.uploadTime} </li>
                      <li> Summary: {data.summary} </li>
                    </ul>
                  </Typography>
                </ListItem>
              </List>
            </Drawer>,

          )
        });
    }
  }, [newId, displayFile]);

  function changeDisplayFile(id) {
    setNewId(id);
    setDisplayFile(true);
  }

  function fileContent() {
    if (layout == "grid") {
      return <FileGrid displayFile={changeDisplayFile} />;
    } else {
      return <FileList displayFile={changeDisplayFile} />;
    }
  }

  return (
    <Box sx={{ display: "flex" }}>
      <Box>{fileContent()}</Box>
      {displayFileContent}
    </Box>
  );
}

```

### semantic-drive-frontend/init.sh

```shell
npm run build
npm run start

```

[16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]