# Project export: Sift

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Search everything, find anything.
- Devpost: https://devpost.com/software/sift-8sgr4f
- GitHub: https://github.com/abhi-arya1/siftai
- Demo: https://try-sift-ai.vercel.app/
- Team: 3 GitHub contributor(s) — Abhi Arya (67 commits), Hrishikesh Srihari (20 commits), Ivan Vuong (19 commits)

## Devpost submission (written by the team)

### Inspiration

As students and tech enthusiasts, we use many different platforms day-to-day, oftentimes completing a wide range of tasks while also expanding our digital footprints and knowledge bases. With this resourcefulness arises a problem: Wasting time combing through different platforms with the faintest idea of where something might be, opening a plethora of different tabs, or distributing information to pesky paid services. Managing everything from midterm season to hackathon deadlines, to finding the perfect photograph, and wasting time by searching your work is what Sift aims to combat.

### What it does

Sift allows you to simultaneously search files in your filesystem, GitHub, Notion, Slack, Discord, and the Google Suite, simply with natural language. You can view your files directly in Sift's interface, and get a specialized summary in real-time based on the context of your query, with the power of Groq, streamlining your workflow.

### How we built it

Sift uses the Tauri framework to bundle together a robust Rust-based backend with a Next.js/React frontend, running an optimized build for either Mac or Windows natively. By powering all of our services through Rust, we're able to maintain blazingly fast real-time search speeds while maintaining familiarity with the frontend. Using Rust, we also integrate with multiple OAuth APIs over Sockets, and run a wrapped version of ChromaDB through our backend, allowing us to contain every service entirely on-device with a custom set of Python bindings. By using Chroma, we're able to use natural language to simultaneously query multiple knowledge bases (with Multimodal Embeddings) for relevant data, returning the most confident results to the user. Upon that, Groq generates instant inference to speed up your search, and you can preview files with Rust and Next.js bindings through Tauri.

### Challenges we ran into

OAuth Integrations on the Desktop / In Rust Lack of Browser APIs in the Tauri Environment Building for Cross-Platform Compatability Using Inference to determine Actionable Files Querying hundreds of thousands of files in seconds

### Accomplishments we're proud of

Learning Rust from start to finish. Learning Vector Databases, including embeddings and metadata Building our first end-to-end Desktop App Containing everything on-device for security and efficiency

### What we learned

Rust 🦀 Desktop App Development Vector Databases Binding Rust, Python, and TypeScript SDKs simultaneously How to Sleep during a Hackathon

### What's next

In the future, we plan to add inference functionality that suggests actions to take in real-time, such as managing calendar events, starting a note, or running code processes. Also, implementing focus-based search is something that really intrigues us as we look forward in Sift's journey, and specializing the platform's tools for different knowledge bases can empower your workflow with Sift even further.

## README (from the GitHub repository)

<div align="center">
    <img width="400" alt="Screenshot 2024-10-20 at 10 47 30 AM" src="https://github.com/user-attachments/assets/b6581642-d9cf-41ca-b484-40c680d2c1c2">
    <h1>Sift</h1>
    <h3><i>Search everything, find anything</i></h3>
    <span><a href="https://devpost.com/software/sift-8sgr4f">Calhacks 11.0 Submission</a></span>
</div>

## Inspiration
As students and tech enthusiasts, we use many different platforms day-to-day, oftentimes completing a wide range of tasks while also expanding our digital footprints and knowledge bases. With this resourcefulness arises a problem: Wasting time combing through different platforms with the faintest idea of where something might be, opening a plethora of different tabs, or distributing information to pesky paid services. Managing everything from midterm season to hackathon deadlines, to finding the perfect photograph, and wasting time by searching your work is what Sift aims to combat.

## What it does
Sift allows you to simultaneously search files in your filesystem, GitHub, Notion, Slack, Discord, and the Google Suite, simply with natural language. You can view your files directly in Sift's interface, and get a specialized summary in real-time based on the context of your query, with the power of Groq, streamlining your workflow.

## How we built it
Sift uses the Tauri framework to bundle together a robust Rust-based backend with a Next.js/React frontend, running an optimized build for either Mac or Windows natively. By powering all of our services through Rust, we're able to maintain blazingly fast real-time search speeds while maintaining familiarity with the frontend. Using Rust, we also integrate with multiple OAuth APIs over Sockets, and run a wrapped version of ChromaDB through our backend, allowing us to contain every service entirely on-device with a custom set of Python bindings. By using Chroma, we're able to use natural language to simultaneously query multiple knowledge bases (with Multimodal Embeddings) for relevant data, returning the most confident results to the user. Upon that, Groq generates instant inference to speed up your search, and you can preview files with Rust and Next.js bindings through Tauri.

## Challenges we ran into
- OAuth Integrations on the Desktop / In Rust
- Lack of Browser APIs in the Tauri Environment
- Building for Cross-Platform Compatability
- Using Inference to determine Actionable Files
- Querying hundreds of thousands of files in seconds

## Accomplishments that we're proud of
- Learning Rust from start to finish.
- Learning Vector Databases, including embeddings and metadata
- Building our first end-to-end Desktop App
- Containing everything on-device for security and efficiency

## What we learned
Rust 🦀
- Desktop App Development
- Vector Databases
- Binding Rust, Python, and TypeScript SDKs simultaneously
- How to Sleep during a Hackathon

## What's next for Sift
In the future, we plan to add inference functionality that suggests actions to take in real-time, such as managing calendar events, starting a note, or running code processes. Also, implementing focus-based search is something that really intrigues us as we look forward in Sift's journey, and specializing the platform's tools for different knowledge bases can empower your workflow with Sift even further.


## Detected evidence (automated analysis)

Indexed codebase: 60 recognized source files, 185 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Rust (language) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- JavaScript (language) — claimed on Devpost, not found in the code
- Node.js (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (83 of 83)

```
.gitignore
app/file_preview.tsx
app/globals.css
app/layout.tsx
app/page.tsx
app/search_box.tsx
bun.lockb
clean_ports.sh
components.json
components/renderer/markdown_styles.css
components/renderer/parser.ts
components/renderer/renderer.css
components/theme-provider.tsx
components/ui/input.tsx
components/ui/scroll-area.tsx
lib/chromalib.ts
lib/groq_parsing.ts
lib/utils.ts
next.config.mjs
package.json
postcss.config.mjs
README.md
sift-landing/.gitignore
sift-landing/app/globals.css
sift-landing/app/layout.tsx
sift-landing/app/page.tsx
sift-landing/bun.lockb
sift-landing/components.json
sift-landing/components/content.tsx
sift-landing/components/footer.tsx
sift-landing/components/hero.tsx
sift-landing/components/navbar.tsx
sift-landing/components/theme-provider.tsx
sift-landing/components/ui/bento-grid.tsx
sift-landing/components/ui/button.tsx
sift-landing/components/ui/dropdown-menu.tsx
sift-landing/components/ui/marquee.tsx
sift-landing/components/ui/meteors.tsx
sift-landing/components/ui/navigation-menu.tsx
sift-landing/components/ui/shiny-button.tsx
sift-landing/components/ui/sparkles-text.tsx
sift-landing/components/ui/toggle.tsx
sift-landing/lib/utils.ts
sift-landing/next.config.mjs
sift-landing/package.json
sift-landing/postcss.config.mjs
sift-landing/README.md
sift-landing/tailwind.config.ts
sift-landing/tsconfig.json
src-tauri/.gitignore
src-tauri/.taurignore
src-tauri/auth_page.html
src-tauri/build.rs
src-tauri/Cargo.lock
src-tauri/Cargo.toml
src-tauri/icons/icon.icns
src-tauri/pybindings/.gitignore
src-tauri/pybindings/chroma_sdk.py
src-tauri/pybindings/google.py
src-tauri/pybindings/init_gh.py
src-tauri/pybindings/init_local.py
src-tauri/pybindings/init_notion.py
src-tauri/pybindings/requirements.txt
src-tauri/pybindings/slack.py
src-tauri/response.example.json
src-tauri/services.md
src-tauri/src/apis/fileserv.rs
src-tauri/src/apis/mod.rs
src-tauri/src/chroma/chrapi.rs
src-tauri/src/chroma/chrust.rs
src-tauri/src/chroma/mod.rs
src-tauri/src/invokes/auth_page.html
src-tauri/src/invokes/github_get.rs
src-tauri/src/invokes/mod.rs
src-tauri/src/invokes/oauth.rs
src-tauri/src/invokes/run_cmd.rs
src-tauri/src/main.rs
src-tauri/src/queryapi.py
src-tauri/src/util/config.rs
src-tauri/src/util/mod.rs
src-tauri/tauri.conf.json
tailwind.config.ts
tsconfig.json
```

### Dependencies

- package.json: @emotion/react@^11.13.3, @emotion/styled@^11.13.0, @fortawesome/fontawesome-svg-core@^6.6.0, @fortawesome/free-brands-svg-icons@^6.6.0, @fortawesome/react-fontawesome@^0.2.2, @headlessui/react@^2.1.10, @mui/icons-material@^6.1.4, @nextui-org/kbd@^2.0.34, @radix-ui/react-icons@^1.3.0, @radix-ui/react-scroll-area@^1.2.0, @tabler/icons-react@^3.19.0, @tauri-apps/api@1.0.0, @tauri-apps/cli@1.0.0, @types/katex@^0.16.7, @types/node@^20, @types/prismjs@^1.26.4, @types/react@^18, @types/react-dom@^18, autoprefixer@^10.4.20, chromadb@^1.9.2, chromadb-default-embed@^2.13.2, class-variance-authority@^0.7.0, clsx@^2.1.1, framer-motion@^11.11.9, groq-sdk@^0.7.0, katex@^0.16.11, lucide-react@^0.453.0, next@14.2.15, next-themes@^0.3.0, postcss@^8, postcss-cli@^11.0.0, prismjs@^1.29.0, react@^18, react-dom@^18, remark@^15.0.1, remark-html@^16.0.1, tailwind-merge@^2.5.4, tailwindcss@^3.4.14, tailwindcss-animate@^1.0.7, tauri@^0.15.0, typescript@^5
- sift-landing/package.json: @fortawesome/free-brands-svg-icons@^6.6.0, @fortawesome/react-fontawesome@^0.2.2, @radix-ui/react-dropdown-menu@^2.1.2, @radix-ui/react-icons@^1.3.0, @radix-ui/react-navigation-menu@^1.2.1, @radix-ui/react-slot@^1.1.0, @radix-ui/react-toggle@^1.1.0, @tabler/icons-react@^3.19.0, @types/node@^20, @types/react@^18, @types/react-dom@^18, class-variance-authority@^0.7.0, clsx@^2.1.1, framer-motion@^11.11.9, ldrs@^1.0.2, lucide-react@^0.453.0, next@14.2.15, next-themes@^0.3.0, postcss@^8, react@^18, react-dom@^18, react-icons@^5.3.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5
- src-tauri/Cargo.toml: actix-cors@0.6, actix-files@0.6.0-beta.16, actix-web@4.0, anyhow@1.0, async-recursion@0.1.0, async-std@1.10, base64@0.13, chromadb@0.5.0, crossbeam-channel@0.5, dirs@3.0, dotenv@0.15, dotenvy@0.15, futures@0.3, ignore@0.4, lru-cache@0.1.2, mime_guess@2.0, num_cpus@1.14, octocrab@0.10.0, oneshot@0.1.8, open@5.0.0, pdf-extract@0.7.10, rayon@1.5, reqwest@0.11, serde@1.0, serde_json@1.0, serde_qs@0.12.0, tauri@1.0.0, tauri-build@1.0.0, tauri-plugin-localhost@0.1.0, thiserror@1.0, threadpool@1.8, tokio@1, tokio-util@0.6, urlencoding@2.1.3, warp@0.3
- src-tauri/pybindings/requirements.txt: altgraph@==0.17.4, annotated-types@==0.7.0, anyio@==4.6.2.post1, asgiref@==3.8.1, backoff@==2.2.1, bcrypt@==4.2.0, build@==1.2.2.post1, cachetools@==5.5.0, certifi@==2024.8.30, charset-normalizer@==3.4.0, chroma-hnswlib@==0.7.6, chromadb@==0.5.15, click@==8.1.7, coloredlogs@==15.0.1, Deprecated@==1.2.14, durationpy@==0.9, fastapi@==0.115.2, filelock@==3.16.1, flatbuffers@==24.3.25, fsspec@==2024.9.0, google-auth@==2.35.0, googleapis-common-protos@==1.65.0, grpcio@==1.67.0, h11@==0.14.0, httpcore@==1.0.6, httptools@==0.6.4, httpx@==0.27.2, huggingface-hub@==0.26.0, humanfriendly@==10.0, idna@==3.10, importlib_metadata@==8.4.0, importlib_resources@==6.4.5, kubernetes@==31.0.0, macholib@==1.16.3, markdown-it-py@==3.0.0, mdurl@==0.1.2, mmh3@==5.0.1, monotonic@==1.6, mpmath@==1.3.0, numpy@==2.1.2, oauthlib@==3.2.2, onnxruntime@==1.19.2, opentelemetry-api@==1.27.0, opentelemetry-exporter-otlp-proto-common@==1.27.0, opentelemetry-exporter-otlp-proto-grpc@==1.27.0, opentelemetry-instrumentation@==0.48b0, opentelemetry-instrumentation-asgi@==0.48b0, opentelemetry-instrumentation-fastapi@==0.48b0, opentelemetry-proto@==1.27.0, opentelemetry-sdk@==1.27.0, opentelemetry-semantic-conventions@==0.48b0, opentelemetry-util-http@==0.48b0, orjson@==3.10.7, overrides@==7.7.0, packaging@==24.1, posthog@==3.7.0, protobuf@==4.25.5, pyasn1@==0.6.1, pyasn1_modules@==0.4.1, pydantic@==2.9.2, pydantic_core@==2.23.4, Pygments@==2.18.0, pyinstaller@==6.11.0, pyinstaller-hooks-contrib@==2024.9, PyPika@==0.48.9, pyproject_hooks@==1.2.0, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, PyYAML@==6.0.2, requests@==2.32.3, requests-oauthlib@==2.0.0, rich@==13.9.2, rsa@==4.9, setuptools@==75.2.0, shellingham@==1.5.4, six@==1.16.0, sniffio@==1.3.1, starlette@==0.40.0, sympy@==1.13.3, tenacity@==9.0.0, tokenizers@==0.20.1, tqdm@==4.66.5, typer@==0.12.5, typing_extensions@==4.12.2, urllib3@==2.2.3, uvicorn@==0.32.0, uvloop@==0.21.0, watchfiles@==0.24.0, websocket-client@==1.8.0, websockets@==13.1, wrapt@==1.16.0, zipp@==3.20.2

### Recent commits (newest first)

- Update README.md
- we outtie (fr)
- we outtie
- Merge branch 'main' of https://github.com/abhi-arya1/siftai
- slackination
- final landing
- PDF MASHALLAH
- stuff
- stuff
- stuff
- test
- test
- stuff
- add readme
- updates
- groq
- notion done
- notion done
- add some notion stuff
- Merge branch 'main' of https://github.com/abhi-arya1/siftai

## Key source files (fetched from GitHub, selected and truncated for size)

### src-tauri/services.md

```markdown
# Rust-based Ports

(all hosted on `localhost:{PORT}`)

- 35435 - GitHub OAuth

  - Run this by invoking `gh_oauth` from the Next.js Frontend

- 35436 - Chroma Database

  - Persistently Running

- 35438 - File Server API

  - Run this by requesting `http://localhost:35438/path/to/file` from the Next.js Frontend
  - The path must be beyond the user's HOME directory.
  - For example, if the abs path is `/Users/name/Documents/file.txt`, the path should be `/Documents/file.txt`

- 35439 - Slack OAuth

  - Run this by invoking `slk_oauth` from the Next.js frontend

- 35440 - Discord OAuth

  - Run this by invoking `disc_oauth` from the Next.js frontend

- 35441 - Notion OAuth

  - Run this by invoking `ntn_oauth` from the Next.js frontend

- 35442 - Google OAuth

  - Run this by invoking `ggl_oauth` from the Next.js frontend

- 35443 - Chroma Server
  - Run this by requesting `http://localhost:35443/query/{query}/{results#}` from the Next.js frontend

```

### package.json

```
{
  "name": "siftai",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@emotion/react": "^11.13.3",
    "@emotion/styled": "^11.13.0",
    "@fortawesome/fontawesome-svg-core": "^6.6.0",
    "@fortawesome/free-brands-svg-icons": "^6.6.0",
    "@fortawesome/react-fontawesome": "^0.2.2",
    "@headlessui/react": "^2.1.10",
    "@mui/icons-material": "^6.1.4",
    "@nextui-org/kbd": "^2.0.34",
    "@radix-ui/react-icons": "^1.3.0",
    "@radix-ui/react-scroll-area": "^1.2.0",
    "@tabler/icons-react": "^3.19.0",
    "@tauri-apps/api": "1.0.0",
    "@types/katex": "^0.16.7",
    "@types/prismjs": "^1.26.4",
    "autoprefixer": "^10.4.20",
    "chromadb": "^1.9.2",
    "chromadb-default-embed": "^2.13.2",
    "class-variance-authority": "^0.7.0",
    "clsx": "^2.1.1",
    "framer-motion": "^11.11.9",
    "groq-sdk": "^0.7.0",
    "katex": "^0.16.11",
    "lucide-react": "^0.453.0",
    "next": "14.2.15",
    "next-themes": "^0.3.0",
    "postcss-cli": "^11.0.0",
    "prismjs": "^1.29.0",
    "react": "^18",
    "react-dom": "^18",
    "remark": "^15.0.1",
    "remark-html": "^16.0.1",
    "tailwind-merge": "^2.5.4",
    "tailwindcss-animate": "^1.0.7",
    "tauri": "^0.15.0"
  },
  "devDependencies": {
    "@tauri-apps/cli": "1.0.0",
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "postcss": "^8",
    "tailwindcss": "^3.4.14",
    "typescript": "^5"
  }
}

```

### sift-landing/package.json

```
{
  "name": "sift-landing",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@fortawesome/free-brands-svg-icons": "^6.6.0",
    "@fortawesome/react-fontawesome": "^0.2.2",
    "@radix-ui/react-dropdown-menu": "^2.1.2",
    "@radix-ui/react-icons": "^1.3.0",
    "@radix-ui/react-navigation-menu": "^1.2.1",
    "@radix-ui/react-slot": "^1.1.0",
    "@radix-ui/react-toggle": "^1.1.0",
    "@tabler/icons-react": "^3.19.0",
    "class-variance-authority": "^0.7.0",
    "clsx": "^2.1.1",
    "framer-motion": "^11.11.9",
    "ldrs": "^1.0.2",
    "lucide-react": "^0.453.0",
    "next": "14.2.15",
    "next-themes": "^0.3.0",
    "react": "^18",
    "react-dom": "^18",
    "react-icons": "^5.3.0",
    "tailwind-merge": "^2.5.4",
    "tailwindcss-animate": "^1.0.7"
  },
  "devDependencies": {
    "typescript": "^5",
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "postcss": "^8",
    "tailwindcss": "^3.4.1"
  }
}

```

### src-tauri/Cargo.toml

```
[package]
name = "app"
version = "0.1.0"
description = "A Tauri App"
authors = ["you"]
license = ""
repository = ""
default-run = "app"
edition = "2021"
rust-version = "1.57"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[build-dependencies]
tauri-build = { version = "1.0.0", features = [] }

[dependencies]
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
tauri = { version = "1.0.0", features = ["api-all", "macos-private-api"] }
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.11", features = ["json"] }
dotenvy = "0.15"
dotenv = "0.15"
warp = { version = "0.3", features = ["tls"] }
open = "5.0.0"
ignore = "0.4"
rayon = "1.5"
dirs = "3.0"
threadpool = "1.8"
crossbeam-channel = "0.5"
num_cpus = "1.14"
tauri-plugin-localhost = "0.1.0"
mime_guess = "2.0"
oneshot = "0.1.8"
tokio-util = "0.6"
actix-web = "4.0"
actix-files = { version = "0.6.0-beta.16" }
base64 = "0.13"
actix-cors = "0.6"
thiserror = "1.0"
urlencoding = "2.1.3"
octocrab = "0.10.0"
futures = "0.3"
anyhow = "1.0"
async-std = "1.10"
async-recursion = "0.1.0"
lru-cache = "0.1.2"
serde_qs = "0.12.0"
chromadb = "0.5.0"
pdf-extract = "0.7.10"


[features]
# by default Tauri runs in production mode
# when `tauri dev` runs it is executed with `cargo run --no-default-features` if `devPath` is an URL
default = ["custom-protocol"]
# this feature is used used for production builds where `devPath` points to the filesystem
# DO NOT remove this
custom-protocol = ["tauri/custom-protocol"]

```

### src-tauri/pybindings/requirements.txt

```
altgraph==0.17.4
annotated-types==0.7.0
anyio==4.6.2.post1
asgiref==3.8.1
backoff==2.2.1
bcrypt==4.2.0
build==1.2.2.post1
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.4.0
chroma-hnswlib==0.7.6
chromadb==0.5.15
click==8.1.7
coloredlogs==15.0.1
Deprecated==1.2.14
durationpy==0.9
fastapi==0.115.2
filelock==3.16.1
flatbuffers==24.3.25
fsspec==2024.9.0
google-auth==2.35.0
googleapis-common-protos==1.65.0
grpcio==1.67.0
h11==0.14.0
httpcore==1.0.6
httptools==0.6.4
httpx==0.27.2
huggingface-hub==0.26.0
humanfriendly==10.0
idna==3.10
importlib_metadata==8.4.0
importlib_resources==6.4.5
kubernetes==31.0.0
macholib==1.16.3
markdown-it-py==3.0.0
mdurl==0.1.2
mmh3==5.0.1
monotonic==1.6
mpmath==1.3.0
numpy==2.1.2
oauthlib==3.2.2
onnxruntime==1.19.2
opentelemetry-api==1.27.0
opentelemetry-exporter-otlp-proto-common==1.27.0
opentelemetry-exporter-otlp-proto-grpc==1.27.0
opentelemetry-instrumentation==0.48b0
opentelemetry-instrumentation-asgi==0.48b0
opentelemetry-instrumentation-fastapi==0.48b0
opentelemetry-proto==1.27.0
opentelemetry-sdk==1.27.0
opentelemetry-semantic-conventions==0.48b0
opentelemetry-util-http==0.48b0
orjson==3.10.7
overrides==7.7.0
packaging==24.1
posthog==3.7.0
protobuf==4.25.5
pyasn1==0.6.1
pyasn1_modules==0.4.1
pydantic==2.9.2
pydantic_core==2.23.4
Pygments==2.18.0
pyinstaller==6.11.0
pyinstaller-hooks-contrib==2024.9
PyPika==0.48.9
pyproject_hooks==1.2.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
PyYAML==6.0.2
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.9.2
rsa==4.9
setuptools==75.2.0
shellingham==1.5.4
six==1.16.0
sniffio==1.3.1
starlette==0.40.0
sympy==1.13.3
tenacity==9.0.0
tokenizers==0.20.1
tqdm==4.66.5
typer==0.12.5
typing_extensions==4.12.2
urllib3==2.2.3
uvicorn==0.32.0
uvloop==0.21.0
watchfiles==0.24.0
websocket-client==1.8.0
websockets==13.1
wrapt==1.16.0
zipp==3.20.2

```

### app/layout.tsx

```typescript
import type { Metadata } from "next";
import { Inter } from "next/font/google";
import "./globals.css";
import { ThemeProvider } from "@/components/theme-provider";

const inter = Inter({ subsets: ["latin"] });

export const metadata: Metadata = {
  title: "Sift.ai",
  // description: "Generated by create next app",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en" className="overscroll-none overflow-clip">
      <body className={inter.className}>
        <ThemeProvider
          attribute="class"
          defaultTheme="system"
          enableSystem
          disableTransitionOnChange
        >
          {children}
        </ThemeProvider>
      </body>
    </html>
  );
}

```

### sift-landing/app/layout.tsx

```typescript
import type { Metadata } from "next";
import { Inter } from "next/font/google";
import "./globals.css";
import { ThemeProvider } from "@/components/theme-provider";

const inter = Inter({ subsets: ["latin"] });

export const metadata: Metadata = {
  title: "SiftAI",
  description: "Search everything, find anything.",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <ThemeProvider
        attribute="class"
        defaultTheme="light"
        storageKey="tuna-theme"
      >
        <body className={`${inter.className} transition-all duration-300`}>
          {children}
        </body>
      </ThemeProvider>
    </html>
  );
}

```

### sift-landing/app/page.tsx

```typescript
import Content from "@/components/content";
import Footer from "@/components/footer";
import Hero from "@/components/hero";

import dynamic from "next/dynamic";
const Navbar = dynamic(() => import("@/components/navbar"), {
  ssr: false,
});

export default function Home() {
  return (
    <div>
      <main className="flex min-h-screen flex-col items-center justify-center">
        <Navbar />
        {/* <div className="absolute h-[200vw] max-h-[2000px] md:max-h-[1000px] lg:max-h-[1000px] left-0 right-0 top-0 lg:top-0 md:top-0 bg-[linear-gradient(to_right,#4f4f4f2e_1px,transparent_1px),linear-gradient(to_bottom,#4f4f4f2e_1px,transparent_1px)] bg-[size:14px_24px] [mask-image:radial-gradient(ellipse_80%_50%_at_50%_0%,#000_70%,transparent_110%)]" />  */}
        <Hero />
        <Content />
      </main>
      <Footer />
    </div>
  );
}

```

### app/page.tsx

```typescript
"use client";
import React, {
  useCallback,
  useState,
  useEffect,
  useRef,
  Fragment,
} from "react";
import Image from "next/image";
import SearchBox from "./search_box";
import { remark } from "remark";
import html from "remark-html";
// import sift_logo from "../src-tauri/icons/sift_logo.png";
import { Menu, Transition, Dialog } from "@headlessui/react";
import { motion } from "framer-motion";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import GoogleIcon from "@mui/icons-material/Google";
import { faDiscord } from "@fortawesome/free-brands-svg-icons";
import { getClient, ResponseType } from "@tauri-apps/api/http";
import {
  Command,
  Search,
  Settings,
  Slack,
  CheckIcon,
  ArrowUpRight,
  ArrowDown,
  MoveUpRight,
  Zap,
  XIcon,
  Code,
  FileText,
  ImageIcon,
} from "lucide-react";
import {
  IconBrandGithub,
  IconBrandNotion,
  IconBrandGoogle,
  IconBrandDiscordFilled,
} from "@tabler/icons-react";
import { ScrollArea } from "@/components/ui/scroll-area";
import { Input } from "@/components/ui/input";
import { Kbd } from "@nextui-org/kbd";
import { cn } from "@/lib/utils";
import { invoke } from "@tauri-apps/api/tauri";
import { queryChroma, ChromaFile } from "@/lib/chromalib";
import { getSummary } from "@/lib/groq_parsing";
import { parseMarkdown } from "@/components/renderer/parser";

type SearchResultItem = {
  id: number;
  filename: string;
  abspath: string;
  local: boolean;
  filecontent: string;
};



const mockResults: SearchResultItem[] = [
  {
    id: 1,
    filename: "random.txt",
    abspath: "/Documents/random.txt",
    local: true,
    filecontent: "This is a sample document content...",
  },
  {
    id: 2,
    filename: "image.jpg",
    abspath: "https://example.com/image.jpg",
    local: true,
    filecontent: "https://example.com/image.jpg",
  },
];

const IntegrationCard = ({ logo: Logo, name, isAuthenticated, onClick }) => {
  const [isHovered, setIsHovered] = useState(false);

  return (
    <div
      className="w-[50%] p-3 flex flex-grid rounded-lg dark:bg-[#1f1f1f] border border-1 hover:disabled:border border-bg-white/10 hover:bg-gray-100 dark:hover:bg-white/10 transition-colors duration-150 ease-in-out cursor-pointer"
      onClick={onClick}
      onMouseEnter={() => setIsHovered(true)}
      onMouseLeave={() => setIsHovered(false)}
    >
      <div className="flex items-center gap-x-2">
        <Logo className="text-black dark:text-white" />
        <span className="dark:text-white">{name}</span>
      </div>
      {isAuthenticated ? (
        <CheckIcon className="pl-2 text-green-500" />
      ) : isHovered ? (
        <MoveUpRight className="pl-1 ml-auto text-gray-500 transition-all duration-300 hover:scale-110 ease-in-out" />
      ) : (
        <ArrowUpRight className="pl-1 ml-auto text-gray-500 transition-all duration-300 hover:scale-110 ease-in-out" />
      )}
    </div>
  );
};

const actionSchema = {
  txt: [
    { id: "copyContents", label: "Copy Contents", shortcut: "⌘ C", command: "copyContents" },
    { id: "copyFilePath", label: "Copy Filepath", shortcut: "⌘ ⇧ C", command: "copyFilePath" },
  ],
  image: [
    { id: "copyFilePath", label: "Copy Filepath", shortcut: "⌘ ⇧ C", command: "copyFilePath" },
    { id: "view", label: "View Image in New Tab", shortcut: "⌘ V", command: "viewImage" },
  ],
  pdf: [
    { id: "copyFilePath", label: "Copy Filepath", shortcut: "⌘ ⇧ C", command: "copyFilePath" },
    { id: "open", label: "Open in Viewer", shortcut: "⌘ O", command: "openFile" },
  ],
  code: [
    { id: "copyContents", label: "Copy Contents", shortcut: "⌘ C", command: "copyContents" },
    { id: "copyFilePath", label: "Copy Filepath", shortcut: "⌘ ⇧ C", command: "copyFilePath" },
  ],
  default: [
    { id: "copyContents", label: "Copy Contents", shortcut: "⌘ C", command: "copyContents" },
    { id: "copyFilePath", label: "Copy Filepath", shortcut: "⌘ ⇧ C", command: "copyFilePath" },
  ],
};


const getFileType = (filename: string): string => {
  const ext = filename.split(".").pop()?.toLowerCase() || "";
  if (["jpg", "jpeg", "png", "gif", "svg"].includes(ext)) return "image";
  if (["js", "ts", "py", "java", "cpp", "html", "css"].includes(ext))
    return "code";
  if (["pdf"].includes(ext)) return "pdf";
  return "text";
};


const FilePreview = ({ file, pdfUrl }: { file: ChromaFile, pdfUrl: string }) => {
  const [imageSrc, setImageSrc] = useState<string | null>(null);

  useEffect(() => {
    const fileType = getFileType(file.metadata.filepath);

    if (fileType === "image") {
      const dataUrl = `data:image/${file.metadata.filepath.split(".").pop()};base64,${file.document}`;
      setImageSrc(dataUrl);
    }
  }, [file]);

  const fileType = getFileType(file.metadata.filepath);

  // Image rendering
  if (fileType === "image" && imageSrc) {
    return <img src={imageSrc} alt={file.metadata.filepath} className="rounded-lg object-fit" />;
  }

  // PDF rendering in iframe
  if (fileType === "pdf") {
    // const pdfUrl = `http://localhost:35438${file.metadata.filepath.replace("Users/ashwa/", "")}`;
    // return (
    //   <iframe
    //     src={pdfUrl}
    //     title={file.metadata.filepath}
    //     className="w-full h-full rounded-lg border border-orange-500"
    //     width={800}
    //     height={600}
    //   ></iframe>
    // );
    return (
      <div>
        {pdfUrl ? (
          <iframe src={pdfUrl} width="100%" height="600px" title="PDF Viewer"></iframe>
        ) : (
          <p>Loading PDF...</p>
        )}
      </div>
    )
  }

  // Text-based file rendering (includes code and plain text)
  return (
    <div className="h-full w-full p-4 rounded-lg overflow-scroll">
      {fileType === "code" ? (
        <pre className="bg-gray-50 dark:bg-[#1f1f1f] rounded-lg">
          <code className="text-sm dark:text-white font-mono">{file.document}</code>
        </pre>
      ) : (
        <div className="bg-white dark:bg-[#1f1f1f] text-black dark:text-white whitespace-pre-wrap">
    
[truncated — 20071 more characters]
```

### src-tauri/src/main.rs

```rust
#![cfg_attr(
    all(not(debug_assertions), target_os = "windows"),
    windows_subsystem = "windows"
)]

use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
use tauri::{CustomMenuItem, Menu, Submenu};
use tokio::signal;
use util::{config_path, db_formatted_path};
use std::fs::File;
use std::io::Read;


mod apis;
mod chroma;
mod invokes;
mod util;

fn start_chroma_server(command: &str) -> Arc<Mutex<std::process::Child>> {
    let chrdb = if cfg!(target_os = "windows") {
        Arc::new(Mutex::new(
            Command::new("cmd")
                .args(&["/C", command]) // Use "/C" for cmd.exe to run a command
                .stdout(Stdio::inherit()) // Inherit standard output for logging
                .stderr(Stdio::inherit()) // Inherit standard error for logging
                .spawn()
                .expect("Failed to start Chroma server on Windows"),
        ))
    } else {
        Arc::new(Mutex::new(
            Command::new("sh")
                .arg("-c")
                .arg(command)
                .stdout(Stdio::inherit()) // Inherit standard output for logging
                .stderr(Stdio::inherit()) // Inherit standard error for logging
                .spawn()
                .expect("Failed to start Chroma server on Unix-like system"),
        ))
    };

    let _ = Arc::clone(&chrdb);
    chrdb
}

fn start_chroma_db() {
    chroma::run_python_sdk(
        &db_formatted_path().as_str(),
        &chroma::Action::GetOrCreate {
            collection_name: "siftfiles".to_string(),
        },
        false,
    )
    .expect("Failed to create Chroma database collection");
}

fn start_chroma_query_agent() -> Arc<Mutex<std::process::Child>> {
    let chrqr = Arc::new(Mutex::new(
        Command::new("uvicorn")
            .arg("src.queryapi:app")
            .arg("--host")
            .arg("127.0.0.1")
            .arg("--port")
            .arg("35443")
            .stdout(Stdio::inherit()) // Inherit standard output for logging
            .stderr(Stdio::inherit()) // Inherit standard error for logging
            .spawn()
            .expect("Failed to start Chroma server on Windows"),
    ));

    let _ = Arc::clone(&chrqr);
    chrqr
}

fn init_local_files() -> Arc<Mutex<std::process::Child>> {
    let proc = Arc::new(Mutex::new(
        Command::new("python3")
            .arg("./pybindings/init_local.py")
            .spawn()
            .expect("Failed to startup local files"),
    ));

    let _ = Arc::clone(&proc);
    proc
}

fn init_gh_files() -> Arc<Mutex<std::process::Child>> {
    let proc = Arc::new(Mutex::new(
        Command::new("python3")
            .arg("./pybindings/init_gh.py")
            .spawn()
            .expect("Failed to startup local files"),
    ));

    let _ = Arc::clone(&proc);
    proc
}

#[tauri::command]
fn run_subprocess(command: String) -> Result<String, String> {
    invokes::run_cmd(command)
}

#[tauri::command]
fn end_app() {
    std::process::exit(0);
}

#[tauri::command]
fn read_pdf_file(file_path: String) -> Result<Vec<u8>, String> {
    let mut file = File::open(&file_path).map_err(|e| e.to_string())?;
    let mut buffer = Vec::new();
    file.read_to_end(&mut buffer).map_err(|e| e.to_string())?;
    Ok(buffer)
}

#[tauri::command]
async fn gh_oauth() -> Result<String, String> {
    invokes::github_oauth().await
}

#[tauri::command]
async fn gh_find(token: &str) -> Result<String, String> {
    invokes::get_repos_and_files(token).await
}

#[tauri::command]
async fn slk_oauth() -> Result<String, String> {
    invokes::slack_oauth().await
}

#[tauri::command]
async fn disc_oauth() -> Result<String, String> {
    invokes::discord_oauth().await
}

#[tauri::command]
async fn ntn_oauth() -> Result<String, String> {
    invokes::notion_oauth().await
}

#[tauri::command]
async fn ggl_oauth() -> Result<String, String> {
    invokes::google_oauth().await
}

fn main() {
    let quit = CustomMenuItem::new("quit".to_string(), "Quit");
    let submenu = Submenu::new("File", Menu::new().add_item(quit));

    let app_cfg = util::load_config().unwrap();
    println!("\nBeginning Sift.AI Startup...\n");

    let command = "chroma run --path /Users/ashwa/Desktop/sift_datastore --port 35436".to_string();
    let _ = start_chroma_server(&command);

    println!("Chroma server is running in the background on http://localhost:35436.\n");
    println!("AppConfig: {:?}", app_cfg);

    start_chroma_db();
    println!("Chroma database is configured.\n");

    println!("Instantiating File Cache\n");

    // let local_handle = thread::spawn(|| {
    //     init_local_files();
    //     println!("Local files are initialized.\n");
    // });

    // let gh_handle = thread::spawn(|| {
    //     init_gh_files();
    //     println!("GitHub files are initialized.\n");
    // });

    // // Join the threads to ensure both complete
    // local_handle.join().expect("Failed to initialize local files");
    // gh_handle.join().expect("Failed to initialize GitHub files");

    println!("Both tasks are completed.");

    let _ = start_chroma_query_agent();
    println!("Running Chroma Query Agent on http://localhost:35443...\n");

    println!("\nCompleted Startup Configurations\n");

    tauri::Builder::default()
        .setup(|app| {
            let app_handle = app.handle();

            // Spawn the background server and handle Ctrl+C signal
            tauri::async_runtime::spawn(async move {
                let server_handle = tokio::spawn(apis::fileserv::serve());

                // Handle Ctrl+C for graceful shutdown
                tokio::spawn(async move {
                    if signal::ctrl_c().await.is_ok() {
                        println!("Ctrl+C detected, shutting down...");
                        app_handle.exit(0);
                    }
                });

                if let Err(e) = server_handle.await {
                    eprintln!("Server error: {}", e);
                }
            });

            Ok
[truncated — 928 more characters]
```

[52 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]