# Project export: calfax

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: From calories to macronutrients, calfax is your pocket nutritionist, making healthy choices easier and more informed than ever. Dive deep into your diet with calfax and make every bite count!
- Devpost: https://devpost.com/software/calfax
- GitHub: https://github.com/rasooly-dev/calhacks-2023
- Demo: https://calhacks-2023.onrender.com/
- Video: https://www.youtube.com/embed/vMyJcg7qA_Q?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — rasooly-dev (14 commits), CollinJH (9 commits)

## Devpost submission (written by the team)

### Inspiration

Me and Collin share a passion for dining out, but we're also committed to maintaining healthy, active lifestyles. We realized that there was a gap in the market for a tool that would allow individuals like us to make informed decisions about their food choices seamlessly. This realization was the spark that ignited our journey to create this project.

### What it does

By simply snapping a photo of your meal, the app utilizes advanced AI to identify the food items and instantly provides a detailed nutritional breakdown. Built with a cloud-based backend, 'Calfax' ensures fast, accurate, and seamless access to the nutritional information you need, making informed dining choices easier than ever. How We Built It We employed NextJS for the frontend and fullstack development. For the backend, we utilized Convex, leveraging their backend-as-a-service software solution. To access hosted open-source multimodal models, we integrated with the Replicate API. Furthermore, we collaborated with together.ai's API to finetune the LLaMa model. This ensured that our application provided consistent and accurate outputs regarding nutritional values.

### Challenges we ran into

Every project comes with its set of hurdles, and ours was no exception. Grappling with Convex was a novel experience. We had to familiarize ourselves with various tools like mutations, actions, stores, and more. Understanding the concept of backend-as-a-service was a challenge in itself. Extracting the food contents from an image was another significant hurdle. We experimented with various methods of extracting data from images before landing on one that perfectly suited our needs. Another major challenge was sourcing a reliable dataset for our nutritional facts. Before we stumbled upon the ideal dataset, we explored the possibility of requesting popular food databases.

### Accomplishments we're proud of

Successfully integrating multiple AI models and backend services was a significant milestone for us. We managed to combine the Replicate API and together.ai's API, which was a challenging yet rewarding experience. The fact that 'Calfax' can identify food items from images and provide a basic nutritional breakdown is an accomplishment we're genuinely proud of, given the technical challenges we faced.

### What we learned

Throughout the development of 'Calfax', we gained hands-on experience in finetuning AI models and working with cloud-based backends. We learned the nuances of multimodal models, especially in the context of image data extraction and conversion. The introduction to Convex's backend-as-a-service was a new learning curve, teaching us about modern backend solutions. The process of sourcing and integrating a dataset for nutritional facts also provided insights into the importance of data in such applications.

### What's next

We're keen on refining 'Calfax' based on user feedback and real-world usage. Potential future steps include exploring collaborations with local eateries to integrate specific menu items, adding features to cater to specific dietary needs, and possibly integrating with other health-focused platforms. Our primary goal is to ensure 'Calfax' serves its users effectively and becomes a reliable tool for nutritional insights.

## README (from the GitHub repository)

This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.


## Detected evidence (automated analysis)

Indexed codebase: 24 recognized source files, 31 KB.
- CSS (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (32 of 32)

```
.gitignore
app/components/ConvexClientProvider.tsx
app/components/ImageForm.module.css
app/components/ImageForm.tsx
app/components/ImageUpload.module.css
app/components/ImageUpload.tsx
app/globals.css
app/LandingPage.module.css
app/layout.tsx
app/page.tsx
app/session/[id]/page.tsx
app/session/[id]/Session.module.css
convex/_generated/api.d.ts
convex/_generated/api.js
convex/_generated/dataModel.d.ts
convex/_generated/server.d.ts
convex/_generated/server.js
convex/README.md
convex/replicate.ts
convex/session.ts
convex/sessiondb.ts
convex/store.ts
convex/together.ts
convex/tsconfig.json
next.config.js
nutrition-data/.ipynb_checkpoints/data_exploration-checkpoint.ipynb
nutrition-data/data_exploration.ipynb
nutrition-data/nutrients_csv.csv
nutrition-data/output.jsonl
package.json
README.md
tsconfig.json
```

### Dependencies

- package.json: @types/node@^20, @types/react@^18, @types/react-dom@^18, axios@^1.6.0, convex@^1.5.1, multer@^1.4.5-lts.1, next@14.0.0, react@^18, react-dom@^18, replicate@^0.20.1, typescript@^5

### Recent commits (newest first)

- Merge pull request #1 from rasooly-dev/alt-branch
- Add Loading Pages + Final Adjustments
- Fix Catastrophe
- thingy
- pushing
- Fix Loading on Session Page
- Add Session Page
- styles
- update
- Finally got this working!
- displaying information for user
- Add File Upload
- Merge branch 'main' into alt-branch
- Add Upload Image Logic
- Add File Handler
- Merge remote-tracking branch 'refs/remotes/origin/alt-branch' into alt-branch
- Fix Layout
- things
- upload file function
- file upload functionality

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "calhacks-2023",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "axios": "^1.6.0",
    "convex": "^1.5.1",
    "multer": "^1.4.5-lts.1",
    "next": "14.0.0",
    "react": "^18",
    "react-dom": "^18",
    "replicate": "^0.20.1"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "typescript": "^5"
  }
}

```

### app/layout.tsx

```typescript
import ConvexClientProvider from './components/ConvexClientProvider';
import './globals.css'

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <ConvexClientProvider>{children}</ConvexClientProvider>
      </body>
    </html>
  );
}

```

### app/page.tsx

```typescript
"use client"

import ImageForm from './components/ImageForm'
import styles from './LandingPage.module.css'

import Pizza from '@/assets/pizza.png'

import { useMutation, useAction } from 'convex/react'
import { api } from "@/convex/_generated/api"

import { useRouter } from 'next/navigation'

import { useState } from 'react'

const UploadPage = () => {
  const generateUploadUrl = useMutation(api.store.generateUploadUrl)
  const createSession = useAction(api.session.createSession)
  const getSessionFromDatabase = useMutation(api.sessiondb.getSessionFromDatabase)

  const router = useRouter()

  const [loading, setLoading] = useState<boolean>(false)

  const handleImageUpload = async (file: File) => {
    setLoading(true)
    const sendImageUrl = await generateUploadUrl()
    fetch(sendImageUrl.toString(), {
      method: 'POST',
      headers: { 'Content-Type': file.type},
      body: file
    })
    .then(response => response.json())
    .then(async data => {
      console.log(data)
      const sessionId = await createSession({
        storageId: data.storageId,
      })

      console.log(router.push(`/session/${sessionId}`))
    })
    .catch(error => {
      console.error(error)
    })
    .finally(() => setLoading(false))
  }

  // const llava = useAction(api.replicate.llava)

  // useEffect(() => {
  //   llava({
  //     image: Pizza
  //   }).then(output => Object.values(output).join(''))
  //     .then(console.log)
  // }, [])

  if (loading)
    return (
      <div className={styles.loading}>
        <h1>Processing Image</h1>
        <p>This will only take a few seconds. Thank you for your patience.</p>

        <span className={styles.movable} />
      </div>
    )

  return (
    <div className={styles.page}>
      <h1> Upload Image </h1>
      <ImageForm handleUpload={handleImageUpload}/>
    </div>
  )
}

export default UploadPage
```

### convex/_generated/server.js

```javascript
/* eslint-disable */
/**
 * Generated utilities for implementing server-side Convex query and mutation functions.
 *
 * THIS CODE IS AUTOMATICALLY GENERATED.
 *
 * Generated by convex@1.5.1.
 * To regenerate, run `npx convex dev`.
 * @module
 */

import {
  actionGeneric,
  httpActionGeneric,
  queryGeneric,
  mutationGeneric,
  internalActionGeneric,
  internalMutationGeneric,
  internalQueryGeneric,
} from "convex/server";

/**
 * Define a query in this Convex app's public API.
 *
 * This function will be allowed to read your Convex database and will be accessible from the client.
 *
 * @param func - The query function. It receives a {@link QueryCtx} as its first argument.
 * @returns The wrapped query. Include this as an `export` to name it and make it accessible.
 */
export const query = queryGeneric;

/**
 * Define a query that is only accessible from other Convex functions (but not from the client).
 *
 * This function will be allowed to read from your Convex database. It will not be accessible from the client.
 *
 * @param func - The query function. It receives a {@link QueryCtx} as its first argument.
 * @returns The wrapped query. Include this as an `export` to name it and make it accessible.
 */
export const internalQuery = internalQueryGeneric;

/**
 * Define a mutation in this Convex app's public API.
 *
 * This function will be allowed to modify your Convex database and will be accessible from the client.
 *
 * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument.
 * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible.
 */
export const mutation = mutationGeneric;

/**
 * Define a mutation that is only accessible from other Convex functions (but not from the client).
 *
 * This function will be allowed to modify your Convex database. It will not be accessible from the client.
 *
 * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument.
 * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible.
 */
export const internalMutation = internalMutationGeneric;

/**
 * Define an action in this Convex app's public API.
 *
 * An action is a function which can execute any JavaScript code, including non-deterministic
 * code and code with side-effects, like calling third-party services.
 * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive.
 * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}.
 *
 * @param func - The action. It receives an {@link ActionCtx} as its first argument.
 * @returns The wrapped action. Include this as an `export` to name it and make it accessible.
 */
export const action = actionGeneric;

/**
 * Define an action that is only accessible from other Convex functions (but not from the client).
 *
 * @param func - The function. It receives an {@link ActionCtx} as its first argument.
 * @returns The wrapped function. Include this as an `export` to name it and make it accessible.
 */
export const internalAction = internalActionGeneric;

/**
 * Define a Convex HTTP action.
 *
 * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object
 * as its second.
 * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`.
 */
export const httpAction = httpActionGeneric;

```

### app/session/[id]/page.tsx

```typescript
"use client"

import { api } from "@/convex/_generated/api"
import { useQuery, useAction, useMutation } from "convex/react"
import { useEffect, useState } from "react"

import styles from './Session.module.css'
import { ObjectType } from "convex/values"

type Session = {
    foodItem: string,
    storageId: string,
    nutritionalValues: {
        carbs: number,
        fat: number,
        fiber: number,
        protein: number,
        saturatedFat: number,
        total_calories: number,
    },
    category: string,
} | null


const PrettyLabels: any = {
    carbs: "Carbohydrates",
    fat: "Fat",
    fiber: "Fiber",
    protein: "Protein",
    saturatedFat: "Saturated Fat",
    total_calories: "Total Calories"
}

const SessionPage = ({ params } : { params: { id: string } }) => {

    const [session, setSession] = useState<Session>(null)

    const getSession = useMutation(api.sessiondb.getSessionFromDatabase)

    useEffect(() => {
        getSession({ sessionId: params.id })
            .then((session: Session) => setSession(session))
    }, [])


    const imageURL = useQuery(api.store.getImage, { storageId: session?.storageId as string })

    
    if (!session && !imageURL)
        return (
            <div className={styles.page}>
                <h1> Loading... </h1>
            </div>
        )

    return (
        <div className={styles.page}>
            <section>
                <h1>Analysis of {session?.foodItem}</h1>
                <img src={imageURL as string} alt="food" />
                <h2> Category: {session?.category}</h2>
            </section>

            <section>
                <table>
                    <tr>
                        <th> Nutrient </th>
                        <th> Value </th>
                    </tr>
                    { Object.entries(session?.nutritionalValues as object).map(([key, value]) => (
                        <tr>
                            <td> {PrettyLabels[key]}</td>
                            <td> {value}{(key != "total_calories") ? 'g' : 'cals'}</td>
                        </tr>
                    ))}
                </table>
            </section>
        </div>
    )
}

export default SessionPage
```

### next.config.js

```javascript
/** @type {import('next').NextConfig} */
const nextConfig = {}

module.exports = nextConfig

```

### app/globals.css

```css
* {
    box-sizing: border-box;
    margin: 0;
    padding: 0;
}
  
```

### convex/store.ts

```typescript
import { mutation, query } from "./_generated/server"
import { v } from "convex/values"

export const generateUploadUrl = mutation(async (ctx) => {
  return await ctx.storage.generateUploadUrl()
})

export const getImage = query({
    args: { storageId: v.optional(v.string()) },
    handler: async (ctx, args) => {
        if (!args.storageId) return null
        const url = await ctx.storage.getUrl(args.storageId)
        return url
    },
  })
```

### convex/sessiondb.ts

```typescript
import { mutation } from "./_generated/server"
import { v } from "convex/values"

export const saveSessionToDatabase = mutation({
    args: { sessionData: v.any() },
    handler: async (ctx, args) => {
        const session = await ctx.db.insert("sessions", args.sessionData)
        return session
    }
})

export const getSessionFromDatabase = mutation({
    args: { sessionId: v.any() },
    handler: async (ctx, args) => {
        const session = await ctx.db.get(args.sessionId)
        return session
    }
})
```

### convex/session.ts

```typescript
"use node"

import { mutation, action } from "./_generated/server"
import { v } from "convex/values"
import { api, internal } from './_generated/api'
import { Id } from "./_generated/dataModel";

export const createSession = action({
  args: { storageId: v.string()},
  handler: async (ctx, args) => {

    const image = await ctx.storage.getUrl(args.storageId)
    const llavaOutput = await ctx.runAction(internal.replicate.llava, { imageURL: image as string })
    const llamaOutput: any = await ctx.runAction(internal.together.llama_tuned, { foodData: llavaOutput as string })

    const sessionData = {
        storageId: args.storageId,
        foodItem: llamaOutput.foodItem,
        category: llamaOutput.category,
        nutritionalValues: llamaOutput.nutritionalValues
    }

    const session: Id = await ctx.runMutation(api.sessiondb.saveSessionToDatabase, { sessionData: sessionData })
    return session
  },
})
```

[13 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]