# Project export: Holistic Health This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Personalized meal recommendations to improve nutrition - Devpost: https://devpost.com/software/holistic-health-xkg69f - GitHub: https://github.com/adityatadimeti/holistichealth - Demo: https://drive.google.com/file/d/1tBL5zwtzuFIhIZv3tHa3BgUQoXOR2Ppo/view?usp=sharing - Team: 2 GitHub contributor(s) — Aditya (7 commits), sathviknallamalli (3 commits) ## Devpost submission (written by the team) ### Inspiration Food powers our progress as a society, yet the indomitable challenge of choosing what to eat has always been an issue. Over the past year, generative AI recipe assistants have emerged that can provide recipes given available ingredients or provide meal recommendations from images of food. However, our teams experience in using these tools revealed a common shortcoming - that these tools often produced generic recommendations, weren't great at incorporating food preferences (which are highly local/variable by nature), and couldn't factor in personal health data in deciding upon meals. To this end, we set out to take advantage of recent advances in interoperability and connectivity of disparate health data sources as well as LLM's/AI agents to synthesize various forms of personal health, diet, and wellness data and provide highly personalized food recommendations. Specifically, our digital health project aims to tackle two main food challenges: 1) a lack of healthy and conscious eating around the world and 2) address health deficiencies through better food options. These challenges are especially pertinent amongst older individuals who carry the highest disease burden but oftentimes don't have the capacity or information available to make informed decisions about the food they are consuming. ### What it does We built Holistic Health, a tool that provides personalized meal recommendations through combining state-of-the-art LLM's with highly personalized data in a human-centered framework to address this challenge. Holistic Health is currently deployed as a web application and allows users to sign up, add background information about themselves and their health and immediately start connecting wearables or other biometrics that provide insight into personalized health measures. From there, Holistic Health provides visualizations showcasing nutrient intake and allows users to generate daily sets of meals that satisfy their personal health requirements. ### How we built it Our web application is built using ReactJS. We used TogetherAI's model hub to finetune a LLAMA-70B and Mixtral-8x7B model (the LLAMA-70B is what is used for inference) for recipe generation using personalized health metrics coming from TerraAPI as well as semantically parse information passed in by the users around their current diet. We also integrate with FatSecret to gather detailed caloric and health information, with some initial work being done to verify self consistency of these results with those from a RAG model trained based on TogetherAI's hub. ### Challenges we ran into One of the biggest challenges we ran into was parsing the data across a variety of wearable integrations to select for the most important pieces of information. Using a combination of scientific literature searches and statistical testing, we ultimately decided upon a couple of monitored metrics that we would use for finetuning. Another issue we faced was identifying the best way to quantify how much a user ate of a specific meal, since being extremely precise (i.e. measuring meals in grams) can decrease the quality of experience for users, but not being precise enough might lead to wildly incorrect estimations of the relative nutrients in specific foods. We ended up settling on a middle of the road solution that allows users to specify quantities of food in terms of units that were defined by a ranking model coming from API calls to FatSecret. ### Accomplishments we're proud of Our team has wanted to work on a personalized medicine and health problem for a while now, so we are very excited that TreeHacks allowed us to pursue that this year. We are extremely proud of our existing feature set and the lack of significant hallucination in model responses due to the fine tuning. We also were excited by our capabilities to integrate almost every wearable out there through the TerraAPI. ### What we learned We learned a lot about using new APIs and a LOT about prompt engineering and finetuning of models. We also learned a lot on preventing model hallucination while preserving diversity of model outputs. We also spent a considerable amount of time understanding how various metabolic factors can be affected by one another. ### What's next We have a couple of ideas that we want to pursue next. Firstly, we want to polish the UI to more concisely show recipe information as well as add more relevant pieces of health information into our visualization suite. From there, we have a couple of technical ideas to further improve recipe generation, from improving quality through embedding models and/or similarity searches to additional finetuning with a more diverse set of templates. In terms of features, we hope to add in fitness recommendations as well (i.e. personalized exercise plans) to ensure that we are addressing recommendations for both the food and fitness problem. Ultimately, we also hope to incorporate more real health data from our APIs to derive more rigorous insights around health! ## README (from the GitHub repository) **Inspiration** Food powers our progress as a society, yet the indomitable challenge of choosing what to eat has always been an issue. Over the past year, generative AI recipe assistants have emerged that can provide recipes given available ingredients or provide meal recommendations from images of food. However, our teams experience in using these tools revealed a common shortcoming - that these tools often produced generic recommendations, weren't great at incorporating food preferences (which are highly local/variable by nature), and couldn't factor in personal health data in deciding upon meals. To this end, we set out to take advantage of recent advances in interoperability and connectivity of disparate health data sources as well as LLM's/AI agents to synthesize various forms of personal health, diet, and wellness data and provide highly personalized food recommendations. Specifically, our digital health project aims to tackle two main food challenges: 1) a lack of healthy and conscious eating around the world and 2) address health deficiencies through better food options. These challenges are especially pertinent amongst older individuals who carry the highest disease burden but oftentimes don't have the capacity or information available to make informed decisions about the food they are consuming. **What it does** We built Holistic Health, a tool that provides personalized meal recommendations through combining state-of-the-art LLM's with highly personalized data in a human-centered framework to address this challenge. Holistic Health is currently deployed as a web application and allows users to sign up, add background information about themselves and their health and immediately start connecting wearables or other biometrics that provide insight into personalized health measures. From there, Holistic Health provides visualizations showcasing nutrient intake and allows users to generate daily sets of meals that satisfy their personal health requirements. **How we built it** Our web application is built using ReactJS. We used TogetherAI's model hub to finetune a LLAMA-70B and Mixtral-8x7B model (the LLAMA-70B is what is used for inference) for recipe generation using personalized health metrics coming from TerraAPI as well as semantically parse information passed in by the users around their current diet. We also integrate with FatSecret to gather detailed caloric and health information, with some initial work being done to verify self consistency of these results with those from a RAG model trained based on TogetherAI's hub. **Challenges we ran into** One of the biggest challenges we ran into was parsing the data across a variety of wearable integrations to select for the most important pieces of information. Using a combination of scientific literature searches and statistical testing, we ultimately decided upon a couple of monitored metrics that we would use for finetuning. Another issue we faced was identifying the best way to quantify how much a user ate of a specific meal, since being extremely precise (i.e. measuring meals in grams) can decrease the quality of experience for users, but not being precise enough might lead to wildly incorrect estimations of the relative nutrients in specific foods. We ended up settling on a middle of the road solution that allows users to specify quantities of food in terms of units that were defined by a ranking model coming from API calls to FatSecret. **Accomplishments that we're proud of** We are extremely proud of our existing feature set and the lack of significant hallucination in model responses due to the fine tuning. We also were excited by our capabilities to integrate almost every wearable out there through the TerraAPI. **What we learned** We learned a lot about using new APIs and a LOT about prompt engineering and finetuning of models. We also learned a lot on preventing model hallucination while preserving diversity of model outputs. We also spent a considerable amount of time understanding how various metabolic factors can be affected by one another. **What's next for Holistic Health** We have a couple of ideas that we want to pursue next. Firstly, we want to polish the UI to more concisely show recipe information as well as add more relevant pieces of health information into our visualization suite. From there, we have a couple of technical ideas to further improve recipe generation, from improving quality through embedding models and/or similarity searches to additional finetuning with a more diverse set of templates. In terms of features, we hope to add in fitness recommendations as well (i.e. personalized exercise plans) to ensure that we are addressing recommendations for both the food and fitness problem. Ultimately, we also hope to incorporate more real health data from our APIs to derive more rigorous insights around health! ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 60 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (57 of 57) ``` .DS_Store fatsecretapi.py holisticreact/.env holisticreact/.gitignore holisticreact/api_data/aktiia.json holisticreact/api_data/apple_activity.json holisticreact/api_data/apple_body.json holisticreact/api_data/apple_daily.json holisticreact/api_data/apple_nutrition.json holisticreact/api_data/apple_sleep.json holisticreact/api_data/eightsleep_daily.json holisticreact/api_data/eightsleep_sleep.json holisticreact/api_data/fitbit_activity.json holisticreact/api_data/fitbit_body.json holisticreact/api_data/fitbit_daily.json holisticreact/api_data/fitbit_nutrition.json holisticreact/api_data/fitbit_sleep.json holisticreact/api_data/googlefit_activity.json holisticreact/api_data/googlefit_body.json holisticreact/api_data/googlefit_daily.json holisticreact/api_data/googlefit_nutrition.json holisticreact/api_data/googlefit_sleep.json holisticreact/api_data/myfitnesspal_activity.json holisticreact/api_data/myfitnesspal_nutrition.json holisticreact/api_data/oura_activity.json holisticreact/api_data/oura_body.json holisticreact/api_data/oura_daily.json holisticreact/api_data/oura_sleep.json holisticreact/app/components/ConnectUser.jsx holisticreact/app/components/Login.jsx holisticreact/app/components/PieChart.js holisticreact/app/components/RestAPI.jsx holisticreact/app/components/test.py holisticreact/app/ConvexClientProvider.jsx holisticreact/app/getData/route.js holisticreact/app/globals.css holisticreact/app/home/page.js holisticreact/app/layout.js holisticreact/app/page.js holisticreact/convex/_generated/api.d.ts holisticreact/convex/_generated/api.js holisticreact/convex/_generated/dataModel.d.ts holisticreact/convex/_generated/server.d.ts holisticreact/convex/_generated/server.js holisticreact/convex/auth.config.js holisticreact/convex/functions.ts holisticreact/convex/README.md holisticreact/convex/tsconfig.json holisticreact/fatsecretapi.py holisticreact/jsconfig.json holisticreact/next.config.mjs holisticreact/package.json holisticreact/postcss.config.js holisticreact/README.md holisticreact/tailwind.config.js README.md terraapi.py ``` ### Dependencies - holisticreact/package.json: @auth0/auth0-react@^2.2.4, @babel/runtime@^7.23.9, @uiw/react-json-view@^2.0.0-alpha.12, autoprefixer@^10.4.17, chart.js@^4.4.1, convex@^1.9.0, next@14.1.0, openai@^4.28.0, postcss@^8.4.35, react@^18, react-chartjs-2@^5.2.0, react-dom@^18, request@^2.88.2, tailwindcss@^3.4.1, together-ai@^0.5.2 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Create README.md - changes - added package - Merge branch 'main' of https://github.com/adityatadimeti/holistichealth - fixed fatsheet - more data for sanjay - data for sanjay - edits - basic website works with connecting to garmin and oura - started react app, running into auth issues, maybe needs https? - refactoring fatsecret code, functional with api call with fatsecret library - initial fatsecretapi attempt ## Key source files (fetched from GitHub, selected and truncated for size) ### holisticreact/package.json ``` { "name": "holisticreact", "version": "0.1.0", "browser": { "child_process": false }, "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@auth0/auth0-react": "^2.2.4", "@babel/runtime": "^7.23.9", "@uiw/react-json-view": "^2.0.0-alpha.12", "chart.js": "^4.4.1", "convex": "^1.9.0", "next": "14.1.0", "openai": "^4.28.0", "react": "^18", "react-chartjs-2": "^5.2.0", "react-dom": "^18", "request": "^2.88.2", "together-ai": "^0.5.2" }, "devDependencies": { "autoprefixer": "^10.4.17", "postcss": "^8.4.35", "tailwindcss": "^3.4.1" }, "proxy": "http://127.0.0.1:5000" } ``` ### holisticreact/app/layout.js ```javascript import { Inter } from "next/font/google"; import "./globals.css"; import ConvexClientProvider from "./ConvexClientProvider"; const inter = Inter({ subsets: ["latin"] }); export const metadata = { title: "Holistic Health", description: "Use this as a skeleton for your project!", }; export default function RootLayout({ children }) { return ( {children} ); } ``` ### holisticreact/app/page.js ```javascript import Image from "next/image"; import { useSearchParams } from "next/navigation"; import ConnectUser from "./components/ConnectUser"; import { Roboto_Flex } from "next/font/google"; import Login from "./components/Login"; const roboto = Roboto_Flex({ subsets: ["latin"] }); export default function Home() { return (
); } ``` ### holisticreact/app/home/page.js ```javascript import ConnectUser from "../components/ConnectUser"; import RestAPI from "../components/RestAPI"; export default function Home() { return (
); } ``` ### holisticreact/app/getData/route.js ```javascript import { NextRequest, NextResponse } from "next/server" export async function POST(req) { let res = await fetchUserData( req.headers.get("user_id"), req.headers.get("start_date"), req.headers.get("end_date"), req.headers.get("scope")) return NextResponse.json({"data": res}) } // This is how to fetch data from our Rest API async function fetchUserData(userID, start_date, end_date, scope) { let request = await fetch(`https://api.tryterra.co/v2/${scope}?user_id=${userID}&start_date=${start_date}&end_date=${end_date}&to_webhook=false`, { method: "GET", headers: { "auth_success_redirect_url": "http://localhost:3000", 'dev-id': process.env.NEXT_PUBLIC_DEV_ID, 'x-api-key': process.env.NEXT_PUBLIC_API_KEY } }) let response = await request.text() return response } ``` ### holisticreact/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.9.0. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### fatsecretapi.py ```python from fatsecret import Fatsecret import os oauth_consumer_key = os.getenv('FATSECRET_CONSUMER_KEY') consumer_secret = os.getenv('FATSECRET_CONSUMER_SECRET') fs = Fatsecret(oauth_consumer_key, consumer_secret) foods = fs.foods_search("chicken breast") fat = foods[0]["food_description"].split("Fat:")[1].split("|")[0].strip() carbs = foods[0]["food_description"].split("Carbs:")[1].split("|")[0].strip() protein = foods[0]["food_description"].split("Protein:")[1].split("|")[0].strip() print(fat, carbs, protein) ``` ### terraapi.py ```python import logging import flask from flask import request from terra.base_client import Terra import os logging.basicConfig(level=logging.INFO) _LOGGER = logging.getLogger("app") API_KEY = os.getenv('TERRA_API_KEY') DEV_ID = os.getenv('TERRA_DEV_ID') SIGNING_SECRET = os.getenv('TERRA_SIGNING_SECRET') terra = Terra(api_key= API_KEY, dev_id= DEV_ID, secret=SIGNING_SECRET) parsed_api_response = terra.list_providers().get_parsed_response() print(parsed_api_response) parsed_api_response = terra.list_users().get_parsed_response() print(parsed_api_response) auth_resp = terra.generate_authentication_url( reference_id="USER ID IN YOUR APP", resource="FITBIT", auth_success_redirect_url="https://success.url", auth_failure_redirect_url="https://failure.url", ).get_parsed_response() print(auth_resp) # app = flask.Flask(__name__) # @app.route("/consumeTerraWebhook", methods=["POST"]) # def consume_terra_webhook() -> flask.Response: # # body_str = str(request.get_data(), 'utf-8') # body = request.get_json() # _LOGGER.info( # "Received webhook for user %s of type %s", # body.get("user", {}).get("user_id"), # body["type"]) # verified = terra.check_terra_signature(request.get_data().decode("utf-8"), request.headers['terra-signature']) # if verified: # return flask.Response(status=200) # else: # return flask.Response(status=403) # auth_resp = terra.generate_authentication_url( # reference_id="USER ID IN YOUR APP", # resource="GARMIN", # auth_success_redirect_url="https://success.url", # auth_failure_redirect_url="https://failure.url", # ).get_parsed_response() # print(auth_resp) # if __name__ == "__main__": # app.run(host="localhost", port=8080) ``` ### holisticreact/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### holisticreact/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { content: [ "./pages/**/*.{js,ts,jsx,tsx,mdx}", "./components/**/*.{js,ts,jsx,tsx,mdx}", "./app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { backgroundImage: { "gradient-radial": "radial-gradient(var(--tw-gradient-stops))", "gradient-conic": "conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))", }, }, }, plugins: [], }; ``` [14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]