# Project export: Inclusiv.ai
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: Inki is a simplified system to navigate the web.
- Devpost: https://devpost.com/software/inclusiv-ai
- GitHub: https://github.com/VolVox99/inklusiv
- Result: winner (Best Use of Monster Generative AI APIs (4x XBox Series S [1st] & 1 million Monster API credits [2nd] & $400 Cash [3rd]))
- Team: 3 GitHub contributor(s) — Ali (25 commits), Shayan Yasir (19 commits), sidereior (12 commits)
## Devpost submission (written by the team)
### Overview
Inclusiv.ai: Empowering Accessibility
### Inspiration
In a world where technology is a cornerstone of daily life, it's crucial that digital access is equitable. However, not everyone experiences technology in the same way. We took this challenge to heart and crafted Inclusiv.ai to revolutionize accessibility, ensuring that individuals with disabilities can navigate the web with ease and confidence. Our main priority was reducing the hassle of different extensions and toggles. Inclusiv.ai has one button and one assistant—Inki.
### What it does
Inclusiv provides a simple way for those with disabilities to navigate the web. We focused on a hassle-free, intuitive approach with only one button. Simply begin a conversation with your assistant, "Inki," by clicking on the microphone button and explain whatever issues have been limiting your experience on the web. With various modes such as colorblind, screen enhancer, screen explainer and summarizer, and an ADHD/Dyslexia mode. Inki is designed to be a dynamic tool, adapting to a variety of needs through its multiple modes: Colorblind Mode: Tailors the web page colors, ensuring that colorblind users can differentiate between colors that are typically hard to distinguish. Screen Enhancer: Amplifies and clarifies website content for those with visual impairments, allowing for easier reading and interaction. Screen Explainer and Summarizer: A mode that not only explains the elements on the screen but also provides concise summaries for quick comprehension, beneficial for users with cognitive disabilities. ADHD/Dyslexia Mode: Alters the web page layout and typography to minimize distractions and optimize for readability, assisting users with attention deficits or dyslexia. Behind these user-facing features, Inclusiv utilizes a combination of large language models and a specific focus on Monster API. This approach has enabled us to create features that are both technically sophisticated and varied, ensuring a broad range of user needs are met.
### How we built it
Inclusiv was meticulously crafted by a synergistic team of two backend developers and one frontend designer, all united by the vision of making web navigation universally accessible. The project was born from a user-centric approach, focusing on the unique challenges faced by individuals with disabilities. Our team's design philosophy hinged on simplicity, leading to the creation of a one-button interface that calls upon Inki, an AI assistant, to activate different accessibility modes including colorblind, screen enhancer, screen explainer, and summarizer, and an ADHD/Dyslexia mode. By leveraging large language models for natural language processing and integrating Monster API's robust algorithms, Inclusiv transcends conventional assistive technologies. This blend of intuitive design and advanced AI capabilities was continuously refined through iterative user testing and feedback, ensuring that the product genuinely resonates with the needs of its users. Inclusiv’s launch is not an end, but a beginning to an ongoing journey of innovation and improvement, with a commitment to evolving and expanding its features to foster an inclusive web experience for all.
### Challenges we ran into
Where to start? Front-end might be harder than the back end! We spent hours fiddling with the optimal UI setup, trying to figure out how best to simplify the experience for a user of any skill level. This is not as simple as it looks, and we spent two hours trying to add a power button. Additionally, we ended up having to pivot our AI model numerous times and switched our approach throughout the project.
### Accomplishments we're proud of
We ended up changing our AI agent multiple times throughout the project to deliver the optimal project. We're really satisfied with our ability to adapt and push ourselves out of our comfort zones throughout the 36 hours. We developed numerous, technically difficult and varied features that we believe will truly aid those with any form of disability.
### What we learned
We learned tons about building a project from scratch, implementing LLMs, and creating an intuitive and appealing front-end experience.
### What's next
We want to launch on the Chrome web store!
## README (from the GitHub repository)
# inklusiv
Disabilities:
adhd
colorblind
dyslexia
Features:
works with different languages
zoom out zoom in
report generation
todo:
general accessibility improvement and standards
answer questions about the site
give explainsite context about users question
## Detected evidence (automated analysis)
Indexed codebase: 28 recognized source files, 615 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (31 of 31)
```
assets/fonts/css/all.css
assets/fonts/css/all.min.css
assets/fonts/css/brands.css
assets/fonts/css/brands.min.css
assets/fonts/css/fontawesome.css
assets/fonts/css/fontawesome.min.css
assets/fonts/css/regular.css
assets/fonts/css/regular.min.css
assets/fonts/css/solid.css
assets/fonts/css/solid.min.css
assets/fonts/css/svg-with-js.css
assets/fonts/css/svg-with-js.min.css
assets/fonts/css/v4-font-face.css
assets/fonts/css/v4-font-face.min.css
assets/fonts/css/v4-shims.css
assets/fonts/css/v4-shims.min.css
assets/fonts/css/v5-font-face.css
assets/fonts/css/v5-font-face.min.css
explainPage-worker.js
manifest.json
monsterapireferences.txt
package.json
popup.html
popup.js
README.md
scripts/adhd.js
scripts/colorblind.js
scripts/dyslexia.js
scripts/magnifyPage.js
scripts/unMagnifyPage.js
styles.css
```
### Dependencies
- package.json: @emotion/react@^11.11.1, @emotion/styled@^11.11.0, @mui/icons-material@^5.14.15, @mui/material@^5.14.12, @reduxjs/toolkit@^1.9.5, @vitejs/plugin-react@^3.1.0, @vitejs/plugin-react-swc@^3.5.0, autoprefixer@^10.4.14, axios@^1.3.5, classnames@^2.3.2, dompurify@^3.0.5, draft-js@^0.11.7, draftjs-to-html@^0.9.1, esbuild@^0.19.12, eslint@^8.38.0, eslint-config-airbnb@^19.0.4, eslint-plugin-import@^2.27.5, eslint-plugin-jsx-a11y@^6.7.1, eslint-plugin-react@^7.32.2, eslint-plugin-react-hooks@^4.6.0, framer-motion@^10.16.5, html-to-draftjs@^1.5.0, html-to-text@^9.0.5, immer@^10.0.2, jquery@^3.6.4, lodash.debounce@^4.0.8, lodash.uniqueid@^4.0.1, mixpanel-browser@^2.48.1, postcss@^8.4.26, prettier@^3.1.0, prettier-plugin-tailwindcss@^0.5.7, react@^18.2.0, react-collapsed@^4.0.4, react-countdown@^2.3.5, react-datepicker@^4.12.0, react-dom@^18.2.0, react-draft-wysiwyg@^1.15.0, react-icons@^4.12.0, react-markdown@^8.0.7, react-redux@^8.0.5, react-router@^6.10.0, react-router-dom@^6.21.1, react-scroll@^1.9.0, react-select@^5.7.3, redux@^4.2.1, swiper@^11.0.5, tailwindcss@^3.3.3, uuid@^9.0.1, vite@^4.2.1, vite-plugin-eslint@^1.8.1
### Recent commits (newest first)
- biug
- bug
- biug
- fix bool temp val
- prompt
- prompt
- push
- f
- push
- f
- f
- Merge pull request #7 from VolVox99/screenshotSnippet
- nothin
- added power button no problem
- Merge pull request #6 from VolVox99/screenshotSnippet
- added button top right of popup.html
- improov
- improvements
- Merge pull request #5 from VolVox99/screenshotSnippet
- Merge branch 'main' of https://github.com/VolVox99/inklusiv into screenshotSnippet
## Key source files (fetched from GitHub, selected and truncated for size)
### package.json
```
{
"name": "starterpack-shayanyasir",
"version": "1.0.0",
"description": "*description*",
"main": "src/index.js",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"author": "",
"license": "ISC",
"devDependencies": {
"@vitejs/plugin-react-swc": "^3.5.0",
"autoprefixer": "^10.4.14",
"eslint": "^8.38.0",
"eslint-config-airbnb": "^19.0.4",
"eslint-plugin-import": "^2.27.5",
"eslint-plugin-jsx-a11y": "^6.7.1",
"eslint-plugin-react": "^7.32.2",
"eslint-plugin-react-hooks": "^4.6.0",
"postcss": "^8.4.26",
"prettier": "^3.1.0",
"prettier-plugin-tailwindcss": "^0.5.7",
"tailwindcss": "^3.3.3",
"vite": "^4.2.1",
"vite-plugin-eslint": "^1.8.1"
},
"dependencies": {
"@emotion/react": "^11.11.1",
"@emotion/styled": "^11.11.0",
"@mui/icons-material": "^5.14.15",
"@mui/material": "^5.14.12",
"@reduxjs/toolkit": "^1.9.5",
"@vitejs/plugin-react": "^3.1.0",
"axios": "^1.3.5",
"classnames": "^2.3.2",
"dompurify": "^3.0.5",
"draft-js": "^0.11.7",
"draftjs-to-html": "^0.9.1",
"esbuild": "^0.19.12",
"framer-motion": "^10.16.5",
"html-to-draftjs": "^1.5.0",
"html-to-text": "^9.0.5",
"immer": "^10.0.2",
"jquery": "^3.6.4",
"lodash.debounce": "^4.0.8",
"lodash.uniqueid": "^4.0.1",
"mixpanel-browser": "^2.48.1",
"react": "^18.2.0",
"react-collapsed": "^4.0.4",
"react-countdown": "^2.3.5",
"react-datepicker": "^4.12.0",
"react-dom": "^18.2.0",
"react-draft-wysiwyg": "^1.15.0",
"react-icons": "^4.12.0",
"react-markdown": "^8.0.7",
"react-redux": "^8.0.5",
"react-router": "^6.10.0",
"react-router-dom": "^6.21.1",
"react-scroll": "^1.9.0",
"react-select": "^5.7.3",
"redux": "^4.2.1",
"swiper": "^11.0.5",
"uuid": "^9.0.1"
}
}
```
### styles.css
```css
body {
width: 200px;
padding: 10px;
}
input[type="checkbox"] {
margin-bottom: 5px;
}
```
### explainPage-worker.js
```javascript
const response = 'This page is a basic page with no special features.';
chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
if (message?.action === 'explainPage') {
const user_input = message?.user_input;
chrome.tabs.captureVisibleTab().then((screenshotDataUrl) => {
fetch('https://api.openai.com/v1/chat/completions', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + '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'
},
body: JSON.stringify({
"model": "gpt-4-vision-preview",
"max_tokens": 2000,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": `You are a web accessibility partner. Based on the screenshot of the website, describe what is on this website for someone who's vision impaired, and how they can navigate it and what they can do with it. Consider the user's request from you: ${user_input}. If the user's input has a specific question based on the site, be sure to first answer that and accomodate the request. In general, be succint with your language, but explain things in depth and comprehensively so they can fully understand the website. Explain the name of the site, the purpose, how to use it, and more. Imagine you are describing it to a blind person. Speak with conviction, even if you are unsure. Do not use words such as "seems" or "appears" or "likely" and so on. Speak with authority and clarity, even if it's not genuine. Aim to keep the entire thing around 3-4 sentences. Speak in second person tone, directly to the user. At the end, be friendly and maybe ask if they have any more questions.`},
{
"type": "image_url",
"image_url": {
"url": screenshotDataUrl,
},
},
],
}
]
})
})
.then(response => response.json())
.then(data => {
var response = data.choices[0].message.content;
sendResponse({
text: response,
})
})
});
} else {
chrome.tabs.query({ active: true, currentWindow: true }, function(tabs) {
var activeTab = tabs[0];
chrome.scripting
.executeScript({
target : {tabId : activeTab.id},
files : [`scripts/${message.action}.js`],
})
.then(() => console.log("injected a function"));
});
}
return true;
});
```
### popup.html
```html
. Current CSS: ${allCSSContent}`
},
]
})
})
.then(response => response.json())
.then(data => {
console.log('gpt loaded')
console.log(data);
var cssCode = data.choices[0].message.content;
var cssCodeCleaned = cssCode.replace('', '').replace('```html', '').replace('```css', '').replace('```', '').trim();
var styleElement = document.createElement('style');
styleElement.textContent = cssCodeCleaned;
document.head.appendChild(styleElement);
})
}
```
### scripts/adhd.js
```javascript
{
minWordLength = 4, // Minimum word length
minTextLength = 20, // Minimum text length
boldRatio = 0.4, // Bold ratio (percentage of letters per word)
doAdhd = true;
const insertTextBefore = (text, node, bold) => {
if (bold) {
let span = document.createElement('span');
span.className = 'bread';
span.appendChild(document.createTextNode(text));
node.parentNode.insertBefore(span, node);
} else {
node.parentNode.insertBefore(document.createTextNode(text), node);
}
}
const processNode = root => {
let walker = document.createTreeWalker(root, NodeFilter.SHOW_TEXT, {
acceptNode: function (node) {
return (
node.parentNode.nodeName !== 'INPUT' &&
node.parentNode.nodeName !== 'NOSCRIPT' &&
node.parentNode.nodeName !== 'SCRIPT' &&
node.parentNode.nodeName !== 'STYLE' &&
node.parentNode.nodeName !== 'TEXTAREA' &&
node.parentNode.nodeName !== 'TITLE' &&
(node.parentNode.nodeName === 'A' ||
node.parentNode.nodeName === 'EM' ||
node.parentNode.nodeName === 'STRONG' ||
node.nodeValue.length >= minTextLength)) ? NodeFilter.FILTER_ACCEPT : NodeFilter.FILTER_REJECT;
}
});
let node;
while (node = walker.nextNode()) {
let text = node.nodeValue;
let wStart = -1, wLen = 0, eng = null;
for (let i = 0; i <= text.length; i++) { // We use <= here because we want to include the last character in the loop
let cEng = i < text.length ? /[\p{Letter}\p{Mark}]/u.test(text[i]) : false;
if (i == text.length || eng !== cEng) {
// State flipped or end of string
if (eng && wLen >= minWordLength) {
let word = text.substring(wStart, wStart + wLen);
let numBold = Math.ceil(word.length * boldRatio);
let bt = word.substring(0, numBold), nt = word.substring(numBold);
insertTextBefore(bt, node, true);
insertTextBefore(nt, node, false);
} else if (wLen > 0) {
let word = text.substring(wStart, wStart + wLen);
insertTextBefore(word, node, false);
}
wStart = i;
wLen = 1;
eng = cEng;
} else {
wLen++;
}
}
node.nodeValue = ''; // Can't remove the node (otherwise the tree walker will break) so just set it to empty
}
};
document.head.appendChild(document.createElement('style')).textContent = `
span.bread {
display: contents !important;
font-weight: bold !important;
}
`;
if (doAdhd){
let breadNode = 'body'
let node = document.querySelector(breadNode);
processNode(node);
}
}
```
[14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]