Project Info
This project did not submit a demo video on Devpost.
Inspiration
People often remark that ChatGPT is bad at creativity. So we thought, what better scenario to quantify the gap in human vs. AI creativity than the New Yorker Caption Contest? After all, the contest is notoriously difficult, with some notable players failing to win for years on end. In the process, we ended up creating a surprisingly fun app that lets you compete to write the funniest caption for a given cartoon, and compare how you do with an AI that has put less than a minute of GPU time into the same task.
What it does
Our application leverages the vision capabilities of GPT-4V to generate inventive captions for the New Yorker Cartoon Contest. We design two reasonable baselines: few-shot GPT-V, where it is given 5 examples of winning image-caption pairs from the past; and a complex chain-of-thought prompt, where it is told to generate many candidates and reject ones which are bad. We then randomly pair these captions up and show them to human participants for rating. Users engage in a straightforward selection process, choosing the caption they find most fitting or amusing among pairs. We use these comparisons to generate an ELO score both for each caption within a contest and for each user on the platform. This allows us to calculate the effective win-rate for each caption and user. Additionally, the application features allow users to explore top-rated captions and draw comparisons between the creativity of GPT-4V and human participants, offering a unique insight into the evolving capabilities of artificial intelligence in the creative domain.
How we built it
The site is a single-page-app built using Reflex, with state stored in Reflex’s integrated SQLite database. The production site is hosted on Fly.io. A primary research challenge for us was making an AI system that could reliably produce humorous captions. We experimented with several different approaches; in the end, we found that the most interesting results came from few-shot GPT-4V (given pairs of (image, winning human caption)), as well as a more complex prompted GPT-4V (a multi-step prompt that asks the model to consider what's unusual in the image, create ~10 candidates, and self-roast). Whether these two baselines do better than human creativity is yet to be seen - and that's where you all come in!
Challenges we ran into
Deployment was surprisingly hard. Reflex's documentation in this area is lacking, and we encountered hard-to-debug errors like a deployment that would fail with absolutely no errors reported to the log. State management in Reflex was a challenging but formative learning experience. While the docs were clear and direct about many aspects such as components and callback functions, there were some nuances to State interactions that we primarily discovered through trial and error. However, the process of iteratively testing and refining our application's state management allowed us to create an intuitive UI. These challenges pushed us to think adaptively about how to best utilize Reflex to meet our project goals.
Accomplishments we're proud of
Aesthetic design of the application in Reflex The statistical ELO rating system we created Design of our system and how it effectively merges LLMs with human creativity within the theater of comedy Sanitation of inputs — given the public nature of our leaderboard, we include a moderation api to ensure singular bad actors aren’t able to spam the leaderboard with profanity or other malicious content.
What we learned
Reflex — it’s actually pretty useful for prototyping, but has several quirks and poorly documented aspects. Formatting, CSS styles, Button animations Typography Deployment of the website Formatting, CSS styles, Button animations Typography Deployment of the website Pushing the boundaries of the API Learned more about multi-modal large AI models The state of the art for language model evaluation in subjective domains, including for hard-to-evaluate topics like humor. Some of us have had experience doing evals for language models, but typically it’s for pretty objective behavior, like MCQ performance. This fuzzier space feels more similar to reward modeling, which is really cool to get experience with!
What's next
Add more social features & login to the application Better AI baselines, like training our own multimodal model, using Google models like Gemini Pro Configuring deployment for public use
Welcome to Reflex!
This is the base Reflex template - installed when you run reflex init.
If you want to use a different template, pass the --template flag to reflex init.
For example, if you want a more basic starting point, you can run:
reflex init --template blank
About this Template
This template has the following directory structure:
├── README.md
├── assets
├── rxconfig.py
└── {your_app}
├── __init__.py
├── components
│ ├── __init__.py
│ └── sidebar.py
├── pages
│ ├── __init__.py
│ ├── dashboard.py
│ ├── index.py
│ └── settings.py
├── styles.py
├── templates
│ ├── __init__.py
│ └── template.py
└── {your_app}.py
See the Project Structure docs for more information on general Reflex project structure.
Adding Pages
In this template, the pages in your app are defined in {your_app}/pages/.
Each page is a function that returns a Reflex component.
For example, to edit this page you can modify {your_app}/pages/index.py.
See the pages docs for more information on pages.
In this template, instead of using rx.add_page or the @rx.page decorator,
we use the @template decorator from {your_app}/templates/template.py.
To add a new page:
- Add a new file in
{your_app}/pages/. We recommend using one file per page, but you can also group pages in a single file. - Add a new function with the
@templatedecorator, which takes the same arguments as@rx.page. - Import the page in your
{your_app}/pages/__init__.pyfile and it will automatically be added to the app.
Adding Components
In order to keep your code organized, we recommend putting components that are
used across multiple pages in the {your_app}/components/ directory.
In this template, we have a sidebar component in {your_app}/components/sidebar.py.
Adding State
As your app grows, we recommend using substates to organize your state. You can either define substates in their own files, or if the state is specific to a page, you can define it in the page file itself.
Analysis
View
Metric
- 36
- 31
- 26
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- HTMLIn code
- OpenAIIn code
- PythonIn code
4 of 4 appear in the indexed code.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
83 KB
Source files
25
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
stanleylcao/treehacks2024
153 files · 13.2 MB · @ 0bfbf17
Structure
Interface
4 files · 3%Screens, components and styles rendered to the user.
Application logic
24 files · 16%Domain rules, services and shared utilities.
Data & schema
7 files · 5%Schema definitions, migrations and data access.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python58%
- HTML38%
- Markdown3%
- CSS1%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
yorknew/requirements.txt
pypi · 21- better-profanity
- black
- datasets
- fire
- fqdn
- isoduration
- jsonpointer
- more-itertools
- mypy
- notebook
- openai
- pandas
- pillow
- pip
- python-dotenv
- reflex
- strictly-typed-pandas
- tenacity
- +3 more
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.