# Project export: The New Yorker Caption Club This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Seeing how long until AI crushes us at New Yorker captions. Ever wished the New Yorker Captioning Contest was more than once a week? Propose captions for classic cartoons and fight against AI. - Devpost: https://devpost.com/software/the-new-yorker-caption-club - GitHub: https://github.com/stanleylcao/treehacks2024 - Demo: https://yorknew.fly.dev/ - Team: 3 GitHub contributor(s) — Kevin Liu (36 commits), BYTurnips (31 commits), Stanley Cao (26 commits) ## Devpost submission (written by the team) ### Inspiration People often remark that ChatGPT is bad at creativity. So we thought, what better scenario to quantify the gap in human vs. AI creativity than the New Yorker Caption Contest? After all, the contest is notoriously difficult, with some notable players failing to win for years on end. In the process, we ended up creating a surprisingly fun app that lets you compete to write the funniest caption for a given cartoon, and compare how you do with an AI that has put less than a minute of GPU time into the same task. ### What it does Our application leverages the vision capabilities of GPT-4V to generate inventive captions for the New Yorker Cartoon Contest. We design two reasonable baselines: few-shot GPT-V, where it is given 5 examples of winning image-caption pairs from the past; and a complex chain-of-thought prompt, where it is told to generate many candidates and reject ones which are bad. We then randomly pair these captions up and show them to human participants for rating. Users engage in a straightforward selection process, choosing the caption they find most fitting or amusing among pairs. We use these comparisons to generate an ELO score both for each caption within a contest and for each user on the platform. This allows us to calculate the effective win-rate for each caption and user. Additionally, the application features allow users to explore top-rated captions and draw comparisons between the creativity of GPT-4V and human participants, offering a unique insight into the evolving capabilities of artificial intelligence in the creative domain. ### How we built it The site is a single-page-app built using Reflex, with state stored in Reflex’s integrated SQLite database. The production site is hosted on Fly.io. A primary research challenge for us was making an AI system that could reliably produce humorous captions. We experimented with several different approaches; in the end, we found that the most interesting results came from few-shot GPT-4V (given pairs of (image, winning human caption)), as well as a more complex prompted GPT-4V (a multi-step prompt that asks the model to consider what's unusual in the image, create ~10 candidates, and self-roast). Whether these two baselines do better than human creativity is yet to be seen - and that's where you all come in! ### Challenges we ran into Deployment was surprisingly hard. Reflex's documentation in this area is lacking, and we encountered hard-to-debug errors like a deployment that would fail with absolutely no errors reported to the log. State management in Reflex was a challenging but formative learning experience. While the docs were clear and direct about many aspects such as components and callback functions, there were some nuances to State interactions that we primarily discovered through trial and error. However, the process of iteratively testing and refining our application's state management allowed us to create an intuitive UI. These challenges pushed us to think adaptively about how to best utilize Reflex to meet our project goals. ### Accomplishments we're proud of Aesthetic design of the application in Reflex The statistical ELO rating system we created Design of our system and how it effectively merges LLMs with human creativity within the theater of comedy Sanitation of inputs — given the public nature of our leaderboard, we include a moderation api to ensure singular bad actors aren’t able to spam the leaderboard with profanity or other malicious content. ### What we learned Reflex — it’s actually pretty useful for prototyping, but has several quirks and poorly documented aspects. Formatting, CSS styles, Button animations Typography Deployment of the website Formatting, CSS styles, Button animations Typography Deployment of the website Pushing the boundaries of the API Learned more about multi-modal large AI models The state of the art for language model evaluation in subjective domains, including for hard-to-evaluate topics like humor. Some of us have had experience doing evals for language models, but typically it’s for pretty objective behavior, like MCQ performance. This fuzzier space feels more similar to reward modeling, which is really cool to get experience with! ### What's next Add more social features & login to the application Better AI baselines, like training our own multimodal model, using Google models like Gemini Pro Configuring deployment for public use ## README (from the GitHub repository) # Welcome to Reflex! This is the base Reflex template - installed when you run `reflex init`. If you want to use a different template, pass the `--template` flag to `reflex init`. For example, if you want a more basic starting point, you can run: ```bash reflex init --template blank ``` ## About this Template This template has the following directory structure: ```bash ├── README.md ├── assets ├── rxconfig.py └── {your_app} ├── __init__.py ├── components │   ├── __init__.py │   └── sidebar.py ├── pages │   ├── __init__.py │   ├── dashboard.py │   ├── index.py │   └── settings.py ├── styles.py ├── templates │   ├── __init__.py │   └── template.py └── {your_app}.py ``` See the [Project Structure docs](https://reflex.dev/docs/getting-started/project-structure/) for more information on general Reflex project structure. ### Adding Pages In this template, the pages in your app are defined in `{your_app}/pages/`. Each page is a function that returns a Reflex component. For example, to edit this page you can modify `{your_app}/pages/index.py`. See the [pages docs](https://reflex.dev/docs/components/pages/) for more information on pages. In this template, instead of using `rx.add_page` or the `@rx.page` decorator, we use the `@template` decorator from `{your_app}/templates/template.py`. To add a new page: 1. Add a new file in `{your_app}/pages/`. We recommend using one file per page, but you can also group pages in a single file. 2. Add a new function with the `@template` decorator, which takes the same arguments as `@rx.page`. 3. Import the page in your `{your_app}/pages/__init__.py` file and it will automatically be added to the app. ### Adding Components In order to keep your code organized, we recommend putting components that are used across multiple pages in the `{your_app}/components/` directory. In this template, we have a sidebar component in `{your_app}/components/sidebar.py`. ### Adding State As your app grows, we recommend using [substates](https://reflex.dev/docs/state/substates/) to organize your state. You can either define substates in their own files, or if the state is specific to a page, you can define it in the page file itself. ## Detected evidence (automated analysis) Indexed codebase: 25 recognized source files, 83 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (46 of 46) ``` .gitignore comics.csv README.md val_explanation.csv val_final.csv val_ranking.csv val_ranking2.csv yorknew/.dockerignore yorknew/.gitignore yorknew/alembic.ini yorknew/alembic/env.py yorknew/alembic/README yorknew/alembic/script.py.mako yorknew/alembic/versions/133c716ee8ea_.py yorknew/alembic/versions/2ef1afab3888_.py yorknew/alembic/versions/97c69e9fc3d9_seed_data.py yorknew/assets/fonts/newyorkertitle.css yorknew/Caddyfile yorknew/data/combined_df.csv yorknew/data/gptv_cot1_samples_10.jsonl yorknew/data/gptv_cot1_samples_100.jsonl yorknew/data/gptv_cot1_samples_5.jsonl yorknew/data/gptv_samples.jsonl yorknew/Dockerfile yorknew/fly.toml yorknew/index.html yorknew/Procfile yorknew/requirements.txt yorknew/rxconfig.py yorknew/yorknew/__init__.py yorknew/yorknew/aboutpage.py yorknew/yorknew/components/navbar.py yorknew/yorknew/components/scroller.py yorknew/yorknew/components/styles.py yorknew/yorknew/database.py yorknew/yorknew/ELO.py yorknew/yorknew/leaderboardpage.py yorknew/yorknew/models/data_processing.py yorknew/yorknew/models/dataset.py yorknew/yorknew/models/eval_gptv.py yorknew/yorknew/models/image_gen/dalle.py yorknew/yorknew/models/notebooks/combine_dfs.py yorknew/yorknew/models/notebooks/dump_images.py yorknew/yorknew/models/notebooks/inspect_gptv.py yorknew/yorknew/ratingpage.py yorknew/yorknew/yorknew.py ``` ### Dependencies - yorknew/requirements.txt: better-profanity, black@==24.2.0, datasets@==2.17.0, fire@==0.5.0, fqdn@==1.5.1, isoduration@==20.11.0, jsonpointer@==2.4, more-itertools@==10.2.0, mypy@==1.8.0, notebook@==7.1.0, openai@==1.12.0, pandas, pillow@==10.2.0, pip@==24.0, python-dotenv@==1.0.1, reflex@==0.4.0, strictly-typed-pandas@==0.2.1, tenacity@==8.2.3, uri-template@==1.3.0, webcolors@==1.13, wheel@==0.42.0 ### Recent commits (newest first) - update about - Merge remote-tracking branch 'refs/remotes/origin/master' - fixed bug with adding caption - authorship - update desc - removed comment about AI generated images - Add better-profanity to readme - Merge branch 'master' of https://github.com/stanleylcao/treehacks2024 - implemented user caption sanitation - Fix issues in prod - fixed bug - fix - Merge remote-tracking branch 'refs/remotes/origin/master' - final touches - rebrand - randomize on startup - finished merges - finish merge - misc fixes again - merged ## Key source files (fetched from GitHub, selected and truncated for size) ### yorknew/requirements.txt ``` black==24.2.0 datasets==2.17.0 fire==0.5.0 fqdn==1.5.1 isoduration==20.11.0 jsonpointer==2.4 more-itertools==10.2.0 mypy==1.8.0 notebook==7.1.0 openai==1.12.0 pillow==10.2.0 pip==24.0 python-dotenv==1.0.1 reflex==0.4.0 strictly-typed-pandas==0.2.1 tenacity==8.2.3 uri-template==1.3.0 webcolors==1.13 wheel==0.42.0 pandas better-profanity ``` ### yorknew/Dockerfile ``` # Stage 1: init FROM python:3.12 as init # Pass `--build-arg API_URL=http://app.example.com:8000` during build ARG API_URL # Copy local context to `/app` inside container (see .dockerignore) WORKDIR /app COPY requirements.txt . # Create virtualenv which will be copied into final container ENV VIRTUAL_ENV=/app/.venv ENV PATH="$VIRTUAL_ENV/bin:$PATH" RUN python3 -m venv $VIRTUAL_ENV # Install app requirements and reflex inside virtualenv RUN pip install -r requirements.txt RUN mv /app/.venv /tmp/.venv COPY . . RUN rm -rf /app/.venv || true && mv /tmp/.venv /app/.venv # Deploy templates and prepare app RUN reflex init # Export static copy of frontend to /app/.web/_static # RUN reflex export --frontend-only --no-zip # Stage 2: copy artifacts into slim image FROM python:3.12-slim ARG API_URL WORKDIR /app COPY --from=init /app /app ENV PATH="/app/.venv/bin:$PATH" API_URL=$API_URL # install caddy RUN apt-get update && apt-get install -y wget tmux RUN wget -O caddy.tar.gz https://github.com/caddyserver/caddy/releases/download/v2.7.6/caddy_2.7.6_linux_amd64.tar.gz && tar -xvf caddy.tar.gz && chmod +x caddy && mv caddy /usr/local/bin RUN wget -O overmind.gz https://github.com/DarthSim/overmind/releases/download/v2.4.0/overmind-v2.4.0-linux-amd64.gz && gzip -d overmind.gz && chmod +x overmind && mv overmind /usr/local/bin COPY Caddyfile /Caddyfile COPY Procfile /Procfile RUN apt-get install -y unzip curl # 3000 = frontend, 8000=backend CMD overmind s ``` ### yorknew/rxconfig.py ```python import reflex as rx import os config = rx.Config( app_name="yorknew", db_url=os.environ.get("REFLEX_DB", "sqlite:///reflex.db"), ) ``` ### yorknew/yorknew/ELO.py ```python # K-factor K = 24 def adjust_rating(winner_rating: float, loser_rating: float) -> tuple[float, float]: R_A, R_B = winner_rating, loser_rating Q_A = 10 ** (R_A / 400) Q_B = 10 ** (R_B / 400) E_A = Q_A / (Q_A + Q_B) E_B = Q_B / (Q_A + Q_B) new_winner_rating = R_A + K * (1 - E_A) new_loser_rating = R_B + K * (0 - E_B) return new_winner_rating, new_loser_rating ``` ### yorknew/yorknew/aboutpage.py ```python """ This contains the about page for explanation of the app """ from rxconfig import config import reflex as rx import yorknew.components.styles def aboutcontent(): return rx.center( rx.vstack( rx.heading("It's time to train your captioning skills!"), rx.text( "This website is inspired by the New Yorker comic \ captioning contest.It's fun for all ages and trains your \ comedy muscles, but it only happens once a week. We're \ offering a platform that generalizes this competition to \ all kinds of images, so people can competitively caption without a care.", font_family="adobe-caslon", font_size="24px", ), rx.text( "You will be competing against two AI systems, both based on the GPT-4V architecture. So make sure you put in more effort than ChatGPT does ;)", font_family="adobe-caslon", font_size="24px", ), rx.text( "Made by Brion, Stanley, and Kevin. This website is unofficial and not affiliated with the New Yorker publication.", font_family="adobe-caslon", font_size="24px", ), align="center", spacing="7", # font_size="2em", padding="4em", ), height="75vh", ) ``` ### yorknew/alembic/env.py ```python from logging.config import fileConfig from sqlalchemy import engine_from_config from sqlalchemy import pool from alembic import context # this is the Alembic Config object, which provides # access to the values within the .ini file in use. config = context.config # Interpret the config file for Python logging. # This line sets up loggers basically. if config.config_file_name is not None: fileConfig(config.config_file_name) # add your model's MetaData object here # for 'autogenerate' support # from myapp import mymodel # target_metadata = mymodel.Base.metadata target_metadata = None # other values from the config, defined by the needs of env.py, # can be acquired: # my_important_option = config.get_main_option("my_important_option") # ... etc. def run_migrations_offline() -> None: """Run migrations in 'offline' mode. This configures the context with just a URL and not an Engine, though an Engine is acceptable here as well. By skipping the Engine creation we don't even need a DBAPI to be available. Calls to context.execute() here emit the given string to the script output. """ url = config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, literal_binds=True, dialect_opts={"paramstyle": "named"}, ) with context.begin_transaction(): context.run_migrations() def run_migrations_online() -> None: """Run migrations in 'online' mode. In this scenario we need to create an Engine and associate a connection with the context. """ connectable = engine_from_config( config.get_section(config.config_ini_section, {}), prefix="sqlalchemy.", poolclass=pool.NullPool, ) with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata ) with context.begin_transaction(): context.run_migrations() if context.is_offline_mode(): run_migrations_offline() else: run_migrations_online() ``` ### yorknew/yorknew/leaderboardpage.py ```python """ This page contains the leaderboard for displaying scores """ from rxconfig import config import reflex as rx import yorknew.components.styles as styles from yorknew.components.scroller import leaderboardscroller import yorknew.database as db theme = { # "fontFamily": "Merriweather, Merriweather, adobe-caslon" "baseFontStyle": "18px" } def rankingscontent(): return rx.vstack( rx.heading("User Leaderboard", padding_top="1em"), rx.text( rx.text.strong(db.State.n_users), " users, ", rx.text.strong(db.State.n_comparisons), " comparisons", ), rx.data_table( data=db.State.user_elo_table, columns=db.user_elo_columns, sort=True, on_paste=True, draw_focus_ring=False, freeze_columns=2, group_header_height=50, header_height=60, max_column_width=300, min_column_width=100, row_height=50, # row_markers="clickable-number", smooth_scroll_x=True, vertical_border=False, # column_select="multi", overscroll_x=100, ), # Heading rx.heading( f"Leaderboard for Contest #\ {db.State.imgidlist[db.State.contest_number_leaderboard]}", padding_top="1em", ), # Image rx.image( src=f"/contest_images/{db.State.imgidlist [db.State.contest_number_leaderboard]}.jpg", height="300px", ), # Scroller leaderboardscroller(), # replace with ranking_page statevar rx.data_table( data=db.State.leaderboard_table, columns=db.entrycolumns, pagination=True, search=True, sort=True, on_paste=True, draw_focus_ring=False, freeze_columns=2, group_header_height=50, header_height=60, max_column_width=300, min_column_width=100, row_height=50, # row_markers="clickable-number", smooth_scroll_x=True, vertical_border=False, # column_select="multi", overscroll_x=100, ), align="center", spacing="7", font_size="2em", ) ``` ### yorknew/yorknew/yorknew.py ```python """This file serves as the starting point for app deployment and nav""" import logging import os from pathlib import Path import pandas as pd import reflex as rx from yorknew.ratingpage import ratingscontent from yorknew.aboutpage import aboutcontent from yorknew.leaderboardpage import rankingscontent import yorknew.database as db import yorknew.components.navbar as navbar logger = logging.Logger(__name__) REFLEX_ROOT_DIR = Path(__file__).parent.parent app = rx.App( stylesheets=[ "fonts/newyorkertitle.css", ], theme=rx.theme( appearance="inherit", has_background=True, radius="large", # accent_color="gray", ), db_url=os.environ.get("REFLEX_DB", "sqlite:///reflex.db"), style={ "accent_color": "#8C1515", # "font_family": "Libre Caslon Text, serif", rx.button: { "font_family": "Merriweather", "transition": "background-color 0.3s ease, transform 0.3s ease", "background-color": "#2E2D29", "_hover": { "transform": "scale(1.1)", "background-color": "#8C1515", }, }, rx.text: { "font_family": "Merriweather", }, rx.heading: { "font_family": "Merriweather", }, "font_family": "Merriweather", "font_size": "10px", }, ) @rx.page( title="The New Yorker Caption Club", on_load=[db.State.randomize_contest_selection, db.State.load_two_captions_to_rate], ) def index() -> rx.Component: return rx.fragment( navbar.navbar(), rx.container( ratingscontent(), padding_top=navbar.navbar_height, ), ) @rx.page(title="About The New Yorker Caption Club") def about() -> rx.Component: return rx.fragment( navbar.navbar(), rx.container( aboutcontent(), padding_top=navbar.navbar_height, ), ) @rx.page( title="The New Yorker Caption Club Leaderboard", on_load=[db.State.get_leaderboard_table, db.State.get_user_elo_table], ) def rankings() -> rx.Component: return rx.fragment( navbar.navbar(), rx.container( rankingscontent(), padding_top=navbar.navbar_height, ), ) app.add_page(index) app.add_page(about) app.add_page(rankings) ``` ### yorknew/yorknew/ratingpage.py ```python """ This contains the rating page for ease of access to users """ from rxconfig import config import reflex as rx from yorknew.components.scroller import ratingscroller import yorknew.database as db def ratingscontent(): return rx.center( rx.vstack( # Heading rx.heading( f"Contest #{db.State.imgidlist[db.State.contest_number_rating]}", size="7"), rx.text( "Vote for your favorite caption.", size="7", font_family="adobe-caslon" ), # Image rx.image( src=f"/contest_images/{db.State.imgidlist[db.State.contest_number_rating]}.jpg", height="33vh"), # Scroller ratingscroller(), # replace with rating_page statevar # Default two choices rx.cond( db.State.caption_1, rx.hstack( rx.button( db.State.caption_1.caption, width="400px", # Fixed width height="100px", font_size="20px", padding="10px", font_family="adobe-caslon", overflow="hidden", type="submit", on_click=db.State.button_1_click, style={'text-align': 'center'} ), rx.button( db.State.caption_2.caption, width="400px", # Fixed width height="100px", font_size="20px", padding="10px", font_family="adobe-caslon", type="submit", on_click=db.State.button_2_click, style={'text-align': 'center'} ), ), ), rx.form( rx.vstack( # Custom choice entry rx.text("Don't like either? Write your own!", size="3"), rx.hstack( rx.input( name="new_name", placeholder="My leaderboard name...", style={"width": "200px", "height": "50px"}, ), rx.input( name="new_caption", placeholder="My superior caption...", style={"width": "400px", "height": "50px"}, ), rx.button( "Submit my caption", size="4", type="submit", ), ), align="center", ), on_submit=db.State.handle_submit, reset_on_submit=True, ), align="center", spacing="7", font_size="2em", ), margin_top="10em", ) ``` ### yorknew/yorknew/database.py ```python from collections import defaultdict import reflex as rx from yorknew.ELO import adjust_rating from sqlmodel import Session, select from os import listdir from typing import Any, List, Sequence from random import randrange from better_profanity import profanity import random path_to_contest_images = "./assets/contest_images/" entrycolumns = ["Rating", "User", "Caption"] user_elo_columns = ["Rank", "User", "Elo", "Votes"] columns: list[dict[str, str]] = [ { "title": "Rating", "type": "int", }, { "title": "User", "type": "string", }, { "title": "Caption", "type": "string", }, ] def generate_image_list(imgpath): imgidlist = listdir(imgpath) imgidlist = [s.replace(".jpg", "") for s in imgidlist] imgidlist.sort(key=int) return imgidlist # Database for the rankings of each caption class Entry(rx.Model, table=True): subject: int name: str caption: str rating: int class Comparison(rx.Model, table=True): subject: int caption_1: str caption_2: str name_1: str name_2: str one_is_better: bool imgidlist = generate_image_list(path_to_contest_images) # State for updating the current panel being observed class State(rx.State): # imagelist: List[str] = listdir(path_to_contest_images) # imgidlist: List[str] = [s.replace('.jpg', '') for s in imagelist] imgidlist: List[str] = imgidlist contest_number_leaderboard: int = 0 contest_number_rating: int = 0 # For leaderboard display leaderboard_table: list[list] user_elo_table: list[list] n_users: int = 0 n_comparisons: int = 0 # For rating page caption_1: Entry = None caption_2: Entry = None @staticmethod def convert_entry_to_list(entry) -> tuple[int, str, str]: return [round(entry.rating, 0), entry.name, entry.caption] def randomize_contest_selection(self): self.contest_number_leaderboard = random.sample(range(len(self.imgidlist)), 1)[ 0 ] self.contest_number_rating = self.contest_number_leaderboard def get_leaderboard_table(self): with rx.session() as session: entry_list = session.exec( Entry.select.where( Entry.subject == self.imgidlist[self.contest_number_leaderboard] ) ) self.leaderboard_table = list( # sort by ELO rating sorted( map(State.convert_entry_to_list, entry_list), key=lambda x: x[0], reverse=True, ) ) def get_user_elo_table(self): with rx.session() as session: users = session.exec(select(Entry.name).distinct()).all() comparisons: Sequence[Comparison] = list( session.exec( Comparison.select.where(True).order_by(Comparison.id) ).all() ) user_elo_table = {user: 0 for user in users} for comparison in comparisons: winner = ( comparison.name_1 if comparison.one_is_better else comparison.name_2 ) loser = comparison.name_2 if comparison.one_is_better else comparison.name_1 user_elo_table[winner], user_elo_table[loser] = adjust_rating( user_elo_table[winner], user_elo_table[loser] ) self.n_users = len(users) self.n_comparisons = len(comparisons) self.user_elo_table = list( sorted( [ [ name, round(rating, 0), len( [ c for c in comparisons if (c.name_1 == name or c.name_2 == name) ] ), ] for name, rating in user_elo_table.items() ], key=lambda x: x[1], reverse=True, ) ) self.user_elo_table = [ [i + 1, name, rating, n_comparisons] for i, (name, rating, n_comparisons) in enumerate(self.user_elo_table) ] def clear_db(self): with rx.session() as session: entry_list = session.exec(Entry.select.where(True)) for e in entry_list: session.delete(e) session.commit() @staticmethod def strip_quotes(entry: Entry): s = entry.caption if len(s) > 1 and s[0] == s[-1] and s.startswith(("'", '"')): entry.caption = s[1:-1] return entry def load_two_captions_to_rate(self): with rx.session() as session: entry_list = list( session.exec( Entry.select.where( Entry.subject == self.imgidlist[self.contest_number_rating] ) ) ) print(f"contest number rating {self.contest_number_rating}") print(f"entry list {entry_list}") print(self.caption_1) if len(entry_list) >= 2: print(f"entry list!") self.caption_1, self.caption_2 = tuple( map(State.strip_quotes, random.sample(entry_list, 2)) ) else: self.caption_1, self.caption_2 = None, None def update_captions_rating( self, session: Session, caption_1_new_r: int, caption_2_new_r: int ) -> None: self.caption_1.rating = caption_1_new_r session.add(self.caption_1) self.caption_2.rating = caption_2_new_r session.add(self.caption_2) def add_new_caption(self, session, subject, name, caption): print(f"subject = {subject}, {name}, {caption}") session.add( E [truncated — 4631 more characters] ``` [15 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]