# Project export: Data Poisoning Website This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Artists want to protect their art from web scrapers, and they can upload their art to our website to do so. - Devpost: https://devpost.com/software/data-poisoning-website - GitHub: https://github.com/floatingtrees/app - Team: 1 GitHub contributor(s) — floatingtrees (17 commits) ## Devpost submission (written by the team) No Devpost description available. ## README (from the GitHub repository) # Treehacks project (all building done in 1.5 days) Run npm run dev to start the client side and python3 server.py to run the server. Contributors: Jonathan Zhou ## Detected evidence (automated analysis) Indexed codebase: 23 recognized source files, 37 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code - PyTorch (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (26 of 26) ``` .gitignore .nvmrc black_box_tests.py black_box.py components/date.js components/imageUpload.js components/layout.js components/layout.module.css components/text_box.js components/upload_section.js FGSM.py generate.py lib/posts.js next.config.js package.json pages/_app.js pages/api/hello.js pages/index.js pages/likeButton.js pages/posts/uploads.js posts/pre-rendering.md posts/ssg-ssr.md README.md server.py styles/global.css styles/utils.module.css ``` ### Dependencies - package.json: date-fns@^2.29.3, formidable-serverless@^1.1.1, gray-matter@^4.0.3, next@latest, react@18.2.0, react-dom@18.2.0, remark@^14.0.2, remark-html@^15.0.1 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Merge branch 'main' of https://github.com/floatingtrees/app - tested hop-skip-jump - testing - working on grad approx - Update README.md - Update README.md - Update README.md - Update README.md - started hop-skip-hump attack - Update README.md - final iter - working iter 1 - fixed the image rendering issue - made adversarial generation work - added images - finally worked - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### posts/pre-rendering.md ```markdown --- title: 'Two Forms of Pre-rendering' date: '2022-01-01' --- Next.js has two forms of pre-rendering: **Static Generation** and **Server-side Rendering**. The difference is in **when** it generates the HTML for a page. - **Static Generation** is the pre-rendering method that generates the HTML at **build time**. The pre-rendered HTML is then _reused_ on each request. - **Server-side Rendering** is the pre-rendering method that generates the HTML on **each request**. Importantly, Next.js lets you **choose** which pre-rendering form to use for each page. You can create a "hybrid" Next.js app by using Static Generation for most pages and using Server-side Rendering for others. ``` ### posts/ssg-ssr.md ```markdown --- title: 'When to Use Static Generation v.s. Server-side Rendering' date: '2022-01-02' --- We recommend using **Static Generation** (with and without data) whenever possible because your page can be built once and served by CDN, which makes it much faster than having a server render the page on every request. You can use Static Generation for many types of pages, including: - Marketing pages - Blog posts - E-commerce product listings - Help and documentation You should ask yourself: "Can I pre-render this page **ahead** of a user's request?" If the answer is yes, then you should choose Static Generation. On the other hand, Static Generation is **not** a good idea if you cannot pre-render a page ahead of a user's request. Maybe your page shows frequently updated data, and the page content changes on every request. In that case, you can use **Server-Side Rendering**. It will be slower, but the pre-rendered page will always be up-to-date. Or you can skip pre-rendering and use client-side JavaScript to populate data. ``` ### package.json ``` { "private": true, "scripts": { "build": "next build", "dev": "next dev", "start": "next start" }, "dependencies": { "date-fns": "^2.29.3", "formidable-serverless": "^1.1.1", "gray-matter": "^4.0.3", "next": "latest", "react": "18.2.0", "react-dom": "18.2.0", "remark": "^14.0.2", "remark-html": "^15.0.1" }, "engines": { "node": ">=18" } } ``` ### server.py ```python from flask import Flask, jsonify, request from flask_cors import CORS import os from werkzeug.utils import secure_filename from PIL import Image import FGSM, generate import base64 import torch global primary global desired_class_2 model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True) model.eval() app = Flask(__name__) CORS(app) @app.route('/data', methods=['GET']) def get_data(): # Return some data data = { "message": "Hello from Flask!" } return jsonify(data) UPLOAD_FOLDER = 'uploads' ALLOWED_EXTENSIONS = {'png', 'jpg', 'jpeg', 'gif'} app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER def allowed_file(filename): return '.' in filename and \ filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS @app.route('/upload', methods=['POST']) def upload_file(): # Check if the post request has the file part if 'file' not in request.files: return jsonify({'message': 'No file part'}), 400 file = request.files['file'] # If user does not select file, browser also # submit an empty part without filename if file.filename == '': return jsonify({'message': 'No selected file'}), 400 if file and allowed_file(file.filename): filename = secure_filename(file.filename) save_path = os.path.join(app.config['UPLOAD_FOLDER'], filename) file.save(save_path) try: wanted_class = desired_class_2 except: wanted_class = None old_category, new_category = generate.generate(model, save_path, label = None, string_label = wanted_class) image_path = 'uploads/image.png' # Ensure the image path is correct and accessible with open(image_path, "rb") as image_file: encoded_string = base64.b64encode(image_file.read()).decode('utf-8') primary = {'message': 'File uploaded successfully', 'image': encoded_string, "old_category" : old_category, "new_category" : new_category} return jsonify(primary), 200 return jsonify({'message': 'File type not allowed'}), 400 @app.route('/send-classes', methods=['POST']) def recieve_class(): desired_class = request.data.decode('utf-8') print(desired_class) global desired_class_2 desired_class_2 = desired_class return jsonify("Placeholder"), 200 if __name__ == '__main__': app.run(debug=True, port=5001, host="0.0.0.0") ``` ### pages/index.js ```javascript import { useState } from 'react'; import ImageUploader from '../components/imageUpload' import React from 'react'; export default function HomePage() { const [message, setMessage] = useState(''); const handleClick = async (e) => { const response = await fetch('/api/retrieve-image', { method: 'POST', body: formData, }); }; return (

Protect your Artwork from Web Scrapers

); } ``` ### next.config.js ```javascript module.exports = { async rewrites() { return [ { source: '/api/:path*', destination: 'http://localhost:5001/:path*' }] } } ``` ### black_box_tests.py ```python import torch import math import black_box from PIL import Image from torchvision import transforms class model_query_fn: def __init__(self, model): self.model = model self.model.eval() def __call__(self, img): preds = self.model(img) value = torch.argmax(preds, dim = 1) return value model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True) model.eval() image = Image.open("rabbit.png") preprocess = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]) image = torch.unsqueeze(preprocess(image), 0) adv_image = Image.open("samoyed.jpg") adv_image = torch.unsqueeze(preprocess(adv_image), 0) attack = black_box.HopSkipJump(model_query_fn(model), clip_min = -2.5, clip_max = 2.5, targeted = True) image = attack.generate(image, max_iter = 10000, num_evals = 64, target = 331, adversarial_sample = adv_image) ``` ### black_box.py ```python import torch import math class HopSkipJump: def __init__(self, model_query_fn, clip_min = None, clip_max = None, input_shape = torch.tensor([3, 224, 224]), targeted = False): self.model_query_fn = model_query_fn self.total_classes = 1000 self.input_shape = input_shape self.targeted = targeted self.theta = 0.01 / (torch.sqrt(torch.prod(input_shape))) self.clip_min = clip_min self.clip_max = clip_max def _clip(self, tensor): return torch.clip(tensor, self.clip_min, self.clip_max) def generate(self, initial_input, adversarial_sample, max_iter, num_evals = 32, target = None): self.target = target first_iter = True current_image = initial_input for i in range(max_iter): delta = self._compute_delta(current_image, adversarial_sample, first_iter) current_image = self._clip(self._binary_search(current_image, initial_input, delta, target)) dist = torch.linalg.norm(current_image - initial_input) grads = self._approximate_gradient(current_image, initial_input, num_evals, target, delta) epsilon = self._geometric_progression(current_image, grads, dist, num_evals, delta, i, target) current_image = self._clip(current_image + epsilon * grads) current_image = self._binary_search(current_image, initial_input, delta, target) first_iter = False return current_image def _approximate_gradient(self, current_image, initial_input, num_evals, target, delta): shape = list(current_image.size()) shape[0] = num_evals noise = torch.randn(shape) perturbed_sample = self._clip(current_image.clone().detach() + delta * noise) noise = (perturbed_sample - current_image) / delta decisions = self._validate_sample(perturbed_sample) output_shape = decisions.size() fval = 2 * torch.reshape(decisions.float(), output_shape) - 1.0 broadcast_dest = [1] * len(noise.size()) broadcast_dest[0] = -1 fval = torch.broadcast_to(fval.reshape(broadcast_dest), noise.size()) if (int(torch.mean(fval)) == 1): gradf = torch.mean(noise, dim = 0) elif (int(torch.mean(fval)) == -1): gradf = - torch.mean(noise, dim = 0) else: fval = fval.detach().clone() - torch.mean(fval) gradf = torch.mean(fval * noise, dim = 0) gradf = gradf / torch.linalg.norm(gradf) return gradf def _compute_delta(self, current_image, initial_input, first_iter): if first_iter: return 0.1 * (self.clip_max - self.clip_min) distance = torch.linalg.norm(current_image - initial_input) return torch.sqrt(torch.prod(self.input_shape)) * self.theta * distance def _binary_search(self, current_image, initial_input, delta, target): au = 1 al = 0 alpha = 0.5 while abs(au - al) > self.theta: alpha = (au + al) / 2 perturbed_sample = self._clip((1 - alpha) * initial_input + alpha * current_image) model_output = self._validate_sample(perturbed_sample) if model_output == 1: self.cached_result = perturbed_sample au = alpha else: al = alpha return self.cached_result def _geometric_progression(self, current_image, grads, dist, num_evals, delta, i, target): epsilon = dist.clone().detach() / math.sqrt(i) def phi(epsilon): new = self._clip(current_image + epsilon * grads) success = self._validate_sample(new) return success iterations = 10 while not phi(epsilon): iterations -= 1 epsilon /= 2 if iterations == 0: break return epsilon def _validate_sample(self, adjusted_input): model_output = self.model_query_fn(adjusted_input) return torch.where(model_output == self.target, 1, 0) ``` ### FGSM.py ```python import torch def _accuracy(a, b): a = torch.argmax(a, dim = 1) print(a.size()) b = torch.argmax(b, dim = 1) values = torch.tensor([0], dtype = torch.int64) print("HERE", torch.sum(torch.heaviside(torch.abs(a - b), values))/a.size()[0]) return torch.sum(torch.heaviside(torch.abs(a - b), values))/a.size()[0] def FGSM_attack2(model, image, label, from_logits = True, loss_fn = torch.nn.CrossEntropyLoss(), epsilon = 0.001): labels = torch.zeros((1, 1000)) labels[0, label] = 1 data = image.clone().detach() original_preds = model(data) data.requires_grad = True outputs = model(data) if from_logits: outputs = torch.nn.functional.softmax(outputs, dim = 1) loss = loss_fn(outputs, labels) loss.backward() data_grad = data.grad.data perturbed_images = data + epsilon * data_grad.sign() def targeted_attack2(model, image, label, from_logits = True, loss_fn = torch.nn.CrossEntropyLoss(), epsilon = 0.001): labels = torch.zeros((1, 1000)) labels[0, label] = 1 data = image.clone().detach() data.requires_grad = True optimizer = torch.optim.Adam(list([data, ]), maximize = False) for i in range(10): placeholder = torch.nn.functional.interpolate(data, 224) outputs = model(placeholder) if from_logits: outputs = torch.nn.functional.softmax(outputs, dim = 1) loss = loss_fn(outputs, labels) loss.backward() optimizer.step() return data def FGSM_attack(model, dataloader, total_batches, from_logits = True, loss_fn = torch.nn.CrossEntropyLoss(), epsilon = 0.001): generated_images = [] original_labels = [] model.eval() percent_change = 0 count = 0 for batch in dataloader: x, y = batch data = x.clone().detach() original_preds = model(data) data.requires_grad = True outputs = model(data) if from_logits: outputs = torch.nn.functional.softmax(outputs, dim = 1) loss = loss_fn(outputs, y) loss.backward() data_grad = data.grad.data perturbed_images = data + epsilon * data_grad.sign() generated_images.append(perturbed_images) original_labels.append(y) count += 1 new_preds = model(perturbed_images) if count >= total_batches: break return generated_images def PGD_attack(model, dataloader, total_batches, iterations, from_logits = True, loss_fn = torch.nn.CrossEntropyLoss(), epsilon = 0.001): count = 0 running_accuracy = 0 generated_images = [] original_labels = [] for batch in dataloader: x, y = batch original_preds = model(x) accuracy = _accuracy(original_preds, y) for i in range(iterations): data = x.clone().detach() data.requires_grad = True outputs = model(data) if from_logits: outputs = torch.nn.functional.softmax(outputs, dim = 1) loss = loss_fn(outputs, y) loss.backward() data_grad = data.grad.data perturbed_images = data + epsilon * data_grad count += 1 new_preds = model(perturbed_images) new_accuracy = _accuracy(new_preds, y) accuracy_change = torch.abs(new_accuracy - accuracy) print(running_accuracy) running_accuracy += accuracy_change generated_images.append(perturbed_images) original_labels.append(y) if count >= total_batches: break running_accuracy /= count return generated_images, original_labels, running_accuracy def targeted_adversarial_attack(model, dataloader, total_batches, iterations, target_label, from_logits = True, loss_fn = torch.nn.CrossEntropyLoss()): count = 0 percent_change = 0 generated_images = [] original_labels = [] running_accuracy = 0 for batch in dataloader: x, y = batch data = x.clone().detach() data.requires_grad = True original_preds = model(data) accuracy = _accuracy(original_preds, y) optimizer = torch.optim.Adam(list([data, ]), maximize = False) for i in range(iterations): optimizer.zero_grad() outputs = model(data) if from_logits: outputs = torch.nn.functional.softmax(outputs, dim = 1) labels = target_label loss = loss_fn(outputs, labels) loss.backward() optimizer.step() accuracy = _accuracy(original_preds, y) new_preds = model(data) new_accuracy = loss_fn(new_preds, labels) running_accuracy += torch.abs(new_accuracy - accuracy) count += 1 generated_images.append(data) original_labels.append(y) if count >= total_batches: break running_accuracy /= count return generated_images, original_labels, running_accuracy ``` ### generate.py ```python import torch from PIL import Image import FGSM from torchvision import transforms import numpy as np categories = ['tench', 'goldfish', 'great white shark', 'tiger shark', 'hammerhead', 'electric ray', 'stingray', 'cock', 'hen', 'ostrich', 'brambling', 'goldfinch', 'house finch', 'junco', 'indigo bunting', 'robin', 'bulbul', 'jay', 'magpie', 'chickadee', 'water ouzel', 'kite', 'bald eagle', 'vulture', 'great grey owl', 'European fire salamander', 'common newt', 'eft', 'spotted salamander', 'axolotl', 'bullfrog', 'tree frog', 'tailed frog', 'loggerhead', 'leatherback turtle', 'mud turtle', 'terrapin', 'box turtle', 'banded gecko', 'common iguana', 'American chameleon', 'whiptail', 'agama', 'frilled lizard', 'alligator lizard', 'Gila monster', 'green lizard', 'African chameleon', 'Komodo dragon', 'African crocodile', 'American alligator', 'triceratops', 'thunder snake', 'ringneck snake', 'hognose snake', 'green snake', 'king snake', 'garter snake', 'water snake', 'vine snake', 'night snake', 'boa constrictor', 'rock python', 'Indian cobra', 'green mamba', 'sea snake', 'horned viper', 'diamondback', 'sidewinder', 'trilobite', 'harvestman', 'scorpion', 'black and gold garden spider', 'barn spider', 'garden spider', 'black widow', 'tarantula', 'wolf spider', 'tick', 'centipede', 'black grouse', 'ptarmigan', 'ruffed grouse', 'prairie chicken', 'peacock', 'quail', 'partridge', 'African grey', 'macaw', 'sulphur-crested cockatoo', 'lorikeet', 'coucal', 'bee eater', 'hornbill', 'hummingbird', 'jacamar', 'toucan', 'drake', 'red-breasted merganser', 'goose', 'black swan', 'tusker', 'echidna', 'platypus', 'wallaby', 'koala', 'wombat', 'jellyfish', 'sea anemone', 'brain coral', 'flatworm', 'nematode', 'conch', 'snail', 'slug', 'sea slug', 'chiton', 'chambered nautilus', 'Dungeness crab', 'rock crab', 'fiddler crab', 'king crab', 'American lobster', 'spiny lobster', 'crayfish', 'hermit crab', 'isopod', 'white stork', 'black stork', 'spoonbill', 'flamingo', 'little blue heron', 'American egret', 'bittern', 'crane', 'limpkin', 'European gallinule', 'American coot', 'bustard', 'ruddy turnstone', 'red-backed sandpiper', 'redshank', 'dowitcher', 'oystercatcher', 'pelican', 'king penguin', 'albatross', 'grey whale', 'killer whale', 'dugong', 'sea lion', 'Chihuahua', 'Japanese spaniel', 'Maltese dog', 'Pekinese', 'Shih-Tzu', 'Blenheim spaniel', 'papillon', 'toy terrier', 'Rhodesian ridgeback', 'Afghan hound', 'basset', 'beagle', 'bloodhound', 'bluetick', 'black-and-tan coonhound', 'Walker hound', 'English foxhound', 'redbone', 'borzoi', 'Irish wolfhound', 'Italian greyhound', 'whippet', 'Ibizan hound', 'Norwegian elkhound', 'otterhound', 'Saluki', 'Scottish deerhound', 'Weimaraner', 'Staffordshire bullterrier', 'American Staffordshire terrier', 'Bedlington terrier', 'Border terrier', 'Kerry blue terrier', 'Irish terrier', 'Norfolk terrier', 'Norwich terrier', 'Yorkshire terrier', 'wire-haired fox terrier', 'Lakeland terrier', 'Sealyham terrier', 'Airedale', 'cairn', 'Australian terrier', 'Dandie Dinmont', 'Boston bull', 'miniature schnauzer', 'giant schnauzer', 'standard schnauzer', 'Scotch terrier', 'Tibetan terrier', 'silky terrier', 'soft-coated wheaten terrier', 'West Highland white terrier', 'Lhasa', 'flat-coated retriever', 'curly-coated retriever', 'golden retriever', 'Labrador retriever', 'Chesapeake Bay retriever', 'German short-haired pointer', 'vizsla', 'English setter', 'Irish setter', 'Gordon setter', 'Brittany spaniel', 'clumber', 'English springer', 'Welsh springer spaniel', 'cocker spaniel', 'Sussex spaniel', 'Irish water spaniel', 'kuvasz', 'schipperke', 'groenendael', 'malinois', 'briard', 'kelpie', 'komondor', 'Old English sheepdog', 'Shetland sheepdog', 'collie', 'Border collie', 'Bouvier des Flandres', 'Rottweiler', 'German shepherd', 'Doberman', 'miniature pinscher', 'Greater Swiss Mountain dog', 'Bernese mountain dog', 'Appenzeller', 'EntleBucher', 'boxer', 'bull mastiff', 'Tibetan mastiff', 'French bulldog', 'Great Dane', 'Saint Bernard', 'Eskimo dog', 'malamute', 'Siberian husky', 'dalmatian', 'affenpinscher', 'basenji', 'pug', 'Leonberg', 'Newfoundland', 'Great Pyrenees', 'Samoyed', 'Pomeranian', 'chow', 'keeshond', 'Brabancon griffon', 'Pembroke', 'Cardigan', 'toy poodle', 'miniature poodle', 'standard poodle', 'Mexican hairless', 'timber wolf', 'white wolf', 'red wolf', 'coyote', 'dingo', 'dhole', 'African hunting dog', 'hyena', 'red fox', 'kit fox', 'Arctic fox', 'grey fox', 'tabby', 'tiger cat', 'Persian cat', 'Siamese cat', 'Egyptian cat', 'cougar', 'lynx', 'leopard', 'snow leopard', 'jaguar', 'lion', 'tiger', 'cheetah', 'brown bear', 'American black bear', 'ice bear', 'sloth bear', 'mongoose', 'meerkat', 'tiger beetle', 'ladybug', 'ground beetle', 'long-horned beetle', 'leaf beetle', 'dung beetle', 'rhinoceros beetle', 'weevil', 'fly', 'bee', 'ant', 'grasshopper', 'cricket', 'walking stick', 'cockroach', 'mantis', 'cicada', 'leafhopper', 'lacewing', 'dragonfly', 'damselfly', 'admiral', 'ringlet', 'monarch', 'cabbage butterfly', 'sulphur butterfly', 'lycaenid', 'starfish', 'sea urchin', 'sea cucumber', 'wood rabbit', 'hare', 'Angora', 'hamster', 'porcupine', 'fox squirrel', 'marmot', 'beaver', 'guinea pig', 'sorrel', 'zebra', 'hog', 'wild boar', 'warthog', 'hippopotamus', 'ox', 'water buffalo', 'bison', 'ram', 'bighorn', 'ibex', 'hartebeest', 'impala', 'gazelle', 'Arabian camel', 'llama', 'weasel', 'mink', 'polecat', 'black-footed ferret', 'otter', 'skunk', 'badger', 'armadillo', 'three-toed sloth', 'orangutan', 'gorilla', 'chimpanzee', 'gibbon', 'siamang', 'guenon', 'patas', 'baboon', 'macaque', 'langur', 'colobus', 'proboscis monkey', 'marmoset', 'capuchin', 'howler monkey', 'titi', 'spider monkey', 'squirrel monkey', 'Madagascar cat', 'indri', 'Indian elephant', 'African elephant', 'lesser panda', 'giant panda', 'barracouta', 'eel', 'coho', 'rock beauty', 'anemone fish', 'sturgeon', 'gar', 'lionfish', 'puffer', 'abacus', 'abaya', 'academic gown', 'accordion', 'acoustic guita [truncated — 8575 more characters] ``` [12 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]