# Project export: EcoSim This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Shape Earth’s future one decision at a time. Tackle climate change through bold policy, innovation, and action in a dynamic simulation where every choice sparks real planetary impact. - Devpost: https://devpost.com/software/ecosim-pw3ic4 - GitHub: https://github.com/rithwikn05/UCB-AI-Hack-2025 - Team: 3 GitHub contributor(s) — Rith Wik (8 commits), anpa01 (8 commits), Kinjal Govil (3 commits) ## Devpost submission (written by the team) ### Inspiration In a world flooded with climate misinformation, education must go beyond facts—it must engage. This game transforms passive awareness into active understanding by letting players experience the real-world impact of their choices. Through interactive storytelling and dynamic simulation, it cuts through confusion, debunks myths, and empowers users to explore evidence-based solutions. You can’t fact-check the future—but you can simulate it. ### What it does Our platform is an interactive climate change simulator that lets users explore how their decisions—like enacting policies, adopting technologies, or shifting behaviors—impact the planet. Through a dynamic, game-like experience, users can test the outcomes of climate strategies in real time, learning how local actions cascade into global consequences. The goal is to combat misinformation and promote systems thinking by making climate science engaging, visual, and experiential. ### How we built it The simulation runs on a multi-layered AI stack. Fetch.ai agents, powered by Grok, model decentralized decision-making and policy interactions across sectors and regions. Anthropic’s Claude is used to label, classify, and contextualize user-generated data and in-game scenarios, ensuring semantic coherence and accuracy. The core climate engine is driven by Google’s Gemini, which runs high-fidelity simulations that translate user actions into environmental and socio-economic outcomes—such as CO₂ levels, sea rise, energy transitions, and ecosystem shifts. The system uses emergent behavior to reflect real-world complexity and uncertainty in climate dynamics. ### Challenges we ran into Accessing high-quality satellite imagery for climate modeling poses significant hurdles. Public archives often contain data plagued by cloud cover, poor lighting conditions, or inconsistent sensor saturation, making it difficult to isolate usable frames for training or visualization. Finding cloud-free, seasonally consistent, and radiometrically calibrated imagery—especially for specific time periods or locations—requires intensive filtering and cross-referencing across multiple datasets and metadata fields. Compounding this is the challenge of ingesting such data at scale. Many satellite data APIs are rate-limited or backed by slow, legacy servers not optimized for large, parallel workloads. Efficiently scraping or querying these sources requires sophisticated orchestration: massively parallelizing API requests without overloading endpoints, managing retries, caching intermediate results, and balancing throughput with stability. These bottlenecks directly impact the system's ability to deliver timely and reliable simulations grounded in real-world geospatial data. ### Accomplishments we're proud of One of our key achievements is the development of an AI agent capable of interfacing seamlessly with a wide range of major weather and climate APIs. This includes services from NASA, NOAA, ECMWF, Open-Meteo, and commercial platforms, each with their own data formats, authentication schemes, and domain-specific quirks. The agent dynamically adapts to different endpoints—whether it's retrieving historical climate trends, real-time weather, forecast models, or satellite data layers—and intelligently harmonizes the responses into a unified schema for downstream analysis or simulation. It can parse heterogeneous metadata, handle missing or uncertain inputs, and resolve spatial and temporal mismatches across sources. This interoperability transforms fragmented climate intelligence into actionable context, enabling real-time scenario generation and significantly enhancing the responsiveness and realism of the game’s simulation engine. ### What we learned Working with climate data at scale taught us a hard truth: big data analysis is slow—painfully slow—especially when juggling high-resolution geospatial inputs, real-time simulation, and fragmented API ecosystems. It’s easy to fall into the trap of endlessly optimizing data pipelines or chasing more datasets in the name of accuracy, but at some point, we had to ask: Is this making the experience better, or just slower? One of our key takeaways was the importance of drawing a clear line between "data completeness" and building a functional MVP. Having a simulation that runs, teaches, and responds—even if on a smaller, cleaner dataset—is far more valuable than an endlessly delayed product waiting on the perfect archive. Prioritizing responsiveness, clarity, and user experience over data volume helped us stay lean, ship faster, and focus on what really matters: insight, not just information. ### What's next Looking ahead, we’re focused on expanding both the depth and expressiveness of the simulation. On the data side, we plan to integrate additional climate and socio-environmental datasets—including land use change, wildfire risk, migration projections, and climate vulnerability indices—to enrich scenario realism and provide more nuanced user feedback. In parallel, we’re developing a finetuned Stable Diffusion model trained on curated satellite and climate impact imagery. This will allow the simulator to generate visual futures—photorealistic, localized renderings of environments under different climate trajectories, bringing abstract data to life. By combining quantitative simulation with evocative AI-generated visuals, we aim to make the consequences of climate decisions more immediate, visceral, and personal. ## README (from the GitHub repository) # UCB-AI-Hack-2025 ## Inspiration In a world flooded with climate misinformation, education must go beyond facts—it must engage. This game transforms passive awareness into active understanding by letting players experience the real-world impact of their choices. Through interactive storytelling and dynamic simulation, it cuts through confusion, debunks myths, and empowers users to explore evidence-based solutions. You can’t fact-check the future—but you can simulate it. ## What it does Our platform is an interactive climate change simulator that lets users explore how their decisions—like enacting policies, adopting technologies, or shifting behaviors—impact the planet. Through a dynamic, game-like experience, users can test the outcomes of climate strategies in real time, learning how local actions cascade into global consequences. The goal is to combat misinformation and promote systems thinking by making climate science engaging, visual, and experiential. ## How we built it The simulation runs on a multi-layered AI stack. Fetch.ai agents, powered by Grok, model decentralized decision-making and policy interactions across sectors and regions. Anthropic’s Claude is used to label, classify, and contextualize user-generated data and in-game scenarios, ensuring semantic coherence and accuracy. The core climate engine is driven by Google’s Gemini, which runs high-fidelity simulations that translate user actions into environmental and socio-economic outcomes—such as CO₂ levels, sea rise, energy transitions, and ecosystem shifts. The system uses emergent behavior to reflect real-world complexity and uncertainty in climate dynamics. ## Challenges we ran into Accessing high-quality satellite imagery for climate modeling poses significant hurdles. Public archives often contain data plagued by cloud cover, poor lighting conditions, or inconsistent sensor saturation, making it difficult to isolate usable frames for training or visualization. Finding cloud-free, seasonally consistent, and radiometrically calibrated imagery—especially for specific time periods or locations—requires intensive filtering and cross-referencing across multiple datasets and metadata fields. Compounding this is the challenge of ingesting such data at scale. Many satellite data APIs are rate-limited or backed by slow, legacy servers not optimized for large, parallel workloads. Efficiently scraping or querying these sources requires sophisticated orchestration: massively parallelizing API requests without overloading endpoints, managing retries, caching intermediate results, and balancing throughput with stability. These bottlenecks directly impact the system's ability to deliver timely and reliable simulations grounded in real-world geospatial data. ## Accomplishments that we're proud of One of our key achievements is the development of an AI agent capable of interfacing seamlessly with a wide range of major weather and climate APIs. This includes services from NASA, NOAA, ECMWF, Open-Meteo, and commercial platforms, each with their own data formats, authentication schemes, and domain-specific quirks. The agent dynamically adapts to different endpoints—whether it's retrieving historical climate trends, real-time weather, forecast models, or satellite data layers—and intelligently harmonizes the responses into a unified schema for downstream analysis or simulation. It can parse heterogeneous metadata, handle missing or uncertain inputs, and resolve spatial and temporal mismatches across sources. This interoperability transforms fragmented climate intelligence into actionable context, enabling real-time scenario generation and significantly enhancing the responsiveness and realism of the game’s simulation engine. ## What we learned Working with climate data at scale taught us a hard truth: big data analysis is slow—painfully slow—especially when juggling high-resolution geospatial inputs, real-time simulation, and fragmented API ecosystems. It’s easy to fall into the trap of endlessly optimizing data pipelines or chasing more datasets in the name of accuracy, but at some point, we had to ask: Is this making the experience better, or just slower? One of our key takeaways was the importance of drawing a clear line between "data completeness" and building a functional MVP. Having a simulation that runs, teaches, and responds—even if on a smaller, cleaner dataset—is far more valuable than an endlessly delayed product waiting on the perfect archive. Prioritizing responsiveness, clarity, and user experience over data volume helped us stay lean, ship faster, and focus on what really matters: insight, not just information. ## What's next for EcoSim Looking ahead, we’re focused on expanding both the depth and expressiveness of the simulation. On the data side, we plan to integrate additional climate and socio-environmental datasets—including land use change, wildfire risk, migration projections, and climate vulnerability indices—to enrich scenario realism and provide more nuanced user feedback. In parallel, we’re developing a finetuned Stable Diffusion model trained on curated satellite and climate impact imagery. This will allow the simulator to generate visual futures—photorealistic, localized renderings of environments under different climate trajectories, bringing abstract data to life. By combining quantitative simulation with evocative AI-generated visuals, we aim to make the consequences of climate decisions more immediate, visceral, and personal. ## Detected evidence (automated analysis) Indexed codebase: 124 recognized source files, 2704 KB. - Anthropic (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 1170) ``` .DS_Store .env .vscode/settings.json 930_056W_11S_930_chunk_0042.txt analysis_results.txt analyze_satellite_images.py app.py claude.py climate_agent/clean_test_buttons.py climate_agent/climate_research_agent.py climate_agent/fast_api.py climate_agent/global_disaster_test.py climate_agent/test_climate_research_agent.py climate_agent/test_hazard_recognition_accuracy.py climate_agent/test_llm_agents.py climate_agent/working_climate_agents.py curl.py Earth_Engine_REST_API_Quickstart.ipynb finetune_lora.py gemini.py img_paths.txt labels_paths.txt Nitrode_logic/.gitattributes Nitrode_logic/.gitignore Nitrode_logic/APIManager.tscn Nitrode_logic/icon.svg.import Nitrode_logic/project.godot Nitrode_logic/scenes/APIManager.tscn Nitrode_logic/scenes/Main.tscn Nitrode_logic/scenes/UI.tscn Nitrode_logic/scenes/World.tscn Nitrode_logic/scripts/APIManager.gd Nitrode_logic/scripts/GameStateManager.gd Nitrode_logic/scripts/Main.gd Nitrode_logic/scripts/UI.gd Nitrode_logic/scripts/World.gd Nitrode_logic/stub_backend.py Nitrode_logic/tmp/model_1750530424.glb Nitrode_logic/tmp/model_1750530424.glb.import Nitrode_logic/tmp/temp_preview.glb Nitrode_logic/tmp/temp_preview.glb.import read_batch_out.py README.md requirements.txt sd-scripts/_typos.toml sd-scripts/.github/dependabot.yml sd-scripts/.github/FUNDING.yml sd-scripts/.github/workflows/typos.yml sd-scripts/.gitignore sd-scripts/bitsandbytes_windows/cextension.py sd-scripts/bitsandbytes_windows/main.py sd-scripts/docs/config_README-en.md sd-scripts/docs/config_README-ja.md sd-scripts/docs/fine_tune_README_ja.md sd-scripts/docs/gen_img_README-ja.md sd-scripts/docs/masked_loss_README-ja.md sd-scripts/docs/masked_loss_README.md sd-scripts/docs/train_db_README-ja.md sd-scripts/docs/train_db_README-zh.md sd-scripts/docs/train_lllite_README-ja.md sd-scripts/docs/train_lllite_README.md sd-scripts/docs/train_network_README-ja.md sd-scripts/docs/train_network_README-zh.md sd-scripts/docs/train_README-ja.md sd-scripts/docs/train_README-zh.md sd-scripts/docs/train_SDXL-en.md sd-scripts/docs/train_ti_README-ja.md sd-scripts/docs/wd14_tagger_README-en.md sd-scripts/docs/wd14_tagger_README-ja.md sd-scripts/fine_tune.py sd-scripts/finetune/blip/blip.py sd-scripts/finetune/blip/med_config.json sd-scripts/finetune/blip/med.py sd-scripts/finetune/blip/vit.py sd-scripts/finetune/clean_captions_and_tags.py sd-scripts/finetune/hypernetwork_nai.py sd-scripts/finetune/make_captions_by_git.py sd-scripts/finetune/make_captions.py sd-scripts/finetune/merge_captions_to_metadata.py sd-scripts/finetune/merge_dd_tags_to_metadata.py sd-scripts/finetune/prepare_buckets_latents.py sd-scripts/finetune/tag_images_by_wd14_tagger.py sd-scripts/gen_img_diffusers.py sd-scripts/gen_img.py sd-scripts/library/__init__.py sd-scripts/library/adafactor_fused.py sd-scripts/library/attention_processors.py sd-scripts/library/config_util.py sd-scripts/library/custom_train_functions.py sd-scripts/library/deepspeed_utils.py sd-scripts/library/device_utils.py sd-scripts/library/huggingface_util.py sd-scripts/library/hypernetwork.py sd-scripts/library/ipex/__init__.py sd-scripts/library/ipex/attention.py sd-scripts/library/ipex/diffusers.py sd-scripts/library/ipex/gradscaler.py sd-scripts/library/ipex/hijacks.py sd-scripts/library/lpw_stable_diffusion.py sd-scripts/library/model_util.py sd-scripts/library/original_unet.py sd-scripts/library/sai_model_spec.py sd-scripts/library/sdxl_lpw_stable_diffusion.py sd-scripts/library/sdxl_model_util.py sd-scripts/library/sdxl_original_unet.py sd-scripts/library/sdxl_train_util.py sd-scripts/library/slicing_vae.py sd-scripts/library/train_util.py sd-scripts/library/utils.py sd-scripts/LICENSE.md sd-scripts/networks/check_lora_weights.py sd-scripts/networks/control_net_lllite_for_train.py sd-scripts/networks/control_net_lllite.py sd-scripts/networks/dylora.py sd-scripts/networks/extract_lora_from_dylora.py sd-scripts/networks/extract_lora_from_models.py sd-scripts/networks/lora_diffusers.py sd-scripts/networks/lora_fa.py sd-scripts/networks/lora_interrogator.py sd-scripts/networks/lora.py [1050 more files omitted for size] ``` ### Dependencies - requirements.txt: anthropic@>=0.3.0, google-generativeai@>=0.3.0, Pillow@>=10.0.0, python-dotenv@>=1.0.0, requests@>=2.31.0 - sd-scripts/requirements.txt: accelerate@==0.30.0, altair@==4.2.2, bitsandbytes@==0.44.0, diffusers[torch]@==0.25.0, easygui@==0.98.3, einops@==0.7.0, ftfy@==6.1.1, huggingface-hub@==0.24.5, imagesize@==1.4.1, lion-pytorch@==0.0.6, opencv-python@==4.8.1.78, prodigyopt@==1.0, pytorch-lightning@==1.9.0, rich@==13.7.0, safetensors@==0.4.2, tensorboard, toml@==0.10.2, transformers@==4.44.0, voluptuous@==0.13.1 ### Recent commits (newest first) - Enhance README with detailed project overview - updating flask code - last - Flask failing - agent - updating - Merge branch 'main' of github.com:rithwikn05/UCB-AI-Hack-2025 - Adding climate_agent - adding flask stuff and updating lora - adding new stability abilities - updating testing logic and stuff - Revert "updating finetuning and testingn docs" - Revert "Revert "updating finetuning and testingn docs"" - Revert "updating finetuning and testingn docs" - updating finetuning and testingn docs - updating finetuning and testingn docs - anikaits nitrode edits - anthropic start - adding nitrode logic - adding dataset loader logic ## Key source files (fetched from GitHub, selected and truncated for size) ### sd-scripts/README-ja.md ```markdown ## リポジトリについて Stable Diffusionの学習、画像生成、その他のスクリプトを入れたリポジトリです。 [README in English](./README.md) ←更新情報はこちらにあります 開発中のバージョンはdevブランチにあります。最新の変更点はdevブランチをご確認ください。 FLUX.1およびSD3/SD3.5対応はsd3ブランチで行っています。それらの学習を行う場合はsd3ブランチをご利用ください。 GUIやPowerShellスクリプトなど、より使いやすくする機能が[bmaltais氏のリポジトリ](https://github.com/bmaltais/kohya_ss)で提供されています（英語です）のであわせてご覧ください。bmaltais氏に感謝します。 以下のスクリプトがあります。 * DreamBooth、U-NetおよびText Encoderの学習をサポート * fine-tuning、同上 * LoRAの学習をサポート * 画像生成 * モデル変換（Stable Diffision ckpt/safetensorsとDiffusersの相互変換） ## 使用法について * [学習について、共通編](./docs/train_README-ja.md) : データ整備やオプションなど * [データセット設定](./docs/config_README-ja.md) * [SDXL学習](./docs/train_SDXL-en.md) （英語版） * [DreamBoothの学習について](./docs/train_db_README-ja.md) * [fine-tuningのガイド](./docs/fine_tune_README_ja.md): * [LoRAの学習について](./docs/train_network_README-ja.md) * [Textual Inversionの学習について](./docs/train_ti_README-ja.md) * [画像生成スクリプト](./docs/gen_img_README-ja.md) * note.com [モデル変換スクリプト](https://note.com/kohya_ss/n/n374f316fe4ad) ## Windowsでの動作に必要なプログラム Python 3.10.6およびGitが必要です。 - Python 3.10.6: https://www.python.org/ftp/python/3.10.6/python-3.10.6-amd64.exe - git: https://git-scm.com/download/win Python 3.10.x、3.11.x、3.12.xでも恐らく動作しますが、3.10.6でテストしています。 PowerShellを使う場合、venvを使えるようにするためには以下の手順でセキュリティ設定を変更してください。 （venvに限らずスクリプトの実行が可能になりますので注意してください。） - PowerShellを管理者として開きます。 - 「Set-ExecutionPolicy Unrestricted」と入力し、Yと答えます。 - 管理者のPowerShellを閉じます。 ## Windows環境でのインストール スクリプトはPyTorch 2.1.2でテストしています。PyTorch 2.2以降でも恐らく動作します。 （なお、python -m venv～の行で「python」とだけ表示された場合、py -m venv～のようにpythonをpyに変更してください。） PowerShellを使う場合、通常の（管理者ではない）PowerShellを開き以下を順に実行します。 ```powershell git clone https://github.com/kohya-ss/sd-scripts.git cd sd-scripts python -m venv venv .\venv\Scripts\activate pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118 pip install --upgrade -r requirements.txt pip install xformers==0.0.23.post1 --index-url https://download.pytorch.org/whl/cu118 accelerate config ``` コマンドプロンプトでも同一です。 注：`bitsandbytes==0.44.0`、`prodigyopt==1.0`、`lion-pytorch==0.0.6` は `requirements.txt` に含まれるようになりました。他のバージョンを使う場合は適宜インストールしてください。 この例では PyTorch および xfomers は2.1.2／CUDA 11.8版をインストールします。CUDA 12.1版やPyTorch 1.12.1を使う場合は適宜書き換えください。たとえば CUDA 12.1版の場合は `pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu121` および `pip install xformers==0.0.23.post1 --index-url https://download.pytorch.org/whl/cu121` としてください。 PyTorch 2.2以降を用いる場合は、`torch==2.1.2` と `torchvision==0.16.2` 、および `xformers==0.0.23.post1` を適宜変更してください。 accelerate configの質問には以下のように答えてください。（bf16で学習する場合、最後の質問にはbf16と答えてください。） ```txt - This machine - No distributed training - NO - NO - NO - all - fp16 ``` ※場合によって ``ValueError: fp16 mixed precision requires a GPU`` というエラーが出ることがあるようです。この場合、6番目の質問（ ``What GPU(s) (by id) should be used for training on this machine as a comma-separated list? [all]:``）に「0」と答えてください。（id `0`のGPUが使われます。） ## アップグレード 新しいリリースがあった場合、以下のコマンドで更新できます。 [truncated — 2359 more characters] ``` ### sd-scripts/LICENSE.md ```markdown Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control sy [truncated — 8342 more characters] ``` ### requirements.txt ``` anthropic>=0.3.0 Pillow>=10.0.0 python-dotenv>=1.0.0 requests>=2.31.0 google-generativeai>=0.3.0 ``` ### sd-scripts/requirements.txt ``` accelerate==0.30.0 transformers==4.44.0 diffusers[torch]==0.25.0 ftfy==6.1.1 # albumentations==1.3.0 opencv-python==4.8.1.78 einops==0.7.0 pytorch-lightning==1.9.0 bitsandbytes==0.44.0 prodigyopt==1.0 lion-pytorch==0.0.6 tensorboard safetensors==0.4.2 # gradio==3.16.2 altair==4.2.2 easygui==0.98.3 toml==0.10.2 voluptuous==0.13.1 huggingface-hub==0.24.5 # for Image utils imagesize==1.4.1 # for BLIP captioning # requests==2.28.2 # timm==0.6.12 # fairscale==0.4.13 # for WD14 captioning (tensorflow) # tensorflow==2.10.1 # for WD14 captioning (onnx) # onnx==1.15.0 # onnxruntime-gpu==1.17.1 # onnxruntime==1.17.1 # for cuda 12.1(default 11.8) # onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/ # this is for onnx: # protobuf==3.20.3 # open clip for SDXL # open-clip-torch==2.20.0 # For logging rich==13.7.0 # for kohya_ss library -e . ``` ### app.py ```python from flask import Flask, render_template, request, jsonify, url_for, redirect import os import random from climate_agent.working_climate_agents import llm_analyze_location import google.genai as genai from google.genai import types from PIL import Image from io import BytesIO import json import asyncio import shutil from pydantic import BaseModel app = Flask(__name__) GOOGLE_API_KEY=os.getenv("GOOGLE_API_KEY") latitude = None longitude = None label_path = None image_path = None # Placeholder for button labels (could be dynamically set by REST API later) class ImageFeature(BaseModel): name: str location: str size: str class ImageDescription(BaseModel): geographic_features: list[ImageFeature] man_made_features: list[ImageFeature] def generate_modified_satellite_image(prompt, image_path, output_path='output.png'): image = Image.open(image_path) client = genai.Client(api_key=GOOGLE_API_KEY) text_input = ("The input image is a satellite image with various fearures. Modify the image to match the fearures in the following json:\n\n" + prompt) response = client.models.generate_content( model="gemini-2.0-flash-preview-image-generation", contents=[text_input, image], config=types.GenerateContentConfig( response_modalities=['TEXT', 'IMAGE'] ) ) out = "" for part in response.candidates[0].content.parts: if part.text is not None: out += part.text elif part.inline_data is not None: image = Image.open(BytesIO(part.inline_data.data)) image.save(output_path) return out def get_disaster_prevention_techniques(prompt): client = genai.Client(api_key=GOOGLE_API_KEY) response = client.models.generate_content( model="gemini-2.0-flash", contents=f"Generate 4 specific climate change disaster prevention techniques as a comma-separated list. Input disasters: {prompt}. Return ONLY the comma-separated list, no extra text. Example format: prescribed burn, solar panel installation, direct carbon capture, reforestation", config=types.GenerateContentConfig( maxOutputTokens=50, response_modalities=['TEXT'] ) ) for part in response.candidates[0].content.parts: if part.text is not None: # Clean up the response - remove any extra text and get just the techniques text = part.text.strip() # If the response contains a colon, take everything after it if ":" in text: text = text.split(":")[-1].strip() # Remove any leading/trailing quotes or extra formatting text = text.replace('"', '').replace("'", "").strip() return text return "prescribed burn, solar panel installation, direct carbon capture, reforestation" def generate_image_description(image_path): """Generate a JSON description of the satellite image""" try: image = Image.open(image_path) client = genai.Client(api_key=GOOGLE_API_KEY) response = client.models.generate_content( model="gemini-2.0-flash", contents=[ "Analyze this satellite image and describe the geographic and man-made features you can see. Focus on terrain, vegetation, water bodies, buildings, roads, and other notable features.", image ], config=types.GenerateContentConfig( response_modalities=['TEXT'], response_mime_type="application/json", response_schema=ImageDescription ) ) return response.parsed.model_dump_json() except Exception as e: print(f"Error generating image description: {e}") # Return a default description default_desc = ImageDescription( geographic_features=[ ImageFeature(name="terrain", location="center", size="large"), ImageFeature(name="vegetation", location="scattered", size="medium") ], man_made_features=[ ImageFeature(name="unknown", location="unknown", size="small") ] ) return default_desc.model_dump_json() def time_evolve_json_description(json_description, climate_prevention_technique, latitude, longitude): client = genai.Client(api_key=GOOGLE_API_KEY) # Ensure latitude and longitude are strings lat_str = str(latitude) if latitude is not None else "0" lon_str = str(longitude) if longitude is not None else "0" response = client.models.generate_content( model="gemini-2.5-flash", contents=f"Simulate the effects of climate change and human activities over a period of 5 years on the following satellite image description in JSON format: {json_description}\n\nThe image reflects a 14km by 14km area at latitude {lat_str} and longitude {lon_str}. Assume the following climate prevention technique is being used: {climate_prevention_technique}", config=types.GenerateContentConfig( system_instruction="You are a climate change simulation engine. You specialize in simulating the effects of climate change and human activities on satellite image descriptions in JSON format. Ensure that every word in the string is seperated by a space.\n\n", response_modalities=['TEXT'], response_mime_type="application/json", response_schema=ImageDescription ), ) # print(response) return response.parsed.model_dump_json() @app.route('/') def index(): global latitude, longitude, label_path, image_path # Choose a random index between 1 and 1000 random_index = random.randint(1, 1000) image_path = None sentence_path = None analysis = None buttons = [] # ← ensure the name exists even on error image_url = None # Initialize image_url try: with open("img_paths.txt", "r") [truncated — 8278 more characters] ``` ### sd-scripts/bitsandbytes_windows/main.py ```python """ extract factors the build is dependent on: [X] compute capability [ ] TODO: Q - What if we have multiple GPUs of different makes? - CUDA version - Software: - CPU-only: only CPU quantization functions (no optimizer, no matrix multiple) - CuBLAS-LT: full-build 8-bit optimizer - no CuBLAS-LT: no 8-bit matrix multiplication (`nomatmul`) evaluation: - if paths faulty, return meaningful error - else: - determine CUDA version - determine capabilities - based on that set the default path """ import ctypes from .paths import determine_cuda_runtime_lib_path def check_cuda_result(cuda, result_val): # 3. Check for CUDA errors if result_val != 0: error_str = ctypes.c_char_p() cuda.cuGetErrorString(result_val, ctypes.byref(error_str)) print(f"CUDA exception! Error code: {error_str.value.decode()}") def get_cuda_version(cuda, cudart_path): # https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART____VERSION.html#group__CUDART____VERSION try: cudart = ctypes.CDLL(cudart_path) except OSError: # TODO: shouldn't we error or at least warn here? print(f'ERROR: libcudart.so could not be read from path: {cudart_path}!') return None version = ctypes.c_int() check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version))) version = int(version.value) major = version//1000 minor = (version-(major*1000))//10 if major < 11: print('CUDA SETUP: CUDA version lower than 11 are currently not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!!') return f'{major}{minor}' def get_cuda_lib_handle(): # 1. find libcuda.so library (GPU driver) (/usr/lib) try: cuda = ctypes.CDLL("libcuda.so") except OSError: # TODO: shouldn't we error or at least warn here? print('CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!') return None check_cuda_result(cuda, cuda.cuInit(0)) return cuda def get_compute_capabilities(cuda): """ 1. find libcuda.so library (GPU driver) (/usr/lib) init_device -> init variables -> call function by reference 2. call extern C function to determine CC (https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__DEVICE__DEPRECATED.html) 3. Check for CUDA errors https://stackoverflow.com/questions/14038589/what-is-the-canonical-way-to-check-for-errors-using-the-cuda-runtime-api # bits taken from https://gist.github.com/f0k/63a664160d016a491b2cbea15913d549 """ nGpus = ctypes.c_int() cc_major = ctypes.c_int() cc_minor = ctypes.c_int() device = ctypes.c_int() check_cuda_result(cuda, cuda.cuDeviceGetCount(ctypes.byref(nGpus))) ccs = [] for i in range(nGpus.value): check_cuda_result(cuda, cuda.cuDeviceGet(ctypes.byref(device), i)) ref_major = ctypes.byref(cc_major) ref_minor = ctypes.byref(cc_minor) # 2. call extern C function to determine CC check_cuda_result( cuda, cuda.cuDeviceComputeCapability(ref_major, ref_minor, device) ) ccs.append(f"{cc_major.value}.{cc_minor.value}") return ccs # def get_compute_capability()-> Union[List[str, ...], None]: # FIXME: error def get_compute_capability(cuda): """ Extracts the highest compute capbility from all available GPUs, as compute capabilities are downwards compatible. If no GPUs are detected, it returns None. """ ccs = get_compute_capabilities(cuda) if ccs is not None: # TODO: handle different compute capabilities; for now, take the max return ccs[-1] return None def evaluate_cuda_setup(): print('') print('='*35 + 'BUG REPORT' + '='*35) print('Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues') print('For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link') print('='*80) return "libbitsandbytes_cuda116.dll" # $$$ binary_name = "libbitsandbytes_cpu.so" #if not torch.cuda.is_available(): #print('No GPU detected. Loading CPU library...') #return binary_name cudart_path = determine_cuda_runtime_lib_path() if cudart_path is None: print( "WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!" ) return binary_name print(f"CUDA SETUP: CUDA runtime path found: {cudart_path}") cuda = get_cuda_lib_handle() cc = get_compute_capability(cuda) print(f"CUDA SETUP: Highest compute capability among GPUs detected: {cc}") cuda_version_string = get_cuda_version(cuda, cudart_path) if cc == '': print( "WARNING: No GPU detected! Check your CUDA paths. Processing to load CPU-only library..." ) return binary_name # 7.5 is the minimum CC vor cublaslt has_cublaslt = cc in ["7.5", "8.0", "8.6"] # TODO: # (1) CUDA missing cases (no CUDA installed by CUDA driver (nvidia-smi accessible) # (2) Multiple CUDA versions installed # we use ls -l instead of nvcc to determine the cuda version # since most installations will have the libcudart.so installed, but not the compiler print(f'CUDA SETUP: Detected CUDA version {cuda_version_string}') def get_binary_name(): "if not has_cublaslt (CC < 7.5), then we have to choose _nocublaslt.so" bin_base_name = "libbitsandbytes_cuda" if has_cublaslt: return f"{bin_base_name}{cu [truncated — 178 more characters] ``` ### stability.py ```python import replicate prompt = "fire" input = { "prompt": "an ariel view of a landscape that has been affected by {prompt}. Make it zoom out and realistic.", "scheduler": "K_EULER" } output = replicate.run( "stability-ai/stable-diffusion:ac732df83cea7fff18b8472768c88ad041fa750ff7682a21affe81863cbe77e4", input=input ) for index, item in enumerate(output): with open(f"output_{index}.png", "wb") as file: file.write(item.read()) print(prompt) #=> output_0.png written to disk ``` ### read_batch_out.py ```python import anthropic client = anthropic.Anthropic() # Stream results file in memory-efficient chunks, processing one at a time for result in client.messages.batches.results( "msgbatch_01HkcTjaV5uDC8jWR4ZsDV8d", ): match result.result.type: case "succeeded": out_txt = "{\n \"geographic_features\": [\n {\n \"name\": \"" out_txt += result.result.message.content[0].text result.custom_id case "errored": if result.result.error.type == "invalid_request": # Request body must be fixed before re-sending request print(f"Validation error {result.custom_id}") else: # Request can be retried directly print(f"Server error {result.custom_id}") case "expired": print(f"Request expired {result.custom_id}") ``` ### gemini.py ```python import os from google import genai from google.genai import types import pydantic from PIL import Image from io import BytesIO import json from pydantic import BaseModel class ImageFeature(BaseModel): name: str location: str size: str class ImageDescription(BaseModel): geographic_features: list[ImageFeature] man_made_features: list[ImageFeature] def generate_modified_satellite_image(prompt, image_path, output_path='output.png'): image = Image.open(image_path) client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY")) text_input = ("The input image is a satellite image with various fearures. Modify the image to match the fearures in the following json:\n\n" + prompt) response = client.models.generate_content( model="gemini-2.0-flash-preview-image-generation", contents=[text_input, image], config=types.GenerateContentConfig( response_modalities=['TEXT', 'IMAGE'] ) ) out = "" for part in response.candidates[0].content.parts: if part.text is not None: out += part.text elif part.inline_data is not None: image = Image.open(BytesIO(part.inline_data.data)) image.show() image.save(output_path) return out def get_disaster_prevention_techniques(prompt): client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY")) response = client.models.generate_content( model="gemini-2.0-flash", contents="For each of the following disasters list, generate a single climate change disaster prevention technique and return them as a comma separated list (eg: For input (wildfire, mining operations, industrialization, deforestation) a possible output would be: prescribed burn, solar panel installation, direct carbon capture, reforestation: " + prompt + "\n\n" + "", config=types.GenerateContentConfig( maxOutputTokens=len(prompt.split(","))*4, response_modalities=['TEXT'] ) ) for part in response.candidates[0].content.parts: if part.text is not None: return part.text return "prescribed burn, solar panel installation, direct carbon capture, reforestation" def time_evolve_json_description(json_description, climate_prevention_technique, latitude, longitude): client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY")) response = client.models.generate_content( model="gemini-2.5-flash", contents="Simulate the effects of climate change and human activities over a period of 5 years on the following satellite image description in JSON format: " + json_description + "\n\n" + " The image reflects a 14km by 14km area at latitude " + latitude + " and longitude " + longitude + ". Assume the following climate prevention technique is being used: " + climate_prevention_technique, config=types.GenerateContentConfig( system_instruction="You are a climate change simulation engine. You specialize in simulating the effects of climate change and human activities on satellite image descriptions in JSON format. Ensure that every word in the string is seperated by a space.\n\n", response_modalities=['TEXT'], response_mime_type="application/json", response_schema=ImageDescription ), ) # print(response) return response.parsed.model_dump_json() label_path = "/Users/kinjal/Code/UCB-AI-Hack-2025/930_056W_11S_930_chunk_0042.txt" image_path = "/Users/kinjal/Code/UCB-AI-Hack-2025/930_056W_11S_930_chunk_0042.png" button_label = "solar panel installation" latitude = "11S" longitude = "56W" new_json = time_evolve_json_description( json_description=open(label_path, "r").read(), climate_prevention_technique=button_label, latitude=latitude, longitude=longitude ) simulation_description = generate_modified_satellite_image( prompt=new_json, image_path=image_path, output_path="gemini_output.png" ) # def generate_image(prompt, image_path, output_path='output.png', api_key=None): # """ # Generate an image using Google's Gemini model based on a text prompt and an input image. # Args: # prompt (str): Text prompt to guide image generation # image_path (str): Path to the input image file # output_path (str): Path to save the generated image (default: 'output.png') # api_key (str, optional): Google AI API key. If not provided, will try to use GOOGLE_API_KEY environment variable. # Returns: # str: Path to the generated image if successful, None otherwise # """ # try: # # Configure the API key # if api_key: # genai.configure(api_key=api_key) # # Initialize the model # model = genai.GenerativeModel('gemini-2.0-flash-vision') # # Read the input image # image_data = genai.upload_file(path=image_path) # # Generate content # response = model.generate_content([prompt, image_data]) # # Extract the generated image (assuming it's in the response) # if hasattr(response, 'images') and response.images: # with open(output_path, 'wb') as f: # f.write(response.images[0].getvalue()) # return output_path # else: # print("No image was generated in the response") # return None # except GoogleAPIError as e: # print(f"Google API Error: {e}") # return None # except Exception as e: # print(f"An error occurred: {e}") # return None ``` ### finetune_lora.py ```python #!/usr/bin/env python3 import subprocess import sys import os import time import torch from huggingface_hub import snapshot_download, login def check_gpu_availability(): """Check GPU availability and print info""" if torch.cuda.is_available(): gpu_count = torch.cuda.device_count() print(f"✓ CUDA available with {gpu_count} GPU(s)") for i in range(gpu_count): gpu_name = torch.cuda.get_device_name(i) gpu_memory = torch.cuda.get_device_properties(i).total_memory / 1024**3 print(f" GPU {i}: {gpu_name} ({gpu_memory:.1f}GB)") return True else: print("✗ CUDA not available - training will be very slow!") return False def optimize_training_params(gpu_count, gpu_memory_gb): """Optimize training parameters based on available GPU resources""" if gpu_memory_gb >= 24: # RTX 4090, A100, etc. return { 'batch_size': 4 if gpu_count >= 2 else 2, 'resolution': '1024,1024', 'precision': 'bf16', 'gradient_checkpointing': False } elif gpu_memory_gb >= 16: # RTX 4080, etc. return { 'batch_size': 2, 'resolution': '768,768', 'precision': 'fp16', 'gradient_checkpointing': True } elif gpu_memory_gb >= 12: # RTX 4070 Ti, etc. return { 'batch_size': 1, 'resolution': '512,512', 'precision': 'fp16', 'gradient_checkpointing': True } else: # Lower memory GPUs return { 'batch_size': 1, 'resolution': '512,512', 'precision': 'fp16', 'gradient_checkpointing': True } if __name__ == "__main__": print("🚀 Starting LoRA Training Setup") # Check GPU availability has_gpu = check_gpu_availability() if not has_gpu: response = input("No GPU detected. Continue anyway? (y/N): ") if response.lower() != 'y': sys.exit(1) # Configuration - UPDATE THESE VALUES! GCS_BUCKET_NAME = "your-bucket-name" # ← CHANGE THIS to your actual bucket name GCS_BUCKET_PATH = "test_images" # ← CHANGE THIS - trains on 880x550 resolution images # Get GPU info for optimization gpu_memory_gb = 0 gpu_count = 0 if has_gpu: gpu_count = torch.cuda.device_count() gpu_memory_gb = torch.cuda.get_device_properties(0).total_memory / 1024**3 # Optimize parameters based on hardware params = optimize_training_params(gpu_count, gpu_memory_gb) print(f"📊 Optimized settings: {params}") try: # Authenticate with Hugging Face # HF_TOKEN = "hf_your_token_here" # ← REPLACE with your actual HF token hf_token = os.environ.get("HF_TOKEN") login(token=hf_token) # Download model weights print("📥 Downloading model weights...") snapshot_download( repo_id="Djrango/Qwen2vl-Flux", allow_patterns="*.safetensors", local_dir="./models/qwen2vl-flux", token=hf_token # Pass token explicitly ) # Paths PRETRAINED_MODEL = "./models/qwen2vl-flux" IMAGE_FOLDER = "/test_images" OUTPUT_FOLDER = "./output" LOGGING_FOLDER = "./logs" MODEL_OUTPUT_NAME = "my_custom_lora_v1" # ← CHANGE THIS to name your LoRA model # Create output directories os.makedirs(OUTPUT_FOLDER, exist_ok=True) os.makedirs(LOGGING_FOLDER, exist_ok=True) # Training parameters (optimized for GPU) EPOCHS = 10 SAVE_EVERY_N_EPOCHS = 1 LEARNING_RATE = 1e-4 LR_SCHEDULER = "cosine_with_restarts" LR_WARMUP = 0.10 LORA_RANK = 64 LORA_ALPHA = 32 OPTIMIZER = "adamw8bit" SCRIPT_PATH = os.path.join("sd-scripts", "train_network.py") # Build command command = [ sys.executable, SCRIPT_PATH, "--pretrained_model_name_or_path", PRETRAINED_MODEL, "--train_data_dir", IMAGE_FOLDER, "--output_dir", OUTPUT_FOLDER, "--logging_dir", LOGGING_FOLDER, "--output_name", MODEL_OUTPUT_NAME, "--resolution", params['resolution'], "--train_batch_size", str(params['batch_size']), "--max_train_epochs", str(EPOCHS), "--save_every_n_epochs", str(SAVE_EVERY_N_EPOCHS), "--mixed_precision", params['precision'], "--save_precision", params['precision'], "--learning_rate", str(LEARNING_RATE), "--lr_scheduler", LR_SCHEDULER, "--network_module", "networks.lora", "--network_dim", str(LORA_RANK), "--network_alpha", str(LORA_ALPHA), "--optimizer_type", OPTIMIZER, "--network_train_unet_only", ] # Add GPU-specific optimizations if has_gpu: if params['gradient_checkpointing']: command.append("--gradient_checkpointing") if gpu_count > 1: command.extend(["--multi_gpu", "--num_processes", str(gpu_count)]) print("🏋️ Starting LoRA training with command:") print(" ".join(command)) print("=" * 80) # Run training process = subprocess.Popen( command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1, universal_newlines=True ) for line in process.stdout: print(line, end="") process.wait() if process.returncode == 0: print("\n🎉 Training completed successfully!") print(f"📁 Model saved to: {OUTPUT_FOLDER}") else: print(f"\n💥 Training failed with exit code {process.returncode}") excep [truncated — 266 more characters] ``` [98 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]