# Project export: Luminate

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Luminate aims to qualitatively change traditional and generative images. It doesn't change the lighting exposure, it changes the source direction
- Devpost: https://devpost.com/software/luminate-3rcldb
- GitHub: https://github.com/leelandzhang/luminate.git
- Demo: https://pitch.com/v/luminate-3k8m4w
- Team: 1 GitHub contributor(s) — Leeland Zhang (5 commits)

## Devpost submission (written by the team)

### Inspiration

Our journey began with a fascination for the transformative potential of artificial intelligence in the realm of image generation. The idea was to transcend traditional boundaries and introduce an innovative approach that could revolutionize how we interact with and manipulate digital imagery. The project idea's feasibility (and only that) was discussed with a Stanford professor a day prior, and inspired by the existing capabilities of stable diffusion models, we envisioned a system that not only generates images but offers unprecedented control over the lighting conditions within these images, making the process more dynamic and adaptable.

### What it does

Luminate is a cutting-edge platform that integrates advanced image and text generation capabilities. At its core, it leverages a modified stable diffusion model, enhanced with our innovative training techniques, to allow users to alter the direction of lighting sources within an image. This capability enables the adjustment of lighting exposure and shadows, thereby dramatically transforming the visual impact of the generated imagery. In addition, Luminate incorporates a text generation component designed to translate user inputs into precise coordinates, facilitating intuitive and efficient interactions with the system. The product can be used in the future and have powerful capabilities in whether it be photoshop software, model photography, animation, etc.

### How we built it

The development of Luminate involved the integration of several key technologies, including a controlnet diffusion model and CLIP for image generation, coupled with a custom text generation model trained on generated data. This multi-faceted approach allowed us to create a robust platform capable of understanding and executing complex user inputs. Our infrastructure leverages the power of RTX 3090 GPUs and Stanford's computing clusters, though we faced significant challenges in scaling our computational resources to meet the demands of our large dataset.

### Challenges we ran into

One of the primary hurdles we encountered was the sheer size of our dataset, which posed substantial challenges in terms of computational resource requirements. Despite having access to high-performance GPUs and university computing clusters, we struggled to achieve the desired speed and efficiency in our model training process given the time constraint. This was inevitable as stable diffusion even fine-tuning requires a large amount of data. This limitation forced us to explore alternative solutions and optimizations to progress with our project development.

### Accomplishments we're proud of

Despite the challenges, we are immensely proud of what we have achieved with Luminate. We successfully established a comprehensive product pipeline that includes a user-friendly interface, a LLM, and a nearly finalized model capable of generating black and white images. Furthermore, we devised a novel approach involving the use of a different pretrained model to transform these our output from LLM to a text-to-image Intel to generate black and white images based on specified coordinate directions, then use control net to generate images based on this restriction, bringing us close to what our vision is for the final product.

### What we learned

This project has been a tremendous learning opportunity for our team. We gained valuable insights into the complexities of training AI models on large datasets and the importance of computational resources in the field of deep learning. We also gained a full understanding of the entire control-net architecture, as we were doing very base-level development to the point of matrix calculations during training. We learned to navigate the challenges of integrating multiple AI technologies, having used Monster API, Fetch.ai, together.ai, Intel, etc. to create a cohesive and functional system. You can see our model fine tuned by with monster API as follows on hugging face: leelandzhang/moster-api-LLM-light-change (model weights).

### What's next

Looking ahead, first we are focused on overcoming the remaining challenges to fully realize our vision for Luminate, which includes the conclusion of training for our primary light-modification generation model. Physic priors can be added, so we can add physical boundary constraints, this can greatly increase the spacial precision of generated images, and coupled with our lighting, we are capable of generating more stable and accurate images than others in the market and existing tools. We can also improve the user experience with a more developed UI/UX and product usability. As we continue to innovate and expand our platform's capabilities, we are excited about the potential of Luminate to redefine the landscape of AI-powered image generation.

## README (from the GitHub repository)

��#   l u m i n a t e  
 

## Detected evidence (automated analysis)

Indexed codebase: 6 recognized source files, 85 KB.
- Python (language) — detected in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (10 of 10)

```
.DS_Store
app.py
convert_data.py
converted_data.jsonl
Copy_of_together_finetune_law_example.ipynb
README.md
sort.py
text_to_image.ipynb
train_test.py
train.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Masking of api
- Merge pull request #1 from leelandzhang/branch2
- yay
- stuff
- training
- Usage of Intel text-to-image in our pipeline
- Fine tuning with together.ai
- Data to train LLM
- Adding UI/UX for LLM and image
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### app.py

```python
import streamlit as st
from PIL import Image
import asyncio
import openai
import time
import numpy as np
import json
import base64


# class Message(Model):
#     message: str
from openai import OpenAI
client = OpenAI(api_key="mask")
# SEED_PHRASE = "agent1q2pgplfp2su2q320kz2x2w46fyua25wys9ast6gk7rc7crwan8h765ufx9e"
# AGENT_MAILBOX_KEY = "agent1q2pgplfp2su2q320kz2x2w46fyua25wys9ast6gk7rc7crwan8h765ufx9e"
# agent = Agent(
#     name="LightingRec",
#     seed=SEED_PHRASE,
#     mailbox=f"{AGENT_MAILBOX_KEY}@https://agentverse.ai",
# )

async def ask_gpt(prompt):
    response = client.chat.completions.create(
        model="ft:gpt-3.5-turbo-1106:personal::8tLPRAmo",
        messages=[
            {"role": "system", "content": "You are a lighting assistant"},
            {"role": "user", "content": prompt}],
            temperature=0.9
    )
    response_content = response.choices[0].message.content
    return response_content

st.markdown("""
    <h1 style='color: black;'>
        <span style='color: #CCAC00;'>Lumi</span>nate
    </h1>
    """, unsafe_allow_html=True)
st.write("""
    *Customizing a deeper aspect to your images, the lighting* _direction_.
""")
image=None

uploaded_file = st.file_uploader("Upload an image", type=['jpg', 'jpeg', 'png'])
if uploaded_file is not None:
    image = Image.open(uploaded_file)
    col1, col2 = st.columns(2)
    with col1:
        st.image(image, caption='Image uploaded successfully!', use_column_width=True)
    

def model(image, instruction):
    pass


# Chatbot Section
user_input = st.text_input("How would you like the lighting changed?", "")
if user_input:
    with st.spinner('Processing...'):
        response_text = asyncio.run(ask_gpt(user_input))
    st.text_area("Response", value=response_text, height=100, max_chars=None)
    if image is not None:
        # result = asyncio.run(model(image, response_text))
        # st.image(result, caption='Your new image', use_column_width=True)
        st.write("yes image") #temporary, please delete
    else:
        st.write("No image") 


# async def process_image(image_path):
#     with Image.open(image_path) as img:
#         # Convert the image to grayscale to simplify brightness analysis
#         gray_img = img.convert('L')
#         # Convert to numpy array for analysis
#         img_array = np.array(gray_img)
#     return img_array

# def save_uploaded_file(uploaded_file):
#     try:
#         with open(f"./tempDir/{uploaded_file.name}", "wb") as f:
#             f.write(uploaded_file.getbuffer())
#         return f"./tempDir/{uploaded_file.name}"
#     except Exception as e:
#         st.error(f"Failed to save the uploaded image: {e}")
#         return None

# async def handle_image_upload(uploaded_file):
#     saved_path = save_uploaded_file(uploaded_file)
#     if saved_path:
#         # Process the image asynchronously
#         return await process_image(saved_path)
#     return "Failed to process the image."


# def serialize_image_array(img_array):
#     data_b64 = base64.b64encode(img_array.tobytes()).decode('utf-8')
#     serialized_data = json.dumps({
#         "data": data_b64,
#         "shape": img_array.shape,
#         "dtype": str(img_array.dtype)
#     })
#     return serialized_data

# async def send_image_to_agent(ctx: Context, image_path):
#     # Process the image to get a numpy array
#     img_array = await process_image(image_path)

#     # Serialize img_array for transmission. Example: JSON, base64, etc.
#     # Here, we'll pretend to convert it to a JSON string (you'll need to implement this based on your data)
#     serialized_img = serialize_image_array(img_array)

#     # Send the processed image to the agent
#     ctx.logger.info("Sending processed image data to the agent")
#     await ctx.send(SEED_PHRASE, Message(data=serialized_img))


```

### train_test.py

```python
import pytorch_lightning as pl
from torch.utils.data import DataLoader
from MyDataset import MyDataset
from cldm.logger import ImageLogger
from cldm.model import create_model, load_state_dict
from torch.nn import DataParallel

# Configs
resume_path = './models/control_sd15_ini.ckpt'
batch_size = 1
logger_freq = 300
learning_rate = 1e-5
sd_locked = True
only_mid_control = False


# First use cpu to load models. Pytorch Lightning will automatically move it to GPUs.
model = create_model('./models/cldm_v15.yaml').cpu()
model.load_state_dict(load_state_dict(resume_path, location='cpu'))
model.learning_rate = learning_rate
model.sd_locked = sd_locked
model.only_mid_control = only_mid_control


# Misc
dataset = MyDataset()
dataloader = DataLoader(dataset, num_workers=0, batch_size=batch_size, shuffle=True)
logger = ImageLogger(batch_frequency=logger_freq)
trainer = pl.Trainer(gpus=1, precision=32, callbacks=[logger])


# Train!
trainer.fit(model, dataloader)


```

### train.py

```python

from share import *

import pytorch_lightning as pl
# import lightning as pl
from torch.utils.data import DataLoader
#from tutorial_dataset import MyDataset

from cldm.logger import ImageLogger
from cldm.model import create_model, load_state_dict
import numpy as np

#print(os.getcwd())
#print(sys.path)
import sys
import os
from MyDataset import MyDataset

import sys
import os
sys.path.append("../rene")
print("SYS PATH",sys.path)
from rene.utils.loaders import ReneDataset

# Configs
resume_path = './models/control_sd15_ini.ckpt'
batch_size = 4
logger_freq = 300
learning_rate = 1e-5
sd_locked = True
only_mid_control = False
num_epochs = 1

# First use cpu to load models. Pytorch Lightning will automatically move it to GPUs.
model = create_model('./models/cldm_v15.yaml').cpu()
model.load_state_dict(load_state_dict(resume_path, location='cpu'))
model.learning_rate = learning_rate
model.sd_locked = sd_locked
model.only_mid_control = only_mid_control

model1 = create_model('./models/cldm_v15.yaml').cpu()
model1.load_state_dict(load_state_dict(resume_path, location='cpu'))
model1.learning_rate = learning_rate
model1.sd_locked = sd_locked
model1.only_mid_control = only_mid_control


# Misc
dataset = MyDataset()
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
logger = ImageLogger(batch_frequency=logger_freq)
optimizer = model.configure_optimizers()

for batch in dataloader:
    
    inputs = model.get_input(batch,k=0)
    print(inputs)
    1/0

for epoch in range(num_epochs):
    running_loss = 0.0
    for batch in dataloader:
        1/0
        inputs = model.get_input(batch,k=0)
        print(inputs)


        # Zero the parameter gradients
        optimizer.zero_grad()

        # Forward pass
        outputs = model(inputs)
        
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % logger_freq == logger_freq - 1:  # Print every logger_freq mini-batches
            print(f"[{epoch + 1}, {i + 1}] loss: {running_loss / logger_freq}")
            running_loss = 0.0


# Train!
trainer.fit(model, dataloader)


```

### sort.py

```python
import json
import cv2
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset
from tqdm import tqdm

import sys
import os
sys.path.append("../rene")
from rene.utils.loaders import ReneDataset
os.makedirs('./training/lighting', exist_ok=1)

def sort():
    rene = ReneDataset(input_folder="/viscam/projects/scenegen/treehacks24/data/rene_dataset/")
    
    arr = []
    objn = rene._KEYS_TO_LOAD
    for object_key in objn:
        n = len(rene[object_key])
        for light_key in tqdm(range(n-1)):
            #if light_key == 3: exit(0)
            #for light_key2 in tqdm(range(n)):
                light_key2=(light_key+1)%n
                nn = len(rene[object_key][light_key])
                for camera_key in range(nn):
                    source = rene[object_key][light_key][camera_key]
                    target = rene[object_key][light_key2][camera_key]
                    prompt = str(object_key)
                    
                    srcf = str(source["pose"].__dict__["_file_sources"][0]).replace("pose.txt", "image.png")
                    tarf = str(target["pose"].__dict__["_file_sources"][0]).replace("pose.txt", "image.png'")
#                     1/0
#                     v = np.array([0,0,0,1])
#                     res = rene[object_key][light_key2][camera_key]["light"]()@v
#                     res = np.array([res[0]/res[3], res[1]/res[3],res[2]/res[3]])
                    res = rene[object_key][light_key2][camera_key]["light"]()[:,-1]
                    if res[0]<-1:
                        prompt+= " left"
                    elif res[0]>1:
                        prompt+= " middle"
                    else:
                        prompt+= " right"
                    if res[1]<-1:
                        prompt+= " bottom"
                    elif res[1]>1:
                        prompt+= " middle"
                    else:
                        prompt+= " top"
                    if res[2]<-1:
                        prompt+= " front"
                    elif res[2]>1:
                        prompt+= " middle"
                    else:
                        prompt+= " back"
#                     1/0
                        
#                     print("AHHHH",source, target)
#                     exit(0)
                    arr.append((tarf, tarf, prompt))
    return arr
def triplet_to_json(e):
    return {
        "source": e[0],
        "target": e[1],
        "prompt": e[2]
    }
arr = sort()
arr = [triplet_to_json(e) for e in arr]
with open('./training/lighting/prompts.json', 'w+') as f:
#     print(len(arr))
    
    for e in arr:
        f.write(e)
        f.write("\n")

```