# Project export: Geometric Linguistic Transformations This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: Discover and test reusable geometric directions for negation, tense, questions, and other linguistic transformations inside transformer embedding spaces. - Devpost: https://devpost.com/software/vectorlens - GitHub: https://github.com/cresta13/geometric-linguistic-transformations - Video: https://www.youtube.com/embed/XVqW4UGX1oM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Anna Simakova (20 commits) ## Devpost submission (written by the team) ### Inspiration Hi, I’m Anna. I’m thirty-six, I have worked in IT for about fourteen years, and this project began because one day I decided to apply for a master’s program in machine learning. It is one of those strong programs where you can receive a scholarship, but first you have to show that you are capable of doing something real. And then I remembered something. A very long time ago, in the distant year of 2012, I graduated from university and defended a thesis connected with Lie algebras. After that came fourteen years of ordinary IT life: systems, databases, architecture, documentation, code, teams, deadlines, production incidents, and approximately one million things that had very little to do with academic mathematics. I had forgotten a lot. Some ideas that I once understood now looked completely unfamiliar. I had almost no recent academic practice. I did not have a laboratory, a university position, a scientific team, or fifteen free years to study everything again from the beginning. So I asked GPT-5.6 a simple question: Can ideas from Lie algebra be useful for understanding transformers? I did not ask it to write a paper for me. At first, I only wanted to understand whether the question made any sense. That conversation became an experiment. The experiment became a repository. The repository became a growing independent research program. The original question was this: When a transformer changes a statement into a negation, a question, another tense, or another linguistic form, does that change leave a repeatable geometric trace inside the model? Not only a word such as “not”. Not only a question mark. Not only the final form of the sentence. But the transformation itself. The project was inspired by Lie algebra, but the current results do not prove that transformers contain a Lie algebra. That is still an open and much more difficult question. ### What it does The basic idea is simple. I take two related sentences: She is happy. She is not happy. A transformer converts each sentence into a numerical representation called an embedding. I then calculate the difference between the two embeddings: This delta is a vector: a direction and a distance inside the model’s embedding space. Then the research asks: Do examples of the same transformation move in similar directions? Can we distinguish negation from a question, a tense change, uncertainty, passive voice, or formalization? Does the pattern remain when the test sentences discuss completely different topics? Does the delta contain information that is not already present in the transformed sentence? Do different transformer architectures organize these transformations in similar ways? Can transformations be combined? Are the results real, or are we simply detecting surface clues such as particular words and punctuation? The repository is a working and reproducible research workflow. It contains: experiment scripts; controlled linguistic datasets; embedding extraction and analysis pipelines; source-only, target-only, concatenation, and delta baselines; semantic and harder holdout experiments; statistical controls; result tables and figures; reproducible report builders; draft research papers; reviewer-response notes; a dated research diary; positive, negative, and mixed results. The current evidence suggests that the relationship between two sentence embeddings can contain useful information about a linguistic transformation beyond the information found in either endpoint alone. But this is not presented as a finished theorem. Some results are strong. Some are mixed. Some beautiful early results turned out to be misleading. The repository preserves all of them. How I built it I want to be unusually honest about this part: I wrote very little of the code by hand. Codex wrote most of the implementation. I did not suddenly remember every mathematical concept, research method, Python library, and experimental technique that I had not used for fourteen years. Instead, I worked with Codex as a coding and research partner. I brought the question. I decided what I wanted to test. I chose which results looked important and which looked suspicious. I decided when an experiment was not strong enough, when another baseline was needed, and when a beautiful conclusion had to be removed. Codex turned those decisions into working code. It helped me: implement experiment pipelines; refactor research scripts; generate and process datasets; run model comparisons; add harder controls and ablations; calculate statistical results; build tables, figures, and reports; organize the repository; document the experiments; preserve a traceable history of the work. GPT-5.6 helped in a different way. It helped me return to mathematical ideas I had forgotten. It explained unfamiliar concepts in plain language. It helped me read papers paragraph by paragraph, compare my results with related work, formulate objections, and turn unclear questions into experiments that could actually be tested. But AI did not decide what the evidence meant. When a result looked almost perfect, I did not ask Codex to make a better chart. I asked it to try to break the result. Those additional experiments showed that some perfect syntax scores could be explained by surface clues in the transformed sentence. That changed the conclusion of the research. So the real workflow was not: AI generates code, and I accept the result. It was: I ask a question. Codex implements an experiment. We inspect the result. I challenge it. Codex implements another test. The claim becomes smaller, more careful, and hopefully more true. I remain responsible for the questions, decisions, interpretations, and any mistakes that are still present. The research foundation existed before OpenAI Build Week. During Build Week, I used Codex and GPT-5.6 to make the project easier to inspect, reproduce, test, and review, and to document clearly which parts existed before the event and which work was completed during the submission period. Challenges I ran into The first challenge was returning to research after fourteen years away from academic mathematics. I often did not understand a term in a paper. Sometimes I remembered the name of a concept but not how it worked. Sometimes I did not know whether my question was interesting, obvious, impossible, or simply wrong. AI made it possible to cross that first wall quickly, but understanding an explanation is not the same as proving that the explanation is correct. The second challenge was separating real geometric information from easy shortcuts. A model may appear to recognize negation because the transformed sentence contains the word “not”. It may recognize a question because of punctuation or sentence order. It may appear to discover a transformation while actually learning only the final form of the sentence. That is why the project compares delta vectors with the original sentence, the transformed sentence, and both endpoints together. The third challenge was resisting the desire to make the result sound larger than it really was. The project began with Lie algebra. That is an exciting idea. But an exciting origin does not mean that the final evidence proves algebraic structure. Some checks that initially looked impressive turned out to be mathematical or implementation identities rather than scientific evidence. Some hard-holdout experiments weakened the original claim. Some transformations behaved better than others. I kept these results instead of hiding them. The final challenge was working with AI itself. Codex can write code very quickly. GPT-5.6 can give a clear and convincing explanation. Neither speed nor confidence guarantees correctness. The generated code still has to be run. The results still have to be checked. References still have to be verified. A convincing interpretation still has to survive alternative explanations. Accomplishments that I am proud of I am proud that this became a real, open, reproducible research project instead of remaining one interesting conversation with GPT. The repository now contains experiments across multiple transformer architectures, progressively harder evaluation settings, statistical controls, reports, figures, and separate research directions. But I am most proud of the parts where the project proved itself wrong. One early experiment produced perfect classification results. It would have been very easy to place that number at the top of the README and call it a breakthrough. Instead, I asked for stronger baselines. The new tests showed that the endpoint sentence already contained enough surface information to solve much of the task. The perfect result was not evidence of deep geometric generalization. Finding that out made the project better. I am also proud that the repository reports boundary conditions where delta vectors do not win, and that the Lie-inspired experiments are described as diagnostics rather than as proof of a Lie algebra. A negative result is not a failed experiment when it prevents a false conclusion. And, personally, I am proud that at thirty-six, after fourteen years in industry, I returned to a type of work that I had almost decided no longer belonged to me. What I learned I learned that AI does not have to replace human thought. It can remove the wall that prevents a person from reaching the work in the first place. Without AI, returning to this field would have required months just to remember the vocabulary, find the right papers, understand unfamiliar methods, and learn how to implement every experiment. With AI, every unknown concept became a question I could ask immediately. A paper could be discussed one paragraph at a time. A vague idea could become a small experiment. A suspicious result could be attacked from several directions in one evening. This does not make universities, professors, experienced researchers, or peer review unnecessary. It does something else. It makes the entrance wider. My research may be imperfect. Some of its conclusions may change. The main hypothesis may eventually turn out to be unimportant, incorrect, or completely useless. That possibility does not make the attempt worthless. There are more than eight billion people on Earth. Imagine that one billion of them chose one scientific question they genuinely cared about and spent only two to four hours a week exploring it with the help of AI. Most of those experiments would not become famous papers. Many would fail. Some would repeat things that were already known. Some would be badly designed. Some, including mine, might lead nowhere. But a small part of them would find a new connection, notice an overlooked problem, test an unusual idea, improve an existing method, or solve something that makes life easier for millions of other people. One billion people contributing even a tiny amount of honest curiosity would create an amount of scientific attention that humanity has never had before. That is the most important idea behind this project. Not that everyone is suddenly a professor. Not that AI always knows the answer. But that many more people can now begin asking serious questions and making small, verifiable contributions. Science does not become weaker when more people are allowed to approach it. It becomes larger. What was completed during Build Week During OpenAI Build Week, the project did not become a separate demo app. Instead, Codex helped turn the existing research repository into a clearer, more testable research workflow and added a new behavior-level track called GLT-STEER. The main completed Build Week work was: a focused GPT-2 question activation-steering experiment; a no-steering base-rate control showing that GPT-2 does not naturally add question marks under the tested declarative prompts; an out-of-template control using freeform declarative sentences that were not part of the training templates; prompt-robustness checks across multiple prompt wrappers; a content-preservation audit asking whether steering only adds punctuation or also preserves source content; a copy-prompt follow-up showing that copy-like prompts preserve source content much better while still adding question form; a no-steering copy-prompt baseline showing that copy-like prompts alone do not produce question marks; a DistilGPT-2 replication showing that the effect transfers qualitatively but becomes much weaker; a first negation-steering attempt showing that the question-steering recipe does not automatically transfer to other transformations; a harder out-of-template question audit showing that the question effect survives structurally diverse sentences, although content preservation becomes harder; a delta-coherence diagnostic showing that GPT-2 question deltas are much more internally coherent than negation deltas, which helps explain why question steering works more cleanly under the current method; a full GPT-2 layer sweep showing that negation still does not become a clean steering target just by changing the intervention layer; an exclamation-mark control showing that final punctuation markers are much easier to steer than sentence-internal negation; a GPT-2 vs DistilGPT-2 question-delta norm diagnostic showing that DistilGPT-2 compresses later-layer question directions; a direct DistilGPT-2 layer/gain sweep showing that the earlier weak replication was parameter-sensitive rather than a hard model failure; public result summaries and CSV artifacts in the repository; cleanup of public repository structure so the visible repo remains a research artifact rather than a pile of submission notes. The most recent result is important because it narrows the claim in a useful way. The steering vector is not only causing GPT-2 to emit random question marks. Under copy-like prompts, it often preserves the source content and adds question form. But this works much better for prompt formats that already encourage copying or repeating the source sentence, and it remains weak for bare or quoted prompts. The follow-up no-steering audit is important: copy-like prompts by themselves produced 0.0000 question marks across 960 no-steering rows from GPT-2 and DistilGPT-2. This means the prompt explains source retention, but it does not explain the question-form effect. The DistilGPT-2 replication is useful but humbling. It preserves the qualitative target-versus-control separation, but the best question-and-preserved rate drops to 0.4625. That means the effect is not simply universal across GPT-family models; model dependence is now part of the honest story. The first negation-steering attempt is also important. It does not reproduce the question result. Under the same copy-like prompt design, negation target rows do not clearly beat matched controls. This means the project now has a sharper boundary: question steering works best so far, but other transformations need different prompts, metrics, or steering methods. The harder out-of-template audit is encouraging. It uses passive constructions, subordinate clauses, proper names, and numeric/time expressions. The question vector still produces question marks at rates between 0.7125 and 0.7500, while matched controls peak at 0.0375. The content-preserved rate is lower than in simpler sentences, which is exactly the honest boundary we should report. The newest diagnostic helps explain the question-versus-negation split. In GPT-2 hidden states, question transformation deltas are much more clustered than negation deltas across all layers. At the layers used for steering, question mean pairwise cosine is about 0.94-0.97, while negation is about 0.56. This suggests that the question vector is a cleaner intervention direction, while negation is geometrically more heterogeneous and probably harder for an autoregressive model to express by a single simple steering vector. The layer sweep strengthens that boundary: trying every GPT-2 layer did not reveal a clean negation intervention site. The best negation target-and-preserved rate reached only 0.1729, with controls still nontrivial. The exclamation control is a useful sanity check. A trivial final-marker transformation, statement -> statement!, steers very cleanly: up to 1.0000 exclamation-and-preserved in-template and 0.8000 on hard out-of-template sentences. This supports the idea that the question result is partly helped by the fact that ? is a final surface marker, while negation requires changing sentence-internal structure. The GPT-2 vs DistilGPT-2 diagnostic also explains part of the weaker DistilGPT-2 replication. DistilGPT-2 does not have smaller question deltas everywhere, but its later-layer question directions are much smaller than GPT-2's. At the final relative layer, the mean-norm ratio is 0.3341 and the centroid-norm ratio is 0.2923. The direct DistilGPT-2 layer/gain sweep revises that story. The earlier aggregate replication looked weak, but tuning the intervention site and gain recovers a strong setting: at gain=1.0, layer 2, and same_sentence prompts, DistilGPT-2 reaches question-and-preserved rate 0.8250, with matched controls at or below 0.0500. The same sweep also shows that gain matters: gain=0.5 is usable but weaker, while gain=1.5 over-steers and damages preservation. The current careful claim is: A GPT-2 question-transformation activation vector can steer output toward question form, and under copy-like prompts it can often preserve recognizable source content. Copy-like prompts alone do not create the question marks in the tested setup. DistilGPT-2 is also steerable, but it is much more sensitive to layer and gain: the best direct sweep setting reaches 0.8250 question-and-preserved, while over-strong gain collapses preservation. A first negation attempt and a full-layer negation sweep fail to produce a clean negation editor. A follow-up diagnostic suggests that negation deltas are much less geometrically coherent than question deltas, and an exclamation control shows that final punctuation markers are much easier to steer than sentence-internal transformations. A harder out-of-template audit supports generalization for question form but shows that content preservation gets harder as sentences become more complex. This is not yet robust general-purpose semantic editing. This is exactly the kind of result I want the project to preserve: positive, useful, but bounded. What is already done The current public repository contains: GLT-DV: delta-vector diagnostics with endpoint controls; GLT-SPOT: signed-permutation tests for ordered linguistic composition; GLT-MOLT: learned matrix/operator diagnostics and null controls; GLT-XFER: cross-model transfer and alignment stress tests; GLT-AFFECT: early affective-scale geometry controls; GLT-STEER: activation-steering experiments for GPT-2 and DistilGPT-2 question formation; requirements and reproducibility notes; result CSVs and experiment summaries; research drafts and roadmap files; Zenodo/citation metadata for archived research snapshots. ### What's next The immediate next step is criticism. I invite the judges, researchers, and developers who see this project to examine the methods, code, controls, conclusions, and negative results as critically as possible. I would rather discover a serious problem now than build a beautiful theory on top of a weak assumption. The technical research will continue with: stronger multi-seed validation; more difficult semantic controls; naturally written text instead of only controlled examples; additional modern transformer architectures; more languages; cross-model alignment; better tests of transformation composition; stricter semantic-preservation metrics for GLT-STEER; additional replication of question steering on another model family or another seed; redesigned steering tests for transformations beyond question formation, especially negation, tense, and modality; experiments that test whether geometric directions can change model behavior, not only classify transformations. The longer-term question is whether similar methods can describe transformations connected not only with grammar, but also with uncertainty, attitude, emotion, intensity, and subjective experience. Perhaps one day models could be trained and controlled using richer representations of how meaning changes, rather than only through words, labels, and examples. Perhaps this research will contribute to that. Perhaps it will only show that one particular approach does not work. Both outcomes are useful when they are tested honestly and shared openly. This project began with one slightly unreasonable question. I hope it encourages someone else to ask theirs. Anna, thirty-six, with love for everything new and not yet explained. ## README (from the GitHub repository) # IF YOU DO NOT HAVE A BACKGROUND IN MATHEMATICS OR AI, START HERE Modern artificial intelligence systems are becoming increasingly capable. They write texts and computer programs, reason, make decisions, and play an ever-growing role in our lives. Yet their internal workings still remain, to a large extent, a black box. We know how to build neural networks and how to train them. But we still do not fully understand **what structures emerge inside them during training, how they represent knowledge, or which internal rules govern the transformation of information**. A biologist studies living organisms. A geologist studies the Earth and its rocks. This project studies neural models from the inside. It searches for the geometry, algebra, and regularities of their internal representations — or perhaps for something that humanity does not yet have the mathematical language to describe. Until we understand the internal structure of the systems we create, we are like blind kittens trying to defuse a bomb: we already know how to make the device increasingly powerful, but we still do not understand its inner mechanism well enough. This project is an attempt to take at least one more step from observing the behaviour of artificial intelligence toward understanding **how it actually works on the inside**. --- # Geometric Linguistic Transformations Research code, result tables, figures, and draft papers for **GLT** (**Geometric Linguistic Transformations**): an independent research program for testing whether linguistic transformations appear as reusable geometric objects in transformer embedding spaces. This is an active and reproducible research repository. It is not presented as a finished theorem or a final paper, but as a working experimental pipeline, an evidence package, and an open scientific question. ## 1. Plain-Language Overview Modern language models turn sentences into high-dimensional vectors. This project asks a simple question: > If one sentence is changed into another sentence, does the vector movement between them describe the kind of linguistic change that happened? For example: ```text source sentence -> transformed sentence statement -> question present tense -> future tense plain claim -> uncertain claim positive claim -> negated claim ``` For each pair, we compute: ```text delta = embedding(transformed sentence) - embedding(source sentence) ``` Then we test whether these deltas behave like meaningful transformation objects rather than arbitrary differences between two sentences. ### What We Have Found So Far The current evidence supports a cautious version of the idea: - Sentence-pair deltas often contain information about the **type of transformation**. - In the main full-semantic experiments, deltas beat target-only baselines across several models under Linear SVC probes. - In harder sentence-pair holdouts such as UPAT, endpoint features can dominate. This suggests that delta geometry captures transformation type better than absolute transformation identity. - Some ordered transformations show structured composition effects, but this is not a proof of a Lie algebra. - Simple additive deltas predict target embeddings better than learned linear/affine maps, while learned maps are useful for algebraic closure diagnostics. - A first affective-scale experiment suggests that text embeddings treat love/hate-like polarity as curved affective-involvement geometry rather than a simple opposite-axis scale, but this remains a text-only lexical-geometry result and still needs stricter neutral-word controls before promotion. ### What Is Not Claimed This repository does not claim: - that transformer embeddings contain a complete linguistic algebra; - that the reported effects are independent of endpoint wording; - that syntax-holdout `1.0` results prove deep generalization; - that antisymmetry checks are scientific evidence; - that text-only affect geometry is grounded emotional experience; - that this is a submission-ready paper. The project is deliberately conservative: positive results, failures, endpoint leakage, and hard-holdout boundaries are all kept in the record. ## 2. Quick Demo The fastest demo is the GLT-STEER question experiment. It shows a simple behavior-level result: ```text Without steering: GPT-2 repeats a statement. With a question-transformation vector: GPT-2 starts producing question marks. With random or wrong-class vectors: the question-mark effect does not appear. ``` Run the lightweight demo from the repository root: ```powershell .\.venv\Scripts\python.exe scripts\show_glt_steer_demo.py ``` This command does not download models or rerun the experiment. It reads the archived CSV in: ```text results/experiments/gpt2_question_activation_steering_focused_20260714_results/ ``` Headline result: | condition | question mark rate | |---|---:| | target question vector | `0.9350` | | random-norm control | `0.0000` | | wrong-class control | `0.0000` | | negative-target control | `0.0000` | Stronger copy-prompt preservation result: | condition | question-and-preserved rate | |---|---:| | target question vector, copy-like prompts | up to `0.9750` | | wrong-vector / no-steering controls | `0.0000` in the matched headline rows | Prompt-only control: | condition | question mark rate | |---|---:| | copy-like prompts without steering, GPT-2 + DistilGPT-2 | `0.0000` across `960` no-steering rows | Full notes: - `results/experiments/gpt2_question_activation_steering_focused_20260714_results/SUMMARY.md` - `results/experiments/gpt2_question_steering_controls_20260714_results/SUMMARY.md` - `results/experiments/gpt2_question_prompt_robustness_20260715_results/SUMMARY.md` - `results/experiments/gpt2_question_copy_prompt_preservation_20260716_results/SUMMARY.md` - `results/experiments/distilgpt2_question_copy_prompt_preservation_20260716_results/SUMMARY.md` - `results/experiments/question_copy_prompt_none_baseline_20260716_results/SUMMARY.md` - `results/experiments/gpt2_steering_delta_coherence_20260716_results/SUMMARY.md` Safe interpretation: > This is evidence that a question-transformation activation vector can steer GPT-2 toward question-like output form. It is not evidence that semantic editing is solved, and it is not proof of a complete linguistic algebra. ## 3. Technical Overview GLT is organized by current publication priority. Track 1 / GLT-STEER is the active short-paper target; the other tracks are diagnostic or future-paper candidates. ### Track 1 / GLT-STEER: Activation Steering Question: > Can transformation vectors do behavior-level work inside a generative model? Main evidence: - Central GLT-STEER interpretation: **final-position surface markers** such as `?`, `!`, and `...` are reliably steerable in GPT-2 using mean hidden-state delta vectors, while lexical or sentence-internal transformations such as negation and modality do not work under the same recipe. - A broad GPT-2/DistilGPT-2 pilot found that question steering was the clearest target for a focused rerun. - The focused GPT-2 question-steering run completed `6800` generations with no failures. - At layer `2`, gain `0.75`, target question steering produced question marks in `93.75%` of generations. - Across all tested layers at gain `0.75`, target question steering produced question marks in `93.50%` of generations. - Random-norm, wrong-class, and negative-target controls produced `0.00%` question marks in the aggregate control summary. - A follow-up control found no-steering question-mark base rate `0.0000` for both in-template and out-of-template prompts. - The same question vector transferred to 40 freeform out-of-template declarative sentences, producing question marks at rate `0.8375` while all compact controls stayed at `0.0000`. - Prompt robustness controls show that the effect survives four prompt styles, with target question-mark rate `0.7750-0.9875` in-template and `0.8125-0.9250` out-of-template. - Cont [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 118 recognized source files, 1076 KB. - Hugging Face (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 1517) ``` .gitignore .zenodo.json BUILD_WEEK_2026.md CITATION.cff docs/hackathon-build/demo-steering.md docs/hackathon-build/project-story.md docs/hackathon-build/README.md HACKATHON_SUBMISSION.md LICENSE paper/articles/geometric-transformation-vectors/draft.md paper/articles/geometric-transformation-vectors/README.md paper/articles/glt-steer-activation-editors/draft.md paper/articles/glt-steer-activation-editors/README.md paper/articles/lie-style-linguistic-operators/draft.md paper/articles/lie-style-linguistic-operators/README.md paper/articles/README.md paper/related_work_positioning.md paper/research_program.md paper/research_roadmap.md paper/revision_notes_round3.md README.md reports/README.md reports/release_notes_v2026.06.13.md reports/release_notes_v2026.06.24.md requirements.txt research/diary.md research/PROTOCOL.md research/research_state_2026-06-14.md results/ablation_control_table.csv results/ablation_multiseed_aggregated.csv results/ablation_multiseed_mcnemar.csv results/ablation_multiseed_summary.csv results/ablation_y_vs_delta_summary.csv results/confusion_class_recall_ranked.csv results/confusion_class_recall.csv results/confusion_negation_summary.csv results/confusion_top_errors.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v2_gain05_20260717_results/csv/transformation_copy_prompt_raw.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v2_gain05_20260717_results/csv/transformation_copy_prompt_sources.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v2_gain05_20260717_results/csv/transformation_copy_prompt_summary.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v2_gain05_20260717_results/run_status.json results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v2_gain05_20260717_results/SUMMARY.md results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain10_20260717_results/csv/distilgpt2_gain_layer_sweep_aggregate.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain10_20260717_results/csv/transformation_copy_prompt_raw.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain10_20260717_results/csv/transformation_copy_prompt_sources.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain10_20260717_results/csv/transformation_copy_prompt_summary.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain10_20260717_results/run_status.json results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain10_20260717_results/SUMMARY.md results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain15_20260717_results/csv/transformation_copy_prompt_raw.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain15_20260717_results/csv/transformation_copy_prompt_sources.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain15_20260717_results/csv/transformation_copy_prompt_summary.csv results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain15_20260717_results/run_status.json results/experiments/distilgpt2_question_copy_prompt_layer_gain_sweep_v3_gain15_20260717_results/SUMMARY.md results/experiments/distilgpt2_question_copy_prompt_preservation_20260716_results/csv/question_copy_prompt_raw.csv results/experiments/distilgpt2_question_copy_prompt_preservation_20260716_results/csv/question_copy_prompt_sources.csv results/experiments/distilgpt2_question_copy_prompt_preservation_20260716_results/csv/question_copy_prompt_summary.csv results/experiments/distilgpt2_question_copy_prompt_preservation_20260716_results/run_status.json results/experiments/distilgpt2_question_copy_prompt_preservation_20260716_results/SUMMARY.md results/experiments/distilgpt2_question_hard_oot_best_layer2_gain10_20260801_results/csv/distilgpt2_hard_oot_question_preservation_summary.csv results/experiments/distilgpt2_question_hard_oot_best_layer2_gain10_20260801_results/csv/transformation_copy_prompt_raw.csv results/experiments/distilgpt2_question_hard_oot_best_layer2_gain10_20260801_results/csv/transformation_copy_prompt_sources.csv results/experiments/distilgpt2_question_hard_oot_best_layer2_gain10_20260801_results/csv/transformation_copy_prompt_summary.csv results/experiments/distilgpt2_question_hard_oot_best_layer2_gain10_20260801_results/run_status.json results/experiments/distilgpt2_question_hard_oot_best_layer2_gain10_20260801_results/SUMMARY.md results/experiments/full_semantic_pooling_ablation_results/full_semantic_pooling_ablation_pivot.csv results/experiments/full_semantic_pooling_ablation_results/full_semantic_pooling_ablation.csv results/experiments/full_semantic_pooling_ablation_results/prompts.csv results/experiments/full_semantic_pooling_ablation_results/transformation_pairs.csv results/experiments/glt_affect_lexical_contrast_bootstrap_results/csv/lexical_contrast_bootstrap_summary.csv results/experiments/glt_affect_lexical_contrast_bootstrap_results/csv/lexical_contrast_cells.csv results/experiments/glt_affect_lexical_contrast_bootstrap_results/run_config.json results/experiments/glt_affect_lexical_contrast_bootstrap_results/run_status.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/BAAI__bge_m3.done.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/bert_base_multilingual_cased.done.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/distilbert_base_multilingual_cased.done.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/intfloat__multilingual_e5_large.done.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/sentence_transformers__LaBSE.done.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/sentence_transformers__paraphrase_multilingual_mpnet_base_v2.done.json results/experiments/glt_affect_lexical_specificity_control_results/checkpoints/xlm_roberta_base.done.json results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_all_models.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_BAAI__bge_m3.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_bert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_distilbert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_intfloat__multilingual_e5_large.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_sentence_transformers__LaBSE.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_sentence_transformers__paraphrase_multilingual_mpnet_base_v2.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_summary.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_curvature_xlm_roberta_base.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_all_models.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_BAAI__bge_m3.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_bert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_distilbert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_intfloat__multilingual_e5_large.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_sentence_transformers__LaBSE.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_sentence_transformers__paraphrase_multilingual_mpnet_base_v2.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_audit_xlm_roberta_base.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_marker_span_summary.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_all_models.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_BAAI__bge_m3.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_bert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_distilbert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_intfloat__multilingual_e5_large.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_sentence_transformers__LaBSE.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_sentence_transformers__paraphrase_multilingual_mpnet_base_v2.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_summary.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_opposition_xlm_roberta_base.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_specificity_dataset.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_all_models.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_BAAI__bge_m3.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_bert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_distilbert_base_multilingual_cased.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_intfloat__multilingual_e5_large.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_sentence_transformers__LaBSE.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_sentence_transformers__paraphrase_multilingual_mpnet_base_v2.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_summary.csv results/experiments/glt_affect_lexical_specificity_control_results/csv/lexical_steps_xlm_roberta_base.csv results/experiments/glt_affect_lexical_specificity_control_results/run_config.json results/experiments/glt_affect_lexical_specificity_control_results/run_status.json results/experiments/glt_affect_marker_pool_control_results/checkpoints/BAAI__bge_m3.done.json results/experiments/glt_affect_marker_pool_control_results/checkpoints/bert_base_multilingual_cased.done.json [1397 more files omitted for size] ``` ### Dependencies - requirements.txt: matplotlib@==3.10.9, numpy@==2.4.6, pandas@==3.0.3, pillow@==12.2.0, python-dotenv@==1.2.2, scikit-learn@==1.9.0, scipy@==1.17.1, sentencepiece@==0.2.1, tokenizers@==0.22.2, torch@==2.12.0, transformers@==5.11.0 ### Recent commits (newest first) - Refine GLT-STEER final marker narrative - Add question modality composition steering audit - Add marker composition steering results - Add ellipsis steering control - Add DistilGPT2 hard OOT steering audit - Add DistilGPT2 steering gain sweep - Add steering boundary diagnostics - Add steering delta coherence diagnostic - Add hard out-of-template steering audit - Allow custom steering source sets - Add negation steering boundary result - Add generic copy-prompt steering runner - Add DistilGPT2 steering baseline audit - Highlight steering preservation metric - Restore Build Week judge notes - Clarify Build Week steering work - Update steering preservation results and trim public notes - Document Build Week submission positioning - Add GPT-2 steering preservation audits - Add GPT-2 prompt robustness result ## Key source files (fetched from GitHub, selected and truncated for size) ### BUILD_WEEK_2026.md ```markdown # OpenAI Build Week 2026 Provenance This file documents what existed before OpenAI Build Week 2026, what changed during the event, and how Codex and GPT-5.6 were used. ## Submission Identity Project title: ```text Geometric Linguistic Transformations ``` Primary category: ```text Developer Tools ``` Secondary fit: ```text Education ``` Elevator pitch: ```text A reproducible investigation into whether linguistic transformations form geometric directions inside transformers, and whether AI can make serious independent science accessible again. ``` ## Working Project The submitted project is a reproducible research workflow, not a separate web app. It includes: 1. Synthetic controlled linguistic transformation datasets. 2. Scripts for extracting and analyzing transformer representations. 3. Delta-vector, endpoint, concat, source-only, and target-only controls. 4. Semantic and hard holdout experiments. 5. Shuffle, norm-matched, spectral, endpoint, and random-label controls. 6. Composition and learned-operator diagnostics. 7. GPT-2 activation-steering experiments. 8. CSV result tables, summaries, drafts, reports, and reproducibility metadata. 9. A lightweight judge demo that reads committed results. Judge command: ```powershell .\.venv\Scripts\python.exe scripts\show_glt_steer_demo.py ``` ## What Existed Before Build Week Before Build Week, the repository already contained the main GLT research program: - GLT-DV: delta-vector diagnostics with endpoint controls. - GLT-SPOT: signed-permutation composition tests. - GLT-MOLT: learned matrix/operator audits. - GLT-XFER: cross-model transfer and alignment stress tests. - GLT-AFFECT: early graded affective-geometry experiments. - `paper/`, `reports/`, `results/`, `research/`, and `scripts/` as the core research structure. - Zenodo archived snapshots and citation metadata. ## What Changed During Build Week Build Week added and packaged **GLT-STEER**, the behavior-level intervention track: - focused GPT-2 question activation steering; - base-rate controls for question-mark generation; - out-of-template freeform controls; - prompt-wrapper robustness controls; - content-preservation audit; - copy-prompt preservation follow-up; - copy-prompt no-steering baseline audit; - DistilGPT-2 copy-prompt replication; - hard out-of-template question generalization audit; - a lightweight demo command for judges; - public submission and provenance notes. The steering result was strengthened step by step: 1. A target question vector produced question marks while random, wrong-class, and negative controls did not. 2. No-steering base-rate controls showed that GPT-2 did not naturally add question marks under the tested declarative prompts. 3. Out-of-template prompts showed the effect was not restricted to the original template sentences. 4. Prompt-robustness tests showed the effect survived several prompt wrappers. 5. Content-preservation tests showed an important boundary: copy-like prompts preserve sour [truncated — 3690 more characters] ``` ### HACKATHON_SUBMISSION.md ```markdown # OpenAI Build Week Submission Notes This file is the public judge-facing submission guide for **OpenAI Build Week 2026**. It is intentionally practical: what the project is, how to test it, what changed during Build Week, and what to show in the demo video. ## Project **Geometric Linguistic Transformations** Short description: > A reproducible investigation into whether linguistic transformations form geometric directions inside transformers, and whether AI can make serious independent science accessible again. Recommended Devpost category: ```text Developer Tools ``` Secondary fit: ```text Education ``` Why Developer Tools: This is a runnable research workflow: scripts, controls, result tables, summaries, reproducibility metadata, and a lightweight judge demo. It is not a separate web app and not a polished final paper. ## Repository URL ```text https://github.com/cresta13/geometric-linguistic-transformations ``` ## Judge Quick Test From the repository root: ```powershell .\.venv\Scripts\python.exe scripts\show_glt_steer_demo.py ``` This command does not download models or rerun heavy experiments. It reads committed CSV results and prints: - the focused GPT-2 question-steering result; - no-steering question-mark base rates; - out-of-template generalization controls; - prompt-robustness controls. ## What Was Built With Codex and GPT-5.6 During Build Week, the main completed addition was **GLT-STEER**: - GPT-2 question activation steering; - no-steering base-rate controls; - out-of-template freeform source controls; - prompt-wrapper robustness checks; - content-preservation audit; - copy-prompt preservation follow-up; - public result summaries and CSV artifacts; - documentation that makes the result inspectable by judges. Codex was the primary implementation environment for scripts, result processing, debugging, summaries, and repository organization. GPT-5.6 was used as a reasoning partner for mathematical background, related-work reading, and turning vague objections into concrete controls. The human role remained to choose the research direction, decide which claims were honest enough to keep, interpret results, and stay responsible for remaining mistakes. ## Main Result to Show The clearest visible demo is GPT-2 question steering. ```text Source: The cat sat on the mat. No steering: The cat sat on the mat. The cat sat on the mat... Target question steering: The cat sat on the mat. The cat sat on the mat? The cat sat on the mat? Controls: Random, wrong-class, and negative vectors do not produce the same question-mark effect. ``` Concise numbers: ```text Focused GPT-2 question steering: target question vector question-mark rate: 0.9350 random_norm / wrong_class / negative_target: 0.0000 Base-rate control: no-steering question-mark base rate: 0.0000 Out-of-template control: target question vector: 0.8375 all compact controls: 0.0000 Prompt robustness: target question vector stays strong across four prompt style [truncated — 5619 more characters] ``` ### requirements.txt ``` matplotlib==3.10.9 numpy==2.4.6 pandas==3.0.3 pillow==12.2.0 python-dotenv==1.2.2 scikit-learn==1.9.0 scipy==1.17.1 sentencepiece==0.2.1 tokenizers==0.22.2 torch==2.12.0 transformers==5.11.0 ``` ### scripts/main.py ```python import os import gc import numpy as np import pandas as pd import torch import matplotlib.pyplot as plt from dotenv import load_dotenv from transformers import AutoTokenizer, AutoModel load_dotenv() MODELS = os.getenv( "LIE_MODELS", "bert-base-uncased,distilgpt2,gpt2" ).split(",") HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("HUGGINGFACE_API_KEY") OUT_DIR = "results/experiments/lie_llm_compare_results" os.makedirs(OUT_DIR, exist_ok=True) PROMPTS = [ "The scientist discovered a new principle of symmetry.", "The mathematician discovered a new principle of symmetry.", "The scientist did not discover a new principle of symmetry.", "A child found a small black cat in the garden.", "A child did not find a small black cat in the garden.", "The government announced a new economic policy.", "The poet wrote a beautiful poem about memory.", "The engineer designed a machine that predicts weather.", "If the theory is correct, the experiment will confirm it.", "If the theory is wrong, the experiment will disprove it.", ] PAIRS = [ (0, 1, "scientist → mathematician"), (0, 2, "affirmative → negative"), (3, 4, "found → did not find"), (8, 9, "correct → wrong"), ] def mean_pool(hidden_state, attention_mask): mask = attention_mask.unsqueeze(-1).float() return (hidden_state * mask).sum(dim=1) / mask.sum(dim=1).clamp_min(1e-9) def cosine(a, b): return float(np.dot(a, b) / ((np.linalg.norm(a) * np.linalg.norm(b)) + 1e-12)) def analyze_model(model_name): model_name = model_name.strip() print(f"\n=== MODEL: {model_name} ===") tokenizer = AutoTokenizer.from_pretrained(model_name, token=HF_TOKEN) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token or tokenizer.unk_token model = AutoModel.from_pretrained( model_name, output_hidden_states=True, token=HF_TOKEN, ) model.eval() inputs = tokenizer( PROMPTS, padding=True, truncation=True, max_length=128, return_tensors="pt", ) with torch.no_grad(): outputs = model(**inputs) hidden_states = outputs.hidden_states layer_vectors = [] for h in hidden_states: pooled = mean_pool(h, inputs["attention_mask"]) layer_vectors.append(pooled.cpu().numpy()) traj = np.stack(layer_vectors, axis=1) trajectory_rows = [] for i, prompt in enumerate(PROMPTS): vectors = traj[i] transitions = vectors[1:] - vectors[:-1] norms = np.linalg.norm(transitions, axis=1) adjacent_cosines = [ cosine(transitions[j], transitions[j + 1]) for j in range(len(transitions) - 1) ] path_length = float(norms.sum()) displacement = float(np.linalg.norm(vectors[-1] - vectors[0])) efficiency = displacement / (path_length + 1e-12) trajectory_rows.append({ "model": model_name, "prompt_id": i, "prompt": prompt, "num_layers": len(hidden_states), "hidden_dim": traj.shape[-1], "path_length": path_length, "displacement": displacement, "trajectory_efficiency": efficiency, "mean_transition_norm": float(norms.mean()), "mean_adjacent_cosine": float(np.mean(adjacent_cosines)), "curvature_proxy_1_minus_cos": float(np.mean([1 - c for c in adjacent_cosines])), }) pair_rows = [] for a, b, label in PAIRS: layer_distances = np.linalg.norm(traj[a] - traj[b], axis=1) trans_a = traj[a, 1:] - traj[a, :-1] trans_b = traj[b, 1:] - traj[b, :-1] transition_cosines = [ cosine(trans_a[j], trans_b[j]) for j in range(trans_a.shape[0]) ] pair_rows.append({ "model": model_name, "pair": label, "mean_layer_distance": float(layer_distances.mean()), "final_state_distance": float(layer_distances[-1]), "mean_transition_cosine": float(np.mean(transition_cosines)), }) del model gc.collect() return pd.DataFrame(trajectory_rows), pd.DataFrame(pair_rows) def main(): all_traj = [] all_pairs = [] for model_name in MODELS: try: traj_df, pair_df = analyze_model(model_name) all_traj.append(traj_df) all_pairs.append(pair_df) except Exception as e: print(f"FAILED: {model_name}") print(e) trajectory_df = pd.concat(all_traj, ignore_index=True) pair_df = pd.concat(all_pairs, ignore_index=True) trajectory_df.to_csv(f"{OUT_DIR}/all_trajectory_metrics.csv", index=False) pair_df.to_csv(f"{OUT_DIR}/all_pair_metrics.csv", index=False) summary = trajectory_df.groupby("model").agg({ "num_layers": "first", "hidden_dim": "first", "path_length": "mean", "displacement": "mean", "trajectory_efficiency": "mean", "mean_adjacent_cosine": "mean", "curvature_proxy_1_minus_cos": "mean", }).reset_index() summary.to_csv(f"{OUT_DIR}/model_summary.csv", index=False) print("\n=== MODEL SUMMARY ===") print(summary) print("\n=== PAIR METRICS ===") print(pair_df) plt.figure(figsize=(10, 5)) plt.bar(summary["model"], summary["curvature_proxy_1_minus_cos"]) plt.xticks(rotation=30, ha="right") plt.ylabel("Mean curvature proxy") plt.title("Trajectory curvature proxy by model") plt.tight_layout() plt.savefig(f"{OUT_DIR}/model_curvature_comparison.png", dpi=180) plt.close() pivot = pair_df.pivot( index="pair", columns="model", values="mean_transition_cosine", ) pivot.plot(kind="bar", figsize=(12, 6)) plt.ylabel("Mean transition cosine") plt.title("Prompt-pair transition similarity by model") plt.xticks(rotation=30, ha="right") plt.tight_layout() [truncated — 157 more characters] ``` ### scripts/build_track1_effect_intervals.py ```python from pathlib import Path import pandas as pd ROOT = Path(__file__).resolve().parents[1] IN_PATH = ROOT / "results" / "ablation_multiseed_summary.csv" OUT_PATH = ROOT / "results" / "track1_multiseed_effect_intervals.csv" # Two-sided 95% t critical value for n=5 seeds, df=4. T_CRIT_95_DF4 = 2.7764451051977987 def main(): df = pd.read_csv(IN_PATH) pivot = ( df.pivot_table( index=["seed", "model", "classifier"], columns="representation", values="accuracy", ) .reset_index() .rename_axis(None, axis=1) ) pivot["delta_minus_y_only"] = pivot["delta"] - pivot["y_only"] pivot["delta_minus_concat"] = pivot["delta"] - pivot["concat"] rows = [] for (model, classifier), sub in pivot.groupby(["model", "classifier"], sort=True): for effect in ["delta_minus_y_only", "delta_minus_concat"]: values = sub[effect] n = int(values.count()) mean = float(values.mean()) std = float(values.std(ddof=1)) half_width = float(T_CRIT_95_DF4 * std / (n ** 0.5)) rows.append( { "model": model, "classifier": classifier, "effect": effect, "mean": mean, "std": std, "n_seeds": n, "ci95_low": mean - half_width, "ci95_high": mean + half_width, "all_seed_effects_positive": bool((values > 0).all()), } ) out = pd.DataFrame(rows) OUT_PATH.parent.mkdir(parents=True, exist_ok=True) out.to_csv(OUT_PATH, index=False) print(OUT_PATH) if __name__ == "__main__": main() ``` ### scripts/build_signed_permutation_multiple_testing.py ```python from pathlib import Path import numpy as np import pandas as pd ROOT = Path(__file__).resolve().parents[1] EXP = ROOT / "results" / "experiments" OUT = EXP / "lie_algebraic_identities_results" / "csv" def benjamini_hochberg(p_values): p = np.asarray(p_values, dtype=float) n = len(p) order = np.argsort(p) ranked = p[order] adj = np.empty(n, dtype=float) prev = 1.0 for i in range(n - 1, -1, -1): rank = i + 1 val = min(prev, ranked[i] * n / rank) prev = val adj[order[i]] = val return np.clip(adj, 0, 1) def bootstrap_p_less_than_one(values, n_boot=20000, seed=42): rng = np.random.default_rng(seed) values = np.asarray(values, dtype=float) idx = rng.integers(0, len(values), size=(n_boot, len(values))) means = values[idx].mean(axis=1) p = (np.sum(means >= 1.0) + 1) / (n_boot + 1) return float(p) def main(): files = [ EXP / "lie_algebraic_identities_results" / "csv" / "jacobi_raw_all_models.csv", EXP / "lie_algebraic_identities_decoder_results" / "csv" / "jacobi_raw_all_models.csv", ] raw = pd.concat([pd.read_csv(path) for path in files if path.exists()], ignore_index=True) rows = [] for (model, triple), group in raw.groupby(["model", "triple"]): ratios = group["jacobi_to_null_mean_ratio"].to_numpy() mean_ratio = float(ratios.mean()) p_less = bootstrap_p_less_than_one(ratios) rows.append({ "model": model, "triple": triple, "n": int(len(group)), "mean_ratio_to_null": mean_ratio, "bootstrap_p_ratio_less_than_1": p_less, "direction": "below_null" if mean_ratio < 1 else "above_null", }) result = pd.DataFrame(rows).sort_values(["bootstrap_p_ratio_less_than_1", "model", "triple"]) result["bonferroni_p"] = np.minimum(result["bootstrap_p_ratio_less_than_1"] * len(result), 1.0) result["bh_fdr_p"] = benjamini_hochberg(result["bootstrap_p_ratio_less_than_1"].to_numpy()) result["passes_bonferroni_0_05"] = result["bonferroni_p"] < 0.05 result["passes_bh_fdr_0_05"] = result["bh_fdr_p"] < 0.05 out_path = OUT / "signed_permutation_multiple_testing.csv" result.to_csv(out_path, index=False) print(out_path) print(result) if __name__ == "__main__": main() ``` ### scripts/run_glt_molt_spectral_pca_sweep.py ```python from __future__ import annotations import json import os import subprocess import sys import time from pathlib import Path ROOT = Path(__file__).resolve().parents[1] SCRIPT = ROOT / "scripts" / "run_glt_molt_spectral_nulls.py" OUT_ROOT = Path(os.getenv( "GLT_MOLT_PCA_SWEEP_OUT_ROOT", "results/experiments/glt_molt_spectral_pca_sweep_9m_160t_a100_300null_g256_results", )) def parse_list(name: str, default: str) -> list[str]: raw = os.getenv(name, default) return [item.strip() for item in raw.split(",") if item.strip()] def main() -> None: pca_dims = parse_list("GLT_MOLT_PCA_SWEEP_DIMS", "64,128,256") alphas = os.getenv("GLT_MOLT_PCA_SWEEP_ALPHAS", "100") nulls = os.getenv("GLT_MOLT_PCA_SWEEP_NULLS", "300") givens = os.getenv("GLT_MOLT_PCA_SWEEP_GIVENS", "256") templates = os.getenv("GLT_MOLT_PCA_SWEEP_TEMPLATES_PER_LANGUAGE", "160") OUT_ROOT.mkdir(parents=True, exist_ok=True) status_path = OUT_ROOT / "pca_sweep_status.json" runs = [] print("GLT-MOLT SPECTRAL PCA SWEEP", flush=True) print(json.dumps({ "out_root": str(OUT_ROOT), "pca_dims": pca_dims, "alphas": alphas, "nulls": nulls, "givens": givens, "templates_per_language": templates, }, indent=2), flush=True) for dim in pca_dims: out_dir = OUT_ROOT / f"pca_{dim}" env = os.environ.copy() env["GLT_MOLT_SPECTRAL_OUT_DIR"] = str(out_dir) env["GLT_MOLT_SPECTRAL_ALPHAS"] = alphas env["GLT_MOLT_SPECTRAL_NULLS"] = nulls env["GLT_MOLT_SPECTRAL_GIVENS"] = givens env["GLT_MOLT_TEMPLATES_PER_LANGUAGE"] = templates env["GLT_MOLT_PCA_DIM"] = dim print(f"\n=== PCA DIM {dim} ===", flush=True) started = time.ctime() code = subprocess.call([sys.executable, str(SCRIPT)], cwd=str(ROOT), env=env) run = { "pca_dim": int(dim), "out_dir": str(out_dir), "started_at": started, "finished_at": time.ctime(), "return_code": code, } runs.append(run) status_path.write_text(json.dumps({"runs": runs}, indent=2), encoding="utf-8") if code != 0: raise SystemExit(code) status_path.write_text(json.dumps({"runs": runs, "finished_at": time.ctime()}, indent=2), encoding="utf-8") print("DONE", flush=True) if __name__ == "__main__": main() ``` ### scripts/run_track1_spotcheck.py ```python import gc import os from pathlib import Path import numpy as np import pandas as pd from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.svm import LinearSVC import lie_llm_full_semantic_holdout_experiment as full OUT_DIR = Path(os.getenv("SPOTCHECK_OUT_DIR", "results/experiments/track1_spotcheck_results")) OUT_DIR.mkdir(parents=True, exist_ok=True) MODELS = [m.strip() for m in os.getenv("SPOTCHECK_MODELS", "bert-large-uncased").split(",") if m.strip()] N_BASE = int(os.getenv("SPOTCHECK_N_BASE", "100")) def run_classifiers(X, labels, split): train_mask = split == "train" test_mask = split == "test" classifiers = { "logreg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000, class_weight="balanced", n_jobs=-1)), "linear_svc": make_pipeline(StandardScaler(), LinearSVC(class_weight="balanced", max_iter=20000)), } rows = [] for clf_name, clf in classifiers.items(): clf.fit(X[train_mask], labels[train_mask]) pred = clf.predict(X[test_mask]) rows.append({ "classifier": clf_name, "accuracy": float(accuracy_score(labels[test_mask], pred)), "n_train": int(train_mask.sum()), "n_test": int(test_mask.sum()), }) return rows def main(): full.OUT_DIR = str(OUT_DIR) full.N_BASE = N_BASE os.makedirs(full.OUT_DIR, exist_ok=True) pair_df = full.build_dataset() prompts, pair_df = full.index_prompts(pair_df) pair_df.to_csv(OUT_DIR / "transformation_pairs.csv", index=False) pd.DataFrame({"prompt_id": range(len(prompts)), "prompt": prompts}).to_csv(OUT_DIR / "prompts.csv", index=False) labels = pair_df["class"].astype(str).to_numpy() split = pair_df["split"].astype(str).to_numpy() src_idx = pair_df["source_idx"].to_numpy() tgt_idx = pair_df["target_idx"].to_numpy() rows = [] for model_name in MODELS: print(f"\n=== MODEL: {model_name} ===") vectors = full.get_vectors(model_name, prompts) x_src = vectors[src_idx] x_tgt = vectors[tgt_idx] reps = { "x_only": x_src, "y_only": x_tgt, "delta": x_tgt - x_src, "concat": np.concatenate([x_src, x_tgt], axis=1), } for rep_name, X in reps.items(): for row in run_classifiers(X, labels, split): row.update({ "model": model_name, "representation": rep_name, "random_baseline": 1 / len(set(labels)), "n_base": N_BASE, }) print(model_name, rep_name, row["classifier"], f"{row['accuracy']:.4f}") rows.append(row) del vectors, reps gc.collect() result = pd.DataFrame(rows) result.to_csv(OUT_DIR / "spotcheck_representation_ablation.csv", index=False) pivot = result.pivot_table(index=["model", "classifier"], columns="representation", values="accuracy").reset_index() pivot.to_csv(OUT_DIR / "spotcheck_representation_ablation_pivot.csv", index=False) print(pivot) if __name__ == "__main__": main() ``` ### scripts/analyze_confusion_negation.py ```python from pathlib import Path import pandas as pd ROOT = Path(__file__).resolve().parents[1] EXP = ROOT / "results" / "experiments" OUT = ROOT / "results" OUT.mkdir(exist_ok=True) def parse_name(path): stem = path.stem.replace("confusion_", "") if stem.endswith("_linear_svc"): return stem[:-11].replace("__", "/"), "linear_svc" if stem.endswith("_logreg"): return stem[:-7].replace("__", "/"), "logreg" raise ValueError(path.name) def load_confusion(path): model, classifier = parse_name(path) cm = pd.read_csv(path, index_col=0) labels = list(cm.index) rows = [] errors = [] for label in labels: total = float(cm.loc[label].sum()) correct = float(cm.loc[label, label]) recall = correct / total if total else 0.0 rows.append({ "source": "full_semantic", "model": model, "classifier": classifier, "class": label, "support": int(total), "correct": int(correct), "recall": recall, "error_rate": 1.0 - recall, "is_negation": label == "negation", }) for pred in labels: if pred == label: continue count = int(cm.loc[label, pred]) if count: errors.append({ "source": "full_semantic", "model": model, "classifier": classifier, "true_class": label, "predicted_class": pred, "count": count, "true_support": int(total), "share_of_true": count / total if total else 0.0, "involves_negation": label == "negation" or pred == "negation", }) return rows, errors def main(): paths = sorted((EXP / "lie_llm_full_semantic_holdout_results").glob("confusion_*.csv")) all_rows = [] all_errors = [] for path in paths: rows, errors = load_confusion(path) all_rows.extend(rows) all_errors.extend(errors) class_df = pd.DataFrame(all_rows) error_df = pd.DataFrame(all_errors) class_df.to_csv(OUT / "confusion_class_recall.csv", index=False) error_df.sort_values(["count", "share_of_true"], ascending=False).to_csv( OUT / "confusion_top_errors.csv", index=False, ) neg_summary = ( class_df .assign(group=lambda d: d["is_negation"].map({True: "negation", False: "non_negation"})) .groupby(["source", "model", "classifier", "group"]) .agg( mean_recall=("recall", "mean"), mean_error_rate=("error_rate", "mean"), n_classes=("class", "count"), ) .reset_index() ) neg_summary.to_csv(OUT / "confusion_negation_summary.csv", index=False) rank_df = class_df.copy() rank_df["recall_rank_low_is_hard"] = rank_df.groupby(["model", "classifier"])["recall"].rank(method="min") rank_df.sort_values(["model", "classifier", "recall"]).to_csv( OUT / "confusion_class_recall_ranked.csv", index=False, ) print("Saved:") for name in [ "confusion_class_recall.csv", "confusion_top_errors.csv", "confusion_negation_summary.csv", "confusion_class_recall_ranked.csv", ]: print(OUT / name) if __name__ == "__main__": main() ``` ### scripts/lie_llm_centroid_classifier_from_pairwise.py ```python import os import numpy as np import pandas as pd from dotenv import load_dotenv load_dotenv() RESULTS_DIR = os.getenv("LIE_RESULTS_DIR", "results/experiments/lie_llm_large_results") OUT_DIR = os.path.join(RESULTS_DIR, "centroid_classifier_from_pairwise") os.makedirs(OUT_DIR, exist_ok=True) DELTA_CSV = os.path.join(RESULTS_DIR, "delta_vectors_metrics.csv") PAIRWISE_CSV = os.path.join(RESULTS_DIR, "pairwise_delta_similarity.csv") def main(): print("Loading:") print(DELTA_CSV) print(PAIRWISE_CSV) delta_df = pd.read_csv(DELTA_CSV) pairwise_df = pd.read_csv(PAIRWISE_CSV) models = sorted(delta_df["model"].unique()) all_summary = [] for model in models: print(f"\n=== MODEL: {model} ===") model_delta = delta_df[delta_df["model"] == model].reset_index(drop=True) model_pairwise = pairwise_df[pairwise_df["model"] == model].reset_index(drop=True) labels = model_delta["class"].astype(str).to_numpy() classes = sorted(set(labels)) n = len(labels) # Восстанавливаем матрицу cosine similarity sim_matrix = np.eye(n, dtype=float) idx = 0 for i in range(n): for j in range(i + 1, n): sim_matrix[i, j] = model_pairwise.iloc[idx]["cosine"] sim_matrix[j, i] = model_pairwise.iloc[idx]["cosine"] idx += 1 correct = 0 predictions = [] for i in range(n): class_scores = {} for cls in classes: cls_idx = np.where(labels == cls)[0] cls_idx = cls_idx[cls_idx != i] if len(cls_idx) == 0: class_scores[cls] = -999 else: class_scores[cls] = float(sim_matrix[i, cls_idx].mean()) pred = max(class_scores, key=class_scores.get) predictions.append({ "model": model, "index": i, "true_class": labels[i], "predicted_class": pred, "correct": pred == labels[i], **{f"score_{cls}": class_scores[cls] for cls in classes}, }) if pred == labels[i]: correct += 1 pred_df = pd.DataFrame(predictions) acc = correct / n random_baseline = 1 / len(classes) summary_row = { "model": model, "accuracy": acc, "random_baseline": random_baseline, "n_samples": n, "n_classes": len(classes), } all_summary.append(summary_row) print(summary_row) pred_df.to_csv( os.path.join( OUT_DIR, f"predictions_{model.replace('/', '__')}.csv" ), index=False, ) confusion = pd.crosstab( pred_df["true_class"], pred_df["predicted_class"], rownames=["true"], colnames=["predicted"], normalize="index", ) confusion.to_csv( os.path.join( OUT_DIR, f"confusion_{model.replace('/', '__')}.csv" ) ) print(confusion) summary = pd.DataFrame(all_summary) summary.to_csv(os.path.join(OUT_DIR, "classifier_summary.csv"), index=False) print("\n=== SUMMARY ===") print(summary) print("\nSaved to:", OUT_DIR) if __name__ == "__main__": main() ``` [98 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]