# Project export: Basado Language

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Higher Level Systems language with dependent types and other obscure CS concepts
- Devpost: https://devpost.com/software/basado-language
- GitHub: https://github.com/MiguelX413/calhacks_basado
- Demo: https://docs.google.com/presentation/d/1zm7Qg0_Cn0lTLgc4e68YPEfsQD3i5jQ-w0OBfGPxJqc/edit#slide=id.p
- Team: 1 GitHub contributor(s) — MiguelX413 (22 commits)

## Devpost submission (written by the team)

No Devpost description available.

## README (from the GitHub repository)

# Ogre Compiler

## Introduction

Welcome to the Ogre Compiler project! Ogre is an innovative programming language that
seamlessly integrates systems programming with advanced academic and theoretical computer science concepts. This
language is designed to offer a unique blend of features like dependent types, function currying, and functional
programming, catering to both practical and theoretical applications in computer science.

## Current Status

As of now, the project is in its early stages, focusing on the foundational aspects of the language compiler. The
tokenizer, a crucial component for parsing the language, has been implemented. Future developments will include the
construction of the Abstract Syntax Tree (AST) and the LLVM front-end.

### Completed Features

- **Tokenizer**: The tokenizer is responsible for breaking down the source code into tokens, which are the basic
  building blocks for further
  parsing. [View Tokenizer Code](https://github.com/MiguelX413/ogre/blob/master/tokenizer/src/lib.rs)

### Upcoming Features

- **Abstract Syntax Tree (AST)**: The AST will represent the hierarchical syntactic structure of the source code, which
  is essential for the subsequent stages of compilation.
- **LLVM Front-End**: Integration with LLVM will enable the compiler to generate efficient machine code, leveraging
  LLVM's powerful optimization and code generation capabilities.

## License

This project is licensed under [GPL-3.0 License](https://github.com/MiguelX413/ogre/blob/master/LICENSE).


## Detected evidence (automated analysis)

Indexed codebase: 7 recognized source files, 28 KB.
- Rust (language) — detected in the code

## Codebase structure (from repository index)

### Files (13 of 13)

```
.github/workflows/rust.yml
.gitignore
Cargo.lock
Cargo.toml
demo/Cargo.toml
demo/src/main.rs
LICENSE
README.md
tokenizer/Cargo.toml
tokenizer/src/lib.rs
tokenizer/src/types/defs.rs
tokenizer/src/types/impls.rs
tokenizer/src/types/mod.rs
```

### Dependencies

- demo/Cargo.toml: tokenizer

### Recent commits (newest first)

- Rename to Ogre
- Rename to Allium
- Sort punct tokenizing branches lexicographically
- Add `PlusPercent`, `PlusPipe`, `MinusPipe`, `MinusPercent`, `StarPercent`, `StarPipe`, and `ShlPipe` puncts
- Derive `Copy` for `Token`
- Rename `Token`'s `s` field to `lexeme`
- Add even more lints
- Remove `"while"`, `"true"`, `"false"`, `"gen"`; Rename `"typeclass"` to `"class"`; Add `"fn"`
- Invert `unseparated_literal_suffix`
- Adjust lints
- Add some lints
- Create rust.yml
- impl Copy for {TokenType, Literal}
- Rename `TokenKind` to `TokenType`
- Rename terminator_finder to find_unescaped
- Move SplitTokens creation logic, rearrange fields
- Clarify purpose of iterator skips
- Add number parsing examples
- Separate number lit.s into different types, add NonInt parsing, optimize away `core::iter::Skip`s
- Set lint level of `unused_must_use` to deny, must_use SplitTokens::new

## Key source files (fetched from GitHub, selected and truncated for size)

### Cargo.toml

```
[workspace]
members = [
    "tokenizer",
    "demo",
]
resolver = "2"

```

### tokenizer/Cargo.toml

```
[package]
name = "tokenizer"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

```

### demo/Cargo.toml

```
[package]
name = "demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tokenizer = { path = "../tokenizer" }

```

### demo/src/main.rs

```rust
use tokenizer::split_tokens;

pub fn main() {
    [
        "catfood-45",
        "catfood",
        "67z23",
        "catfood&-45",
        "&",
        " -45 - 45 + +45",
        "if +2 + -2 else x := x - 5 ",
        "if {{10 / {45 + 3}} + {2 * 4}} - +5",
        "日本語a+123",
        "cat- 324_32432432432-ref",
        "{2133 ** 21} % 2",
        r#"let my_string := "lol\"test";
let xd: Int := 2;
let multi_line_str := "xd\
sus";"#,
        "let _ := 5;",
        "34_2 432.2_34 234.count_ones() 3424.",
        "240",
    ]
    .into_iter()
    .for_each(|string| {
        println!(
            "{string:?}: {:?}",
            split_tokens(string).collect::<Result<Vec<_>, _>>()
        )
    });
}

```

### .github/workflows/rust.yml

```yaml
name: Rust

on:
  - push
  - pull_request

env:
  CARGO_TERM_COLOR: always
  RUSTFLAGS: "-Dwarnings"

jobs:
  rust:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Run Rustfmt
      run: cargo fmt --check
    - name: Run Clippy
      run: cargo clippy --all-targets --all-features
    - name: Build
      run: cargo build --verbose
    - name: Run tests
      run: cargo test --verbose

```

### tokenizer/src/lib.rs

```rust
#![warn(
    clippy::pedantic,
    clippy::decimal_literal_representation,
    clippy::format_push_string,
    clippy::print_stderr,
    clippy::print_stdout,
    clippy::str_to_string,
    clippy::string_add,
    clippy::string_lit_chars_any,
    clippy::string_to_string,
    clippy::suspicious_xor_used_as_pow,
    clippy::tests_outside_test_module,
    clippy::todo,
    clippy::try_err,
    clippy::undocumented_unsafe_blocks,
    clippy::todo,
    clippy::unimplemented,
    clippy::unnecessary_self_imports,
    clippy::unneeded_field_pattern,
    clippy::unseparated_literal_suffix,
    clippy::unreadable_literal,
    clippy::if_then_some_else_none,
    clippy::impl_trait_in_params,
    clippy::default_numeric_fallback,
    clippy::self_named_module_files,
    clippy::suboptimal_flops,
    clippy::style,
    unused_must_use
)]

pub use crate::types::{
    Comment, Delimiter, Keyword, LineColumn, Literal, Punct, Span, Token, TokenType,
};
use std::borrow::Cow;
use std::fmt::{Display, Formatter};
use std::iter::FusedIterator;

mod types;

#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub enum ParseTokenError<'a> {
    InvalidChar(char, &'a str),
    CapsInImproperIdent(&'a str, usize),
    UnderscoreInProper(&'a str, usize),
    UnterminatedStrLit,
    UnterminatedChrLit,
    InvalidEscape(char),
}

impl<'a> Display for ParseTokenError<'a> {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self {
            Self::InvalidChar(c, _) => write!(f, "Invalid char: {c}"),
            Self::CapsInImproperIdent(s, i) => {
                write!(f, "Caps in improper identifier, {s:?}, at pos {i}")
            }
            Self::UnderscoreInProper(s, i) => {
                write!(f, "Underscore in proper identifier, {s:?}, at pos {i}")
            }
            Self::UnterminatedStrLit => write!(f, "No string terminator found!"),
            Self::UnterminatedChrLit => write!(f, "No char terminator found!"),
            Self::InvalidEscape(c) => write!(f, "Invalid escape \\{c}"),
        }
    }
}

impl<'a> std::error::Error for ParseTokenError<'a> {}

#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct SplitTokens<'a> {
    remainder: &'a str,
    original: &'a str,
    line_column: LineColumn,
}

impl<'a> SplitTokens<'a> {
    #[must_use]
    pub(crate) fn new(remainder: &'a str, original: &'a str, line_column: LineColumn) -> Self {
        Self {
            remainder,
            original,
            line_column,
        }
    }

    #[must_use]
    pub fn remainder(&self) -> &str {
        self.remainder
    }

    #[must_use]
    pub fn original(&self) -> &str {
        self.original
    }

    #[must_use]
    pub fn line_column(&self) -> LineColumn {
        self.line_column
    }
}

macro_rules! sp {
    ($char:pat) => {
        ($char, _)
    };
    ($char1:pat, $char2:pat) => {
        ($char1, Some(($char2, _)))
    };
    ($char1:pat, $char2:pat, $char3:pat) => {
        ($char1, Some(($char2, Some($char3))))
    };
}
macro_rules! st {
    ($char:expr, $token_type:expr, $self:expr) => {{
        let char: char = $char;
        let len: usize = char.len_utf8();
        let token_type: crate::types::TokenType = $token_type;
        let (token, remainder): (&str, &str) = $self.remainder.split_at(len);
        Ok((
            Token::new(
                token_type,
                token,
                Span::new(
                    $self.line_column,
                    LineColumn::new($self.line_column.line, $self.line_column.column + 1),
                ),
            ),
            remainder,
        ))
    }};
    ($char1:expr, $char2:expr, $token_type:expr, $self:expr) => {{
        let (char1, char2): (char, char) = ($char1, $char2);
        let len: usize = char1.len_utf8() + char2.len_utf8();
        let token_type: crate::types::TokenType = $token_type;
        let (token, remainder): (&str, &str) = $self.remainder.split_at(len);
        Ok((
            Token::new(
                token_type,
                token,
                Span::new(
                    $self.line_column,
                    LineColumn::new($self.line_column.line, $self.line_column.column + 2),
                ),
            ),
            remainder,
        ))
    }};
    ($char1:expr, $char2:expr, $char3:expr, $token_type:expr, $self:expr) => {{
        let (char1, char2, char3): (char, char, char) = ($char1, $char2, $char3);
        let len: usize = char1.len_utf8() + char2.len_utf8() + char3.len_utf8();
        let token_type: crate::types::TokenType = $token_type;
        let (token, remainder): (&str, &str) = $self.remainder.split_at(len);
        Ok((
            Token::new(
                token_type,
                token,
                Span::new(
                    $self.line_column,
                    LineColumn::new($self.line_column.line, $self.line_column.column + 3),
                ),
            ),
            remainder,
        ))
    }};
}

macro_rules! find_unescaped {
    ($pat:pat, $str:expr) => {{
        let mut escaped = false;
        let mut char_indices: core::str::CharIndices = $str.char_indices();
        let _ = char_indices.next(); // Skip 1
        char_indices
            .find(|(_, c)| match (c, escaped) {
                ('\\', false) => {
                    escaped = true;
                    false
                }
                ($pat, false) => true,
                (_, true) => {
                    escaped = false;
                    false
                }
                (_, false) => false,
            })
            .map(|(i, _)| i)
    }};
}

#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct ParseEscapesError(char);

impl<'a> From<ParseEscapesError> for ParseTokenError<'a> {
    fn from(parse_escapes_error: ParseEscapesError) -> Self {
        Self::InvalidEscape(parse_escapes_error.0)
    }
}

/// This function is meant to be used on the content between the terminators of string and character literals.
/// # E
[truncated — 13089 more characters]
```

### tokenizer/src/types/mod.rs

```rust
pub use defs::*;

pub mod defs;
pub mod impls;

```

### tokenizer/src/types/impls.rs

```rust
use std::fmt::{Display, Formatter};

use crate::types::defs::{Delimiter, Keyword, Punct};

impl Display for Punct {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self {
            Self::Assign => write!(f, ":="),
            Self::PlusPlus => write!(f, "++"),
            Self::MinusMinus => write!(f, "--"),
            Self::Plus => write!(f, "+"),
            Self::Minus => write!(f, "-"),
            Self::Star => write!(f, "*"),
            Self::StarStar => write!(f, "**"),
            Self::Slash => write!(f, "/"),
            Self::Percent => write!(f, "%"),
            Self::Caret => write!(f, "^"),
            Self::Not => write!(f, "!"),
            Self::And => write!(f, "&"),
            Self::Or => write!(f, "|"),
            Self::Shl => write!(f, "<<"),
            Self::Shr => write!(f, ">>"),
            Self::Eq => write!(f, "="),
            Self::EqEq => write!(f, "=="),
            Self::Gt => write!(f, ">"),
            Self::Lt => write!(f, "<"),
            Self::Ge => write!(f, ">="),
            Self::Le => write!(f, "<="),
            Self::At => write!(f, "@"),
            Self::Underscore => write!(f, "_"),
            Self::Dot => write!(f, "."),
            Self::Comma => write!(f, ","),
            Self::Semi => write!(f, ";"),
            Self::Colon => write!(f, ":"),
            Self::ColonColon => write!(f, "::"),
            Self::RArrow => write!(f, "->"),
            Self::FatArrow => write!(f, "=>"),
            Self::Tilde => write!(f, "~"),
            Self::ForAll => write!(f, "∀"),
            Self::Exists => write!(f, "∃"),
            Self::PlusPercent => write!(f, "+%"),
            Self::PlusPipe => write!(f, "+|"),
            Self::MinusPercent => write!(f, "-%"),
            Self::MinusPipe => write!(f, "-|"),
            Self::StarPercent => write!(f, "*%"),
            Self::StarPipe => write!(f, "*|"),
            Self::ShlPipe => write!(f, "<<|"),
        }
    }
}

impl Display for Delimiter {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self {
            Self::CurlyLeft => write!(f, "{{"),
            Self::CurlyRight => write!(f, "}}"),
            Self::SquareLeft => write!(f, "["),
            Self::SquareRight => write!(f, "]"),
            Self::ParLeft => write!(f, "("),
            Self::ParRight => write!(f, ")"),
        }
    }
}

impl Display for Keyword {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self {
            Self::If => write!(f, "if"),
            Self::Else => write!(f, "else"),
            Self::Match => write!(f, "match"),
            Self::Loop => write!(f, "loop"),
            Self::Let => write!(f, "let"),
            Self::Type => write!(f, "type"),
            Self::Class => write!(f, "class"),
            Self::Ret => write!(f, "ret"),
            Self::Where => write!(f, "where"),
            Self::Miguel => write!(f, "miguel"),
            Self::Kyasig => write!(f, "kyasig"),
            Self::Claim => write!(f, "claim"),
            Self::Cardinality => write!(f, "cardinality"),
            Self::Bytes => write!(f, "bytes"),
            Self::Bits => write!(f, "bits"),
            Self::Fn => write!(f, "fn"),
        }
    }
}

```

### tokenizer/src/types/defs.rs

```rust
#[derive(Copy, Clone, Default, Debug, Eq, Hash, PartialEq)]
pub struct Span {
    pub start: LineColumn,
    pub end: LineColumn,
}

impl Span {
    #[must_use]
    pub fn new(start: LineColumn, end: LineColumn) -> Self {
        Self { start, end }
    }
}

impl From<(LineColumn, LineColumn)> for Span {
    fn from((start, end): (LineColumn, LineColumn)) -> Self {
        Self { start, end }
    }
}

impl From<Span> for (LineColumn, LineColumn) {
    fn from(span: Span) -> Self {
        (span.start, span.end)
    }
}

#[derive(Copy, Clone, Default, Debug, Eq, Hash, PartialEq)]
pub struct LineColumn {
    pub line: usize,
    pub column: usize,
}

impl LineColumn {
    #[must_use]
    pub fn new(line: usize, column: usize) -> Self {
        Self { line, column }
    }
}

impl From<(usize, usize)> for LineColumn {
    fn from((line, column): (usize, usize)) -> Self {
        Self { line, column }
    }
}

impl From<LineColumn> for (usize, usize) {
    fn from(line_column: LineColumn) -> Self {
        (line_column.line, line_column.column)
    }
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub struct Token<'a> {
    pub token_type: TokenType,
    pub lexeme: &'a str,
    pub span: Span,
}

impl<'a> Token<'a> {
    #[must_use]
    pub fn new(token_type: TokenType, lexeme: &'a str, span: Span) -> Self {
        Self {
            token_type,
            lexeme,
            span,
        }
    }

    #[must_use]
    pub fn new_auto_span(
        token_type: TokenType,
        lexeme: &'a str,
        mut line_column: LineColumn,
    ) -> Self {
        Self {
            token_type,
            lexeme,
            span: Span::new(line_column, {
                lexeme.chars().for_each(|c| {
                    if c == '\n' {
                        line_column.column = 0;
                        line_column.line += 1;
                    } else {
                        line_column.column += 1;
                    }
                });
                line_column
            }),
        }
    }
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub enum TokenType {
    Keyword(Keyword),
    Ident,
    ProperIdent,
    Literal(Literal),
    Punct(Punct),
    Delimiter(Delimiter),
    Comment(Comment),
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub enum Keyword {
    If,
    Else,
    Match,
    Loop,
    Let,
    Type,
    Class,
    Ret,
    Where,
    Miguel,
    Kyasig,
    Claim,
    Cardinality,
    Bytes,
    Bits,
    Fn,
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub enum Literal {
    Character,
    String,
    DecInt,
    HexInt,
    OctInt,
    BinInt,
    NonInt,
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub enum Punct {
    Assign,
    PlusPlus,
    MinusMinus,
    Plus,
    Minus,
    Star,
    Slash,
    StarStar,
    Percent,
    Caret,
    Not,
    And,
    Or,
    Shl,
    Shr,
    Eq,
    EqEq,
    Gt,
    Lt,
    Ge,
    Le,
    At,
    Underscore,
    Dot,
    Comma,
    Semi,
    Colon,
    ColonColon,
    RArrow,
    FatArrow,
    Tilde,
    ForAll,
    Exists,
    PlusPercent,
    PlusPipe,
    MinusPipe,
    MinusPercent,
    StarPercent,
    StarPipe,
    ShlPipe,
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub enum Delimiter {
    CurlyLeft,
    CurlyRight,
    SquareLeft,
    SquareRight,
    ParLeft,
    ParRight,
}

#[derive(Copy, Clone, Debug, Eq, Hash, PartialEq)]
pub enum Comment {
    Comment,
    DocComment,
}

```