[PR #43] Semi rewrite #43

New issue

Open

opened 2026-05-22 22:05:17 +01:00 by JakeStanger · 0 comments

JakeStanger commented

2026-05-22 22:05:17 +01:00

Owner

📋 Pull Request Information

Original PR: https://github.com/corn-config/corn/pull/43
Author: @puppymati
Created: 6/15/2025
Status: 🔄 Open

Base: master ← Head: semi-rewrite

📝 Commits (10+)

2f7b71d Clean up cargo manifest
d2b29f6 Add implementation for Value
008abd7 Make object and integer public
27daca2 Fix too compact tests
d1269c8 Implement a full lexer
56abcf6 Add more exaustive integer parsing
6420c79 Implement a lalrpop parser
24926a0 Expose error to library root
c9bdf52 Mark integer as repr transparent
80abd83 Make the lexer and parser borrow the input

📊 Changes

160 files changed (+2203 additions, -3665 deletions)

View changed files

📝 .github/workflows/ci.yml (+2 -2)
➕ .gitmodules (+3 -0)
📝 Cargo.lock (+484 -453)
📝 Cargo.toml (+43 -38)
➖ assets/inputs/array.corn (+0 -3)
➖ assets/inputs/basic.corn (+0 -3)
➖ assets/inputs/basic_empty_let.corn (+0 -3)
➖ assets/inputs/boolean.corn (+0 -4)
➖ assets/inputs/chained.corn (+0 -3)
➖ assets/inputs/chained_complex.corn (+0 -8)
➖ assets/inputs/char.corn (+0 -3)
➖ assets/inputs/comment.corn (+0 -4)
➖ assets/inputs/compact.corn (+0 -15)
➖ assets/inputs/complex.corn (+0 -56)
➖ assets/inputs/complex_keys.corn (+0 -9)
➖ assets/inputs/environment_variable.corn (+0 -3)
➖ assets/inputs/float.corn (+0 -4)
➖ assets/inputs/input.corn (+0 -18)
➖ assets/inputs/input_references_input.corn (+0 -6)
➖ assets/inputs/integer.corn (+0 -5)

...and 80 more files

📄 Description

This PR is a major rewrite of the parser that comes with many improvements and changes. At its core, the main change is the underlying technology.

Instead of using a Pest grammar which is hard to maintain and read, the new parser separates the steps into a proper lexer using Logos and a simple but very efficient parser using LALRPOP. This gets turned into an AST which also allows for more efficient and streamlined implementation of the deserializer.

The parser is largely backwards compatible except where the old parser was wrong. The new parser should be more correct, and some tests have been updated to reflect that. The value enum has also been updated to be more ergonomic and easier to use.

There are a few breaking changes that improve the syntax and were agreed upon beforehand:

Stricter parsing of unquoted keys, most notably internal quotes are no longer allowed
Less compact syntax, stuff like truefalse won't work anymore
String interpolation now uses a syntax similar to the one used in Rust: "$var" → "${var}"
Unicode escapes also use a new syntax that allows for more Unicode planes to be used: "\u{FFFD}"

Enhanced Integer Parsing Support:

Added octal integer support: New 0o prefix syntax (e.g., 0o755 → 493, 0o123 → 83)
Added binary integer support: New 0b prefix syntax (e.g., 0b1010 → 10, 0b1111 → 15)
All integer formats support:
- Negative values: -0xABC, -0o755, -0b1100
- Underscore separators for readability: 1_000_000, 0xFA_FA_FA, 0o12_34_56, 0b1010_1100

Format	Prefix	Example	Result
Decimal	none	42, 1_000_000	42, 1000000
Hexadecimal	0x	0xFF, 0xFA_FA	255, 64250
Octal	0o	0o755, 0o12_34	493, 668
Binary	0b	0b1010, 0b10_11	10, 11

There might be other changes that are more of a result of the older parser not properly respecting the spec at times.

The changes should address all the concerns highlighted in #42.

There are other changes planned and more extensive error handling which will be implemented in other PRs. The new code should allow much easier extensibility.

Lua is fully working, the CLI has been updated too. WASM support is now fully working and fixed.

Overall, the new code is more readable, maintainable, correct, more efficient and possibly even faster, although that wasn't the goal of this rewrite.

Resolves #42

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/corn-config/corn/pull/43 **Author:** [@puppymati](https://github.com/puppymati) **Created:** 6/15/2025 **Status:** 🔄 Open **Base:** `master` ← **Head:** `semi-rewrite` --- ### 📝 Commits (10+) - [`2f7b71d`](https://github.com/corn-config/corn/commit/2f7b71d6076f82feab563abbbdc193f5503b184e) Clean up cargo manifest - [`d2b29f6`](https://github.com/corn-config/corn/commit/d2b29f6949e6afc9bbbe876b377c67f09fa59c29) Add implementation for Value - [`008abd7`](https://github.com/corn-config/corn/commit/008abd7fa315ebb7fceda1c86b30e5ba0453753c) Make object and integer public - [`27daca2`](https://github.com/corn-config/corn/commit/27daca2cba93817e5dfeaa967b4b7f935a3a6a13) Fix too compact tests - [`d1269c8`](https://github.com/corn-config/corn/commit/d1269c8575840f8e435d8f180d05f77b00ff804a) Implement a full lexer - [`56abcf6`](https://github.com/corn-config/corn/commit/56abcf6dc1d8094d0878f269312f44ed1c49117a) Add more exaustive integer parsing - [`6420c79`](https://github.com/corn-config/corn/commit/6420c79c7de4dec530a179e0771de7a7ce38d9a6) Implement a lalrpop parser - [`24926a0`](https://github.com/corn-config/corn/commit/24926a0bcb36f8779b3b5bc5b95a32ba7e5cb588) Expose error to library root - [`c9bdf52`](https://github.com/corn-config/corn/commit/c9bdf52b1db0186cff303d3b40c7b5d07552f570) Mark integer as repr transparent - [`80abd83`](https://github.com/corn-config/corn/commit/80abd83ce44aa1623cf5cd9e3ce690ab98e20ac0) Make the lexer and parser borrow the input ### 📊 Changes **160 files changed** (+2203 additions, -3665 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+2 -2) ➕ `.gitmodules` (+3 -0) 📝 `Cargo.lock` (+484 -453) 📝 `Cargo.toml` (+43 -38) ➖ `assets/inputs/array.corn` (+0 -3) ➖ `assets/inputs/basic.corn` (+0 -3) ➖ `assets/inputs/basic_empty_let.corn` (+0 -3) ➖ `assets/inputs/boolean.corn` (+0 -4) ➖ `assets/inputs/chained.corn` (+0 -3) ➖ `assets/inputs/chained_complex.corn` (+0 -8) ➖ `assets/inputs/char.corn` (+0 -3) ➖ `assets/inputs/comment.corn` (+0 -4) ➖ `assets/inputs/compact.corn` (+0 -15) ➖ `assets/inputs/complex.corn` (+0 -56) ➖ `assets/inputs/complex_keys.corn` (+0 -9) ➖ `assets/inputs/environment_variable.corn` (+0 -3) ➖ `assets/inputs/float.corn` (+0 -4) ➖ `assets/inputs/input.corn` (+0 -18) ➖ `assets/inputs/input_references_input.corn` (+0 -6) ➖ `assets/inputs/integer.corn` (+0 -5) _...and 80 more files_ </details> ### 📄 Description This PR is a major rewrite of the parser that comes with many improvements and changes. At its core, the main change is the underlying technology. Instead of using a Pest grammar which is hard to maintain and read, the new parser separates the steps into a proper lexer using Logos and a simple but very efficient parser using LALRPOP. This gets turned into an AST which also allows for more efficient and streamlined implementation of the deserializer. The parser is largely backwards compatible except where the old parser was wrong. The new parser should be more correct, and some tests have been updated to reflect that. The value enum has also been updated to be more ergonomic and easier to use. There are a few breaking changes that improve the syntax and were agreed upon beforehand: - Stricter parsing of unquoted keys, most notably internal quotes are no longer allowed - Less compact syntax, stuff like `truefalse` won't work anymore - String interpolation now uses a syntax similar to the one used in Rust: `"$var"` → `"${var}"` - Unicode escapes also use a new syntax that allows for more Unicode planes to be used: `"\u{FFFD}"` **Enhanced Integer Parsing Support:** - Added octal integer support: New 0o prefix syntax (e.g., 0o755 → 493, 0o123 → 83) - Added binary integer support: New 0b prefix syntax (e.g., 0b1010 → 10, 0b1111 → 15) - All integer formats support: - Negative values: -0xABC, -0o755, -0b1100 - Underscore separators for readability: 1_000_000, 0xFA_FA_FA, 0o12_34_56, 0b1010_1100 | Format | Prefix | Example | Result | |-------------|--------|-----------------|-------------| | Decimal | none | 42, 1_000_000 | 42, 1000000 | | Hexadecimal | 0x | 0xFF, 0xFA_FA | 255, 64250 | | Octal | 0o | 0o755, 0o12_34 | 493, 668 | | Binary | 0b | 0b1010, 0b10_11 | 10, 11 | There might be other changes that are more of a result of the older parser not properly respecting the spec at times. The changes should address all the concerns highlighted in #42. There are other changes planned and more extensive error handling which will be implemented in other PRs. The new code should allow much easier extensibility. Lua is fully working, the CLI has been updated too. WASM support is now fully working and fixed. Overall, the new code is more readable, maintainable, correct, more efficient and possibly even faster, although that wasn't the goal of this rewrite. Resolves #42 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

JakeStanger added the

pull-request

label

2026-05-22 22:05:17 +01:00