# `PtcRunner.Lisp.Prelude.FormScanner`
[🔗](https://github.com/andreasronge/ptc_runner/blob/main/lib/ptc_runner/lisp/prelude/form_scanner.ex#L1)

Byte-exact top-level form span scanner for raw PTC-Lisp prelude source TEXT.

The public and fast parser implementations track no source positions, and
parse -> swap-form -> re-render via `Formatter` would
reformat every untouched form, destroying byte-identity. `scan/1` instead
walks the RAW TEXT directly, tracking bracket/string/comment depth well
enough to compute the byte span of every top-level form plus the
whitespace/comment "gap" that precedes it, without ever building an AST.

## Result shape

`scan/1` returns `{:ok, result}` where `result` is:

    %{
      forms: [
        %{
          head: "defn",           # see "head/name" below
          name: "get-user",       # nil if the form has no name (see below)
          gap: {0, 12},           # {offset, length} gap BEFORE this form
          span: {12, 84}          # {offset, length} of the form itself
        },
        ...
      ],
      trailing_gap: {96, 4}       # {offset, length} of content after the
                                  # LAST form (whitespace/comments/EOF slack)
    }

Every offset/length pair is in BYTES, not characters. Reconcatenating
`gap ++ span ++ gap ++ span ++ ... ++ trailing_gap` in order reproduces the
original source byte-for-byte — `scan/1` checks this itself (see
"Self-checks" below), so callers never have to.

## head / name

A prelude is not the only source this scanner has to survive: the corpus
gate (below) includes `test/smoke/*.clj` files, which are plain PTC-Lisp
scripts, not preludes — their final top-level "form" is often a bare
return value (a map literal, a vector, ...), not a `(head ...)` list. So
`head`/`name` are defined for ANY top-level value, not only lists:

  * If the form is a list whose first element is a symbol (`(defn ...)`,
    `(let ...)`, `(anything-callable ...)`), `head` is that symbol's
    literal source text.
  * `name` is populated only when `head` is one of the four
    prelude-recognized heads — `ns`, `defn`, `defn-`, `def` — and the
    list's SECOND element is itself a plain symbol (the namespace name for
    `ns`, the def/defn name otherwise). This mirrors
    `PtcRunner.Lisp.Prelude.Compiler`'s `handle_ns_directive/2`,
    `handle_defn/3`, and `handle_def/2`, which all take the name from that
    same position — but this module does NOT call into `Compiler`; the two
    are independent by construction (see "Self-checks").
  * Every other shape — a list with a non-symbol head (or an empty list),
    or any non-list top-level value (a vector, map, set, string, number,
    keyword, ...) — gets a generic `head` describing its syntactic kind
    (`"list"`, `"vector"`, `"map"`, `"set"`, `"short_fn"`, `"string"`,
    `"regex_literal"`, `"var"`, `"quoted_symbol"`, `"keyword"`, `"number"`,
    `"nil"`, `"true"`, `"false"`) and `name: nil`. A bare top-level SYMBOL
    (unusual, but legal) gets its own literal text as `head`.

`head`/`name` are metadata for humans and for `prelude/edit` to key off of
— never re-parsed by this module, and never assumed to be unique.

## Design decisions (fail-closed, on purpose)

1. **Fails closed on surplus top-level delimiters.** The real reader records
   a stray `)`/`]`/`}` between top-level forms while scanning onward so a
   later hard syntax failure can remain the primary diagnostic. If no later
   failure occurs, the reader rejects the recorded closers. This scanner has
   no diagnostic-recovery role, so it rejects the first stray closer directly
   as `%{reason: :unexpected_byte}`.
2. **Character literals classify as `"string"`, not `"char_literal"`.**
   The reader itself has no separate character-literal AST node — `\a`
   and `"a"` both parse to `{:string, "a"}` (see `FastParser.parse_char/1`
   and the language spec §3.5: "Character literals are represented as
   single-character strings internally"). Giving them a distinct `head`
   label here would make this scanner's classification strictly finer
   than the reader's own value model, which the cross-check (below) would
   then have no way to agree with. So both scan to `"string"`.
3. **No semantic validation.** Map literals aren't checked for even arity,
   number/keyword literals aren't validated beyond their character class,
   and namespaced/qualified symbols aren't specially parsed — this module
   only needs byte-accurate token BOUNDARIES, never the parsed VALUE. Full
   semantic validation still happens at the existing compile gate
   (`PtcRunner.Lisp.Prelude.Compiler.compile/1`).

## Self-checks (both run INSIDE `scan/1`, never left to the caller)

1. **Round-trip.** Reconcatenating every gap + form span + the trailing
   gap must reproduce the input byte-for-byte, or scanning fails with
   `%{reason: :scan_roundtrip_mismatch}`.
2. **Cross-check against the real parser.** `scan/1` also derives the
   ordered `{head, name}` list by calling `PtcRunner.Lisp.Parser.parse/1`
   on the SAME source and walking the returned AST (a second,
   independent code path — see `derive_head_name/1` below, which
   classifies parsed AST shapes the same way `tag_label/1` classifies raw
   scan tags, without either function calling the other). Any
   disagreement (including the real parser rejecting source this scanner
   accepted) fails with `%{reason: :scan_cross_check_mismatch}` or
   `%{reason: :scan_cross_check_parse_failed}`. This is also the backstop
   for lexical-grammar drift: if `FastParser`'s reader grammar ever
   changes and this hand-written mirror doesn't get updated to match, the
   cross-check starts failing loudly instead of silently drifting.

With both checks in place, the worst failure mode is a loud refusal —
never a silently wrong span.

## Locating the `(ns ...)` docstring

`locate_ns_doc/2` is a narrowly-scoped helper for source-authoring tools:
given the byte span of a `(ns ...)`
form (as produced by `scan/1` — a `%{head: "ns", ...}` entry), it locates
the docstring token, if any, WITHOUT re-deriving or duplicating any of
`PtcRunner.Lisp.Prelude.Compiler`'s `ns_metadata/1` semantics — it reuses
this module's own `scan_symbol/1`/`scan_value/1`/`skip_ws/1` machinery
against the same span. Span in, span(s) out: callers splice bytes
themselves; this module never rewrites source text.

## Reader-macro coverage

Mirrors every construct the fast parser accepts:
lists `(...)`, vectors `[...]`, maps `{...}`, sets `#{...}`, short-fn
`#(...)`, strings `"..."` and regex literals `#"..."` (identical escape
rules — `\\`, `\"`, `\n`, `\t`, `\r`, any other `\<char>` passed through,
and literal embedded newlines allowed; unterminated is the one string-side
error), character literals (`\newline`, `\space`, `\tab`, `\return`,
`\backspace`, `\formfeed`, or any other single UTF-8 codepoint), var refs
`#'name`, quoted symbols `'name` (quoted COLLECTIONS stay unsupported, same
as the reader), keywords `:name`, numbers (int/float/exponent), the
`##Inf`/`##-Inf`/`##NaN` literals, `nil`/`true`/`false`, comma-as-whitespace,
`;` line comments, and symbols (including the embedded-`'` prime-notation
rest-char, e.g. `inc'`).

# `byte_span`

```elixir
@type byte_span() :: {non_neg_integer(), non_neg_integer()}
```

# `form`

```elixir
@type form() :: %{
  head: String.t(),
  name: String.t() | nil,
  gap: byte_span(),
  span: byte_span()
}
```

# `ns_doc_location`

```elixir
@type ns_doc_location() :: %{
  doc_span: byte_span() | nil,
  insert_at: non_neg_integer()
}
```

# `scan_error`

```elixir
@type scan_error() :: %{:reason =&gt; atom(), optional(atom()) =&gt; term()}
```

# `t`

```elixir
@type t() :: %{forms: [form()], trailing_gap: byte_span()}
```

# `locate_ns_doc`

```elixir
@spec locate_ns_doc(binary(), byte_span()) ::
  {:ok, ns_doc_location()} | {:error, scan_error()}
```

Locates the `(ns ...)` docstring token within `ns_span` — the `{offset,
length}` byte span of a form `scan/1` reported with `head: "ns"`.

Returns `doc_span`, the exact byte span of the quoted string literal
(INCLUDING its surrounding quotes) when the ns form already has a
docstring — e.g. `(ns name "doc" ...)` — or `nil` when it does not (`(ns
name)` / `(ns name {meta})`). `insert_at` is always the byte offset
immediately after the namespace name symbol (before any following
whitespace, docstring, metadata map, or closing paren): a caller replaces
exactly `doc_span` in place when it is present, or splices new bytes at
`insert_at` when it is `nil` — either way, no other byte of the ns form is
ever touched.

Fails closed (`{:error, %{reason: :unexpected_ns_form}}`) on any shape
other than `(ns <symbol> ...)`. Callers are only expected to pass spans
`scan/1` itself already tagged `head: "ns"` for the SAME source, so this
should never trigger for a real caller — it exists so a shape this module
doesn't recognize is refused rather than mis-splice.

# `scan`

```elixir
@spec scan(binary()) :: {:ok, t()} | {:error, scan_error()}
```

Scans `source` and returns the byte-exact top-level form spans, or a
fail-closed `{:error, %{reason: ...}}` — never a guess.

---

*Consult [api-reference.md](api-reference.md) for complete listing*
