Notes from CS 336: Language Models from Scratch, Lecture on Resource Accounting.

Resource accounting determines computational efficiency. Before optimizing training, you need to understand compute and memory characteristics.

Core questions this framework answers:

— How long to train a 70B parameter model on 15 trillion tokens using 1024 H100s?

— Formula: 6 × parameters × tokens = total FLOPs. Factor in hardware specs and MFU of 0.5. Result: 143 days.

— What is the largest model you can train on H100s using AdamW?

— H100s have 80GB HBM. Account for 2 + 2 + 4 + 4 bytes per parameter. Result: approximately 53B parameters.

These are back-of-the-envelope calculations. The goal is not precision but understanding the rough shape of resource constraints.

Resource accounting continued:

Tensors are the building block. Parameters, gradients, optimizer states, activations—everything is a tensor.

Memory usage follows a simple calculus:

— Number of elements × element size = total memory

— A 4×8 matrix in float32 uses 128 bytes

— GPT-3 feedforward matrices reach 2.3 gigabytes per layer

Reducing precision saves memory and time:

— Operating on 16 bits runs faster than 32 bits

— Memory reduction itself saves time because data movement becomes the bottleneck

GPUs keep memory separate from compute units. Parameters and activations must move from memory to compute, execute operations, then move back. Most operations are memory bound, not compute bound.

Resource accounting continued:

Float32 is the baseline precision. 32 bits split into 1 sign bit, 8 exponent bits for dynamic range, and 23 mantissa bits for resolution. Also called fp32 or single precision.

Deep learning needs less precision than scientific computing:

— Float16 cuts memory in half but sacrifices dynamic range. Only 5 exponent bits means underflow and overflow during training. Values like 1e-8 become zero.

— Bfloat16 solves this. Developed in 2018, it keeps 8 exponent bits like fp32 but reduces mantissa to 7 bits. Same dynamic range as fp32, worse resolution, but acceptable for stochastic training.

Training implications:

— Float32 works for small models but requires 4 bytes per parameter

— Float16 is too risky for stability

— Bfloat16 is the current sweet spot

Resource accounting continued:

Mixed precision training uses different precisions for different components. General rule: bf16 for parameters, activations, and gradients. Fp32 for optimizer states.

PyTorch AMP library handles this automatically. It casts operations to bf16 when safe, like matrix multiplications. Leaves risky operations like exponentiation in fp32.

Lower precision is possible:

— Fp8 was standardized four years ago. Two versions depending on whether you need dynamic range or resolution. NVIDIA transformer engine supports this.

— Fp4 exists. NVIDIA nvfp4 has only 4 bits per value—16 possible values between -6 and 6. Not enough for naive training.

— Fp4 uses blocks. Each value gets 4 bits of freedom within a block. Blocks scale up and down. You get more than 4 bits of dynamic range per value but cannot represent full range independently.

Nemotron 3 Super was trained in fp4 this year.

Resource accounting continued:

Low-bit training differs from low-bit inference. Most 1-bit or 2-bit models are trained in bf16 then quantized afterward. Training a 1-bit language model from scratch has not been done credibly.

Tensors default to CPU in PyTorch. Move them to GPU for speed.

Computing with tensors requires accounting for operations. Einops provides a cleaner syntax than raw indexing.

Einops names dimensions instead of using numeric indices:

— Standard: x.transpose(-2, -1) is ambiguous

— Einops: specify dimensions as seq1, hidden, seq2

— Inspired by Einstein summation notation

Einops works as generalized matrix multiplication with explicit bookkeeping. Unlisted dimensions get summed out. This eliminates indexing errors and makes tensor operations readable.

Resource Accounting in AI Models (Stanford CS 336)

Notes from CS 336: Language Models from Scratch, Lecture 1.

The course operates on a from-scratch philosophy. Building everything from the ground up produces understanding that prompting and fine-tuning cannot. Abstractions are leaky. Fundamental research requires tearing up the entire stack.

The design space available through prompting is vastly constrained. Full understanding of how language models work is necessary for fundamental research. That understanding comes from building, not from using pre-built interfaces.

Language models from scratch continued:

Industrialization creates a scale problem. GPT-4 cost roughly $100 million to train in 2023. Current frontier models likely cost $1 billion. Frontier labs provide no details on architecture or training procedures.

Small models are not representative of frontier behavior. At 175B parameters, 80% of flops are spent in MLP layers versus 44% at smaller scales. Optimization targets differ across scales. Emergent behaviors appear only past critical thresholds.

Language models from scratch continued:

Three types of knowledge transfer across scales: mechanics, mindset, and intuitions.

Mechanics transfer reliably. How transformers work, how model parallelism distributes computation, how kernels execute remain constant across scale.

Mindset transfers. Profiling everything, benchmarking systematically, squeezing maximum value from hardware, taking scaling seriously as a design constraint.

Intuitions about data and modeling decisions do not reliably transfer. Those require experimentation at target scale.

Language models from scratch continued:

The bitter lesson is commonly misinterpreted. The wrong interpretation: scale is all that matters, algorithms don't matter.

The correct interpretation: algorithms that scale are what matters.

Accuracy equals efficiency times resources. At small scale, a 2x slower algorithm means waiting twice as long. At frontier scale, that's hundreds of millions of dollars. A 5% efficiency gain becomes economically significant.

OpenAI measured 44x algorithmic efficiency improvement on ImageNet between 2012 and 2019. Hardware improved, but algorithmic gains multiplied the impact.

Language models from scratch continued:

The modern language model lineage runs through neural architectures. LSTMs in the '90s. Bengio's first neural language model in 2003. Seq2Seq, Adam, attention for machine translation. The transformer in 2017.

GPT-3 was a wake-up call. Early replication attempts were weak—Eleuther, Meta's first 175B model, BigScience.

The open ecosystem changed in the last three years. Llama, Mistral, DeepSeek, Qwen now approach frontier performance. Some projects release weights, papers, code, and data.

This course would not exist without the open ecosystem. Published details on MoE architectures and RL systems let us triangulate how frontier models are built.

Building Language Models (Stanford CS336)

Hierarchical predictive processing and the architecture of abstraction.

Predictions don't begin at the top. Every level predicts the level below it. Higher levels are special because they hold the slowest-changing, most invariant, most context-like priors that constrain the entire cascade beneath them.

The topmost levels are substantially endogenous: driven by goals, memory, and the system's own ongoing dynamics rather than current input. This endogeneity enables imagination, expectation, and mind-wandering.

Margulies and colleagues (2016) found that the transmodal default-mode network sits at maximal connectional distance from primary sensory cortex. It's the part of cortex least yoked to immediate signal.

"Abstract causes and context" is shorthand for a cluster of co-varying properties that intensify as you ascend. Each property has its own empirical anchor.

Hierarchical predictive processing continued:

Property 1: Invariance to surface detail.

• Low-level representations are tied to retinal position, lighting, viewpoint
• High-level representations abstract these away
• DiCarlo and colleagues frame ventral-stream processing as progressively "untangling" object identity
• In inferotemporal cortex, what an object is becomes linearly separable from how it currently appears

Quian Quiroga, Fried, Koch (2005) demonstrated concept cells: single human medial-temporal neurons that fire to a specific person or concept invariant to the particular photo, and even to the printed or spoken name. The neuron represents an abstract cause, an entity in the world that could have produced any of those signals.

The hub-and-spoke account (Lambon Ralph, Patterson) proposes the anterior temporal lobe acts as a transmodal hub that distills modality-specific features into amodal concepts.

Property 2: Lengthening temporal scope.

Higher levels integrate over longer windows and represent slower-changing causes. Context.

Hasson and colleagues mapped a hierarchy of temporal receptive windows:
• Early auditory and visual areas: tens to hundreds of milliseconds
• Higher areas (TPJ, precuneus, mPFC): many seconds, tracking what situation am I in

Baldassano and colleagues (2017) extended this into a nested hierarchy of events. Higher regions carry longer events. Event boundaries align across different people watching the same narrative.

Zacks's event-segmentation theory: a perceived event boundary is the moment the current high-level model fails and prediction error spikes. Context at the top is the model of the situation currently in force. You feel it update when it breaks.

Hierarchical predictive processing continued:

Property 3: Relational and structural abstraction.

Abstract cause stops meaning "a thing" and starts meaning "a structure that generalizes."

The hippocampal-entorhinal system applies spatial coding machinery to non-spatial conceptual spaces. Constantinescu, O'Reilly and Behrens (2016) found grid-like coding as people navigated an abstract two-dimensional concept space.

The Tolman-Eichenbaum Machine (Whittington et al., 2020) formalizes how this system learns the relational structure of a domain in a way that transfers to new instances. That transferability is the point of a high-level prior: not a memorized particular but a reusable abstraction that lets the system predict in situations it has never seen.

Property 4: Explicit causal structure.

The high-level variable being estimated is a hidden causal fact about the world's structure.

Körding, Shams and colleagues (2007): multisensory perception involves inferring whether two signals share a common cause. The brain decides whether a seen and heard event are the same event before deciding how to combine them. This is why the ventriloquist effect appears and disappears depending on plausibility.

The abstract quantity at the top of the hierarchy is, in the strict sense, a cause.

Hierarchical predictive processing continued:

Context is the currently active configuration of high-level priors. The subset that's switched on right now and is biasing everything downstream.

Three lines of work define where and how context operates:

1. Prefrontal implementation.

Miller and Cohen (2001) cast the PFC as maintaining goals, rules, and task context as sustained activity patterns that bias processing in posterior and motor regions. Context as a top-down control signal.

Mante, Sussillo and Newsome (2013) made this mechanistic: PFC populations perform context-dependent computation. The same sensory input is routed differently depending on which rule is active, supported by high-dimensional mixed selectivity (Rigotti et al., 2013) that lets a population flexibly remap inputs to outputs as context changes.

2. Schema and situation models.

Bartlett (1932): memory is reconstruction guided by prior schemas. Tse and colleagues (2007): an existing mental schema in medial prefrontal cortex dramatically accelerates the consolidation of schema-congruent new learning.

3. Internalized environmental statistics.

The brain's priors encode the long-run regularities of the world. Weiss, Simoncelli and Adelson (2002) explained motion illusions as the optimal percept given a built-in prior that motion tends to be slow. The light-from-above prior shapes how you read shading as shape. This context was never consciously learned. It's baked into the generative model by a lifetime and an evolutionary history of exposure.

High-level priors descend and reshape low-level sensory processing. Prediction in predictive coding doing visible work.

Murray and colleagues (2002): perceiving a coherent shape reduces activity in V1, consistent with a higher-level interpretation explaining away the lower-level signal.

Kok, de Lange and colleagues (2012): a valid prior expectation both lowers overall V1 activity and sharpens the population's representation of the expected stimulus. A precise predictive-coding signature.

Context at the top is not an abstraction floating free of perception. It is continuously projected downward as the prediction that the sensory levels are tested against.

Hierarchical predictive processing continued:

Two stratifying caveats.

First caveat: almost all findings establish properties of high-level representation—invariance, timescale, transmodality, relational and causal structure, context-as-bias. Most predate or stand independent of the predictive-coding formalism.

Predictive processing interprets these as the priors of a single hierarchical generative model. That interpretation is productive and, in the expectation-on-sensory-cortex results, fairly directly supported.

But the broad identification of "all of this is one generative model's priors implemented by predictive coding" is a framework commitment layered onto robust phenomena, not something the phenomena prove on their own.

Second caveat: the clean ladder of monotonically increasing abstraction is an idealization.

Real cortex has parallel streams, skip connections, and graded rather than strictly tiered organization. The various senses of abstract—invariance, slowness, amodality, relational structure, causal structure—do not perfectly co-localize.

Treating "higher level = abstract cause and context" as a single well-defined tier is a useful simplification that the anatomy only roughly honors.

Hierarchical Predictive Processing

Notes from MIT 6.566 Lecture 3 on software fault isolation through WebAssembly.

Software fault isolation (SFI) enables isolation without OS or VM support. WebAssembly implements this approach by translating portable bytecode to native machine code with instrumentation that enforces safety boundaries.

You can read the unrolled version of this thread here: https://typefully.com/PNeelamraju/C4EUbuY

WebAssembly addresses portability constraints. Developers write code in C, C++, Rust, or Go and compile to WASM bytecode. The browser or runtime translates this bytecode to native x86 or ARM instructions through interpretation or JIT compilation.

The isolation model places a WASM module inside a host process. The module has exported functions callable from outside, internal state it can access, but attempts to access state outside the module boundary trigger exceptions. This boundary is software-defined rather than hardware-enforced.

A baseline implementation uses interpretation. An interpreter loop switches on each opcode, executing operations like add or store while checking bounds. Memory stores verify address < memory size before writing. This provides safety but introduces overhead from constant looping and switching.

JIT compilation improves performance by translating WASM opcodes to native instructions ahead of execution. Most WASM operations map to single native instructions. The compiler must emit instrumentation checks so the generated code remains safe when executed directly.

WASM state consists of disjoint components: code blocks, global variables, local variables per function invocation, a stack, and heap memory. These components never alias. Accessing a global never affects the heap. Modifying the heap never changes code. This separation simplifies both reasoning and translation.

The native representation maps WASM state to concrete machine structures. Code blocks translate to instrumented x86 blocks. Globals become a fixed-size array. The heap maps to a variable-size array with tracked base and size. Locals and stack values occupy the native stack.

Instrumentation adds safety checks to translated code. Global and local accesses can be statically verified at compile time because indices are fixed and counts are known. The compiler rejects out-of-bounds accesses. No runtime checks needed for these operations.

Memory operations require runtime checks. A store to address A in WASM translates to native code that first verifies A + size ≤ memory_size, then performs the store to base + A. Without virtual memory optimizations, every memory access includes this bounds check.

WASM enforces well-defined stack depth at every instruction. The compiler knows the exact stack depth at each point in the bytecode. Control flow branches must agree on stack depth when merging. This enables the JIT to generate stack access code using known offsets from the stack pointer.

Block-level instrumentation optimization hoists checks to block entry. Because WASM uses structured control flow with no mid-block jumps, the compiler can analyze all operations in a block and emit a single combined bounds check covering all accesses in that block.

Control flow enforcement requires jumps target instrumented block starts. Direct calls to known functions compile to native calls with a single stack depth check verifying sufficient space remains. Internal branches compile to direct jumps with no additional checks.

Call indirect implements function pointers through a table mechanism. Rather than storing actual addresses, WASM maintains a table of valid call targets. A function pointer stores an index. The runtime checks index < table_size, then jumps to the registered target at that slot.

This table design makes indirect call validation efficient. The compiler already verified each table entry points to properly instrumented code with correct type signatures. The runtime only checks the index is in bounds, not the target properties.

RWASM demonstrates an alternative implementation strategy. Instead of generating native code directly, translate WASM bytecode to Rust source code. Array accesses become Rust slice operations. The Rust compiler inserts bounds checks and optimizes the resulting binary.

The Rust translation leverages existing compiler infrastructure. Rust provides memory safety guarantees and optimization passes including check hoisting. The no_unsafe_code directive ensures no escape hatches exist. The compiler becomes the isolation enforcement mechanism.

WebAssembly provides isolation without hardware support. The design constraints—structured control flow, disjoint state, well-defined stack depth, explicit call tables—enable efficient native code generation with verifiable safety properties enforced through instrumentation.

Software Fault Isolation in WebAssembly (MIT 6.566 Notes)

Notes from MIT 6.566 Lecture 2 on operating system and virtual machine isolation.

Isolation limits damage when code fails or becomes compromised. Cloud providers separate VMs so one customer cannot access another. Smartphones isolate apps so a game cannot read banking data. The same machinery appears across laptops, servers, and embedded devices.

You can read the unrolled version of this thread here: https://typefully.com/PNeelamraju/u2NtI1W

Virtual memory provides the foundation. Each isolated code instance runs in its own address space using page tables. Instructions operate on private memory, keeping execution boundaries separate.

The challenge emerges when code requires OS services: file systems, networks, process creation. These syscalls cross the isolation boundary.

Linux processes offer baseline isolation. Each process runs with its own memory space and an assigned user ID. The kernel uses UIDs to control access to system objects.

File systems map UIDs to read, write, and execute permissions. Network access has no permission model by default. Process interactions follow same-UID rules.

This model was designed for multi-user systems where 50 users share one computer. It does not fit modern workloads: cloud customers with no relationship to each other, or multiple applications on a phone that should remain isolated regardless of world-readable file permissions.

Linux namespaces overlay kernel abstractions to create flexible isolation. Everything the kernel provides now exists within a namespace: file systems, network interfaces, process lists.

Before namespaces, there was one global file system and one network. Now each namespace can expose different subsets or entirely separate views.

Namespaces enable fine-grained control. Two processes in the blue namespace see one file system tree. A process in the red namespace sees a different root directory with different files. Shared files can exist in both namespaces when overlap is intentional.

Network interfaces can be assigned per namespace or withheld entirely.

LXC (Linux containers) uses namespaces for resource isolation and seccomp filters to restrict which syscalls a process can invoke. This controls the OS kernel attack surface.

LXC blocks 44 obsolete syscalls out of 350 total. More aggressive filtering breaks application compatibility. The trade-off between security and features remains unfavorable.

The kernel remains shared in LXC. Linux exposes roughly 350 syscalls, each with variants, representing millions of lines of C code. More code means more bugs. Monthly privilege escalation vulnerabilities confirm this.

Once you exploit a kernel bug, namespace isolation becomes irrelevant. The kernel enforces isolation but cannot isolate itself.

gVisor addresses this by implementing a user-space kernel called the sentry. All syscalls redirect to the sentry process instead of the host kernel. The sentry implements the network stack, file system operations, and process management in user space.

Multiple sentries run for different containers, each isolated from the others.

The sentry reduces attack surface in two ways. First, applications make zero direct syscalls to the host kernel. Second, even if an attacker compromises the sentry, it can only make 53 filtered syscalls to the host kernel, down from 350.

The sentry is written in Go to avoid buffer overflow vulnerabilities common in C.

gVisor adds a gofer process for file access. The sentry handles syscalls but delegates file operations to the gofer, which checks every file access against a configured policy.

Breaking isolation requires exploiting bugs in the sentry, then bypassing the gofer policy checker. Multiple layers make this substantially harder.

Firecracker uses virtual machines instead of shared kernel abstractions. A guest kernel runs inside the isolation boundary. Processes inside the VM interact with the guest kernel, not the host.

The external interface is hardware, not OS abstractions: virtual disk, virtual network card, serial port.

The VM interface is dramatically simpler than OS abstractions. File systems involve directories, permissions, renaming, growing, truncating, symlinking. Firecracker exposes a disk: an array of 512-byte blocks with read and write operations.

Networking in an OS includes sockets, ports, TCP, UDP, buffering, retransmission, flow control. Firecracker exposes a NIC: send packet, receive packet.

Attack surface shrinks to roughly five operations: read disk block, write disk block, send packet, receive packet, append to serial log. Compare this to 350 syscalls in Linux or the full syscall reimplementation in gVisor.

Firecracker is implemented in Rust with approximately 50,000 lines of code. Memory safety and small codebase reduce vulnerability likelihood.

Security metrics remain imperfect. Attack surface measures how many operations an adversary can attempt. Lines of code estimates bug probability. Executed code measures runtime behavior.

Firecracker may execute more distinct lines than LXC for normal workloads because isolation layers add overhead. But adversaries cannot cause substantially more code to run than normal operation allows.

Trade-offs define the choice between approaches. LXC offers maximum compatibility and performance but exposes the full kernel attack surface. gVisor reduces attack surface while maintaining compatibility through syscall reimplementation, at moderate performance cost.

Firecracker minimizes attack surface through hardware abstraction but complicates file sharing and requires pre-allocated memory for each VM.

Isolation domain size affects side-channel mitigation. Processes leak information through shared CPU caches and microarchitectural state when switching cores. The standard defense: assign domains to separate cores, flush state during switches.

Larger VM-based domains map more naturally to whole cores than small process-based containers.

These isolation mechanisms appear throughout computing: lab environments, web servers, mobile operating systems, cloud data centers. Each context selects techniques based on compatibility requirements, performance constraints, and threat model.

The foundation remains consistent: separate memory, control syscall access, minimize trusted code.

Understanding OS Isolation Techniques (MIT 6.566 Notes)

Deep learning foundations and failure modes, summarized from MIT's lecture series, distilled into X posts by Typefully's Sonnet 4.5. One of the convenient ways to prime yourself before going through the actual full lecture. Also included a couple of infographics generated by GPT.

The thread covers where models succeed, where they break, and what mechanisms enable modern generative systems.  

Sequence modeling addresses data resolved in time. Unlike feed-forward networks that process fixed inputs, sequence models handle variable-length data by maintaining internal state across time steps.

Recurrent Neural Networks introduce a recurrence relation. The hidden state h(t) depends on both the current input x(t) and the previous state h(t-1), creating a form of memory that propagates through the sequence.

The state update follows this pattern: apply weight matrices to both input and prior state, sum the results, apply nonlinear activation. The same weight matrices are reused at every time step during training.

Training RNNs requires backpropagation through time. Error flows backward from the current time step to the beginning of the sequence, necessitating repeated matrix multiplications with the weight matrix.

This creates gradient instability. Large weight values cause exploding gradients. Small weight values cause vanishing gradients, making it difficult to capture long-term dependencies in sequences.

Long Short-Term Memory networks address this by making internal computation more complex. The architecture controls information flow more precisely than standard RNNs, preserving gradient signals across longer sequences.

RNNs face three fundamental bottlenecks. The encoding bottleneck compresses all sequence information into a single state vector. Sequential processing prevents parallelization. Memory constraints limit dependency tracking.

Attention mechanisms eliminate recurrence entirely. Instead of processing sequences step by step, attention computes relevance scores across all positions simultaneously, identifying which parts of the input matter most.

Self-attention operates through three learned transformations. The positional encoding is transformed into Query, Key, and Value matrices. The Query-Key dot product computes similarity scores, scaled and normalized via softmax.

The attention weight matrix multiplies with the Value matrix to extract features. High-scoring positions receive more weight, low-scoring positions less. This operation is fully parallelizable across sequence positions.

Transformers stack multiple attention heads to capture different patterns of importance. Each head learns distinct relationships within the sequence, enabling richer representations than single-head attention.

This architecture now dominates sequence modeling. Transformers power language models like GPT, protein structure prediction in biology, and vision transformers for image processing. The mechanism generalizes across domains.

You can read the unrolled version of this thread here: https://typefully.com/PNeelamraju/fzkDPqP

Deep Learning Insights

We often wonder about our next optimal action. After all, our lives are determined by the choices we make. Decision-making frameworks galore, and it's easy to get lost in the forest of available frameworks. I've talked about a simple formula in my earlier post: https://x.com/PNeelamraju/status/1904942769799635074?s=20 

Another useful framework uses the (self-explanatory) law of least required performance (LRP). 
- This rule applies only to performative aspects.
- You conserve resources on the performance part, never skimping on the actual work. A rough estimate is 90% work and 10% demonstration. 
- You demonstrate your work only when it's time to perform. The effort you put into each of your performances is dictated by the situation at hand, as determined by you.

This is keeping in mind how, most times, we squander our resources to impress someone with very little payoff or impact because the focus on performance compromises on the actual work delivered. Using this law consistently and making it a habit will have huge resource saving over time as you carefully calibrate the work-performance ratio for every situation. Try it out. This is quite the radical approach to life in a world that values performance over substance, but hey, where's the reward if you don't take the risk? 


The Law of Least Required Performance (LRP)

Psychology series: Wilhelm Wundt 

Wilhelm Wundt founded the first experimental psychology laboratory in Leipzig in 1879 , an event that marks the conventional birth of psychology as a scientific discipline. He spent the next four decades producing one of the most ambitious bodies of work in the history of the field: a rigorous experimental program for elementary mental processes, and a ten-volume cultural psychology of language, myth, and religion. Anglophone psychology absorbed only the first half, misread it, and largely forgot the second. Of his 18 catalogued phenomena, 4 survived intact. The pattern of survival is revealing.

You can read the unrolled version of this thread here: https://typefully.com/PNeelamraju/UBixrv1

Mental chronometry is among Wundt's work that survived. Wilhelm Wundt and his Leipzig students used reaction-time differences across tasks to decompose mental acts into distinct processing stages: perception, apperception, discrimination, choice. The logic came from Donders; Wundt built it into the empirical core of his laboratory program. It is now the methodological backbone of cognitive psychology. Sternberg's additive-factors logic (1969), Posner's cuing paradigm, and Ratcliff's diffusion models all sit in direct descent. The underlying program - use reaction time to decompose mental processes - is fully alive. Wundt is a footnote.

The Blickpunkt and Blickfeld: Wundt proposed a two-zone model of consciousness with a broad field of awareness and a narrow focal point of clearly apperceived content. The distinction between focal and ambient attention is now standard across cognitive psychology. Cowan's focus-of-attention model in working memory, Posner's attention networks, the foveal/peripheral processing distinction ---> all operationalize the same structural claim. The vocabulary is entirely modern; the architecture is Wundt's.

Range of apperception. Wundt's students ran experiments to quantify how many independent items could occupy the focus of consciousness simultaneously, arriving at roughly six. Miller's 'magical number 7±2' (1956) addressed the same question and became one of the most cited papers in psychology. Cowan's revised estimate of 4±1 (2001) pushed the field further. Wundt's original estimate sits closer to Cowan's than Miller's. The contemporary working-memory-capacity literature virtually never credits Wundt as the source of the question. One of the cleaner quiet-resurrection cases in the history of psychology.

Heterogony of ends is the outlier. Wundt's claim: in the course of acting on a purpose, side-effects acquire motivational weight and become new ends, so actual goals systematically drift from the intentions that initiated them. It survived not in psychology but in sociology - Merton's sociology of unintended consequences (1936) and Boudon's rational-action theory both build on this principle. In psychology proper it has no direct heir, though goal-shifting and goal-shielding research (Shah, Friedman & Kruglanski 2002) touches the same territory. A genuine contribution that landed in the wrong discipline and was never retrieved.

The larger story: Wundt is probably the most systematically misrepresented figure in the history of psychology. E. G. Boring's influential 1929 history reduced him to a dry elementarist who counted sensations, erasing the voluntarism (the view that volitional acts, not sensations, are the paradigmatic mental processes) and the entire Völkerpsychologie program he spent 20 years writing. Several of his concepts that didn't survive empirical scrutiny are contested precisely because they were independently rediscovered under new names: apperception became the "global workspace", Völkerpsychologie became "cultural psychology", the actuality theory of mind became "enactivism". The man who invented the discipline is mostly remembered for a caricature of it.

Wilhelm Wundt: Psychology Series

To help with the challenge of "unfinished business."

The "eats shoots and leaves" joke never gets old. 

Also, Veo didn't have any trouble understanding the simple prompts, one with the commas and one without. It had the context down perfectly. 