Bård StandalAI and Power
Menu

What is AI?

The AI Act entered into force on 1 August 2024.

Published on 12 July 2024, Regulation (EU) 2024/1689 entered into force twenty days later under Article 113.1 Adding that official text to the current input leaves the trained weights unchanged.

Illustrative current input

When did Regulation (EU) 2024/1689 enter into force?

Official Journal · 12 July 2024

Article 113: entry into force on the twentieth day after publication.

Illustrative output · no source supplied

1 August 2024

The date is correct, but the current input contains no official record.
Illustrative output · Article 113 supplied

12 July + 20 days 1 August 2024

The date can now be checked against the text in the current input.
Trained weights
unchanged
Current input
question only

Adding a source makes the basis for this answer inspectable. It does not guarantee that a model has applied the source correctly.

Model and service

The product determines what reaches the model.

The interface passes the question to software operated by the provider. That software may retrieve Article 113 and place it in the current input. The model then calculates an output. The institution using the service decides whether that output may affect its work.

  1. 01 · UserAsks when the regulation entered into forceQuestion
  2. 02 · Provider softwareAssembles the current inputQuestion + optional record
  3. 03 · Model checkpointCalculates token probabilitiesTrained weights stay fixed
  4. 04 · InterfacePresents the generated answer1 August 2024
  5. 05 · InstitutionSets the consequence of using itUse is decided here

An AI model is the learned mathematical component. The wider system also includes software that carries inputs and outputs toward a human-defined objective.2

For a legal or regulatory answer, I would start with what entered the current input and which record can be checked. The model name alone does not answer either question.

Inference

The Transformer calculates probabilities for the next token.

During ordinary inference, the trained parameters stay fixed. The question and any supplied record become the current context. The model extends that context one token at a time.

Current input

Question When did Regulation (EU) 2024/1689 enter into force?

Record Published 12 July 2024. Entry into force on the twentieth day after publication.

  1. 01Tokens
    Article4821113910twentieth7415day1830publication6924

    The tokenizer turns text into identifiers.

    A token may cover a complete word. It may instead cover part of one or a punctuation mark. The split and identifiers shown here are illustrative; another tokenizer can divide the same text differently.3

  2. 02Vectors
    Article
    Illustrative values
    1. 0.14
    2. −0.82
    3. 0.05
    4. 0.41
    5. …
    6. −0.23

    An embedding supplies a learned numerical representation.

    Each identifier selects a vector to which positional information is added, allowing the model to distinguish where a token appears. Later layers change the representation according to the surrounding context.

  3. 03Attention
    12 July 2024twentieth daypublication
    entry into forceRelative paths shown for explanation, not measured model weights

    Attention mixes information from visible positions.

    Queries are compared with keys to determine how values are combined. In a causal language model, a position cannot use future tokens. An attention pattern is an internal calculation, not automatically an explanation of the answer.4, 5

  4. 04Layers
    Block 01attentionfeed-forward
    Block 02attentionfeed-forward
    Block …attentionfeed-forward
    Block Nattentionfeed-forward

    The same block pattern is applied through many learned layers.

    Each block combines context through attention and transforms each position through a feed-forward network, with residual paths carrying earlier information around both operations. “Large” has no universal threshold; an LLM is a language model trained at substantial scale with many learned parameters.6

  5. 05Next token
    Illustrative ranking
    August
    July
    September

    Augustjoins the context

    Scores become a probability distribution over the vocabulary.

    A decoding rule selects a token. That token is appended to the context and the model runs again. Next-token prediction uses the whole visible context; it is not a lookup of the immediately preceding word.

Training

Training changed the weights used for inference.

The opening answer used a saved model checkpoint. During pretraining, examples supplied the next token in advance. Repeated updates changed the model's parameters before anyone asked the current question.

Illustrative training values

The regulation entered into force in August.

The sentence is the context. “August” is the known target; the displayed values are constructed.
Model weightsstarting values
July31%
the24%
August18%
September12%
other tokens15%
Target probability
18%
Cross-entropy loss
1.715
Parameters
not yet updated

The target receives 18 percent. Loss turns that target probability into one number.

BeforeAugust 18%loss 1.715 · starting weights
After one updateAugust 32%loss 1.139 · changed weights

  1. 01Forward pass

    The current weights produce a probability for every possible next token.

  2. 02Loss

    For the target probability 0.18, cross-entropy is −ln(0.18), or 1.715.

  3. 03Backpropagation

    The chain rule calculates how a small change in each parameter would affect the loss.

  4. 04Optimiser

    The optimiser uses those gradients to change the parameters by a small amount.

  5. 05Next batch

    Another set of examples starts another update.

A gradient is a local measure of sensitivity near the current parameter values. Basic gradient descent subtracts the gradient multiplied by a learning rate. Modern training usually uses related optimisers that change how the update is calculated.7

Backpropagation calculates the gradients; the optimiser applies the update. The distinction matters because neither operation occurs during an ordinary chat response.8

Training

Examples change the saved weights.

context + known target → loss → gradients → parameter update

Inference

The saved weights extend the current context.

context → next-token scores → selected token → longer context

1986–2022

Researchers applied gradient-based training to successive neural-network architectures.

The training loop predates chat assistants. Researchers changed the architecture and trained on much larger collections of examples. GPU computation made larger runs practical.

  1. 1986

    Backpropagation made multilayer weight adjustment practical.

    The 1986 paper described how error information could be carried backwards through a network so that internal representations could be learned.8

  2. 1989
    one learned filter · three positions

    Yann LeCun and his Bell Labs colleagues trained shared filters on postal digits.

    LeCun is a computer scientist and professor at NYU. The 1989 system combined backpropagation with local connections whose weights were shared across image positions. It read handwritten US ZIP-code digits.9, 10

  3. 2012

    ImageNet and GPUs supported a much larger training run.

    AlexNet trained a deep convolutional network on graphics processors and produced a large improvement on the ImageNet object-recognition benchmark.1112

  4. 2014

    Attention let a sequence model use different source positions for each output.

    A neural translation system assigned changing weight to the source positions relevant to the next output instead of relying on one fixed summary of the sentence.13

  5. 2017

    The Transformer organised the architecture around repeated attention blocks.

    The original encoder-decoder design removed sequence-aligned recurrent layers. Stacked attention and position-wise feed-forward networks took their place.4

  6. 2018–20

    Generative pretraining made one checkpoint adaptable to many language tasks.

    GPT applied autoregressive Transformer pretraining before task adaptation. GPT-3 later showed tasks specified through examples in the prompt at much greater scale.1415

  7. 2022

    Further training and a dialogue interface produced the modern assistant.

    InstructGPT used demonstrations and ranked model outputs after pretraining. ChatGPT placed a related model behind a dialogue interface; the model remained one component of the service.1617

Institutional use

The institution decides whether the generated answer enters its work.

The same checkpoint can be placed in services that supply different records. Provider instructions can also differ, as can the consequence attached to the generated text.

Large language models can answer many questions without a retrieved record. Their learned representations support tasks that were not specified one by one during training.15 I do not think “autocomplete” is an adequate account of those capabilities.

That objection does not resolve the institution's problem. Even a correct sentence does not identify its source or show how the service behaves under the conditions in which it will be used. I therefore treat the trained model and the surrounding service as separate objects of analysis.

The provider chooses what context reaches the model and how the answer is presented. The institution using the service decides whether the output may affect a file or a person.

Institutional use

The output may enter a file or affect a person.

Provider service

Instructions and material supplied through the interface

Model checkpoint

Fixed weights calculate the next-token distribution

The physical system

Inference runs on processors and memory.

Accelerator memory holds the saved weights and intermediate values while processors run the matrix operations. The equipment requires electricity and cooling before the service can return another token.

Continue to The system
Sectional drawing of the physical infrastructure beneath a model service.
Checkpoint in accelerator memoryElectricity and cooling keep the machine available

Sources

Records and papers

Open the source list
  1. Regulation (EU) 2024/1689. EUR-Lex. Official Journal text published on 12 July 2024. Article 113 sets entry into force on the twentieth day after publication.
  2. What is AI?. OECD.AI. Distinguishes an AI model from the wider system that uses inputs and outputs.
  3. SentencePiece: A simple and language independent subword tokenizer and detokenizer. arXiv. Supports the distinction between words and model tokens.
  4. Attention Is All You Need. NeurIPS. Introduces the Transformer architecture built around attention.
  5. Attention is not Explanation. NAACL. Tests the limits of treating attention weights as explanations.
  6. Scaling laws for neural language models. arXiv. Reports empirical relationships between loss and training scale.
  7. Deep Learning. MIT Press. Reference for gradient-based optimisation and feed-forward networks.
  8. Learning representations by back-propagating errors. Nature. Describes repeated weight adjustment to reduce output error in multilayer networks.
  9. Backpropagation applied to handwritten zip code recognition. Neural Computation. Documents the Bell Labs convolutional network used on handwritten postal digits.
  10. Yann LeCun faculty profile. New York University. Current institutional biography and research field.
  11. ImageNet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition. Describes the labelled image dataset used for large-scale object recognition.
  12. ImageNet classification with deep convolutional neural networks. NeurIPS. Reports the GPU-trained convolutional network and its benchmark results.
  13. Neural machine translation by jointly learning to align and translate. arXiv. Introduces a mechanism that assigns changing weight to relevant source positions.
  14. Improving language understanding by generative pre-training. OpenAI. Shows generative Transformer pretraining followed by task adaptation.
  15. Language models are few-shot learners. NeurIPS. Documents GPT-3 scale and tasks specified through examples in the prompt.
  16. Training language models to follow instructions with human feedback. NeurIPS. Describes instruction tuning and preference-based post-training while reporting remaining mistakes.
  17. Introducing ChatGPT. OpenAI. Records the November 2022 research release and its relationship to InstructGPT.