Logit Lens — Gemma 2 2B

what's interesting is most of words come as top 4 or top3rd first and then moves to top 1

For each prompt, we apply Gemma's final RMSNorm and lm_head to the residual stream after every layer (layer 0 = input embedding only, layer 26 = full model). Each row shows the top-5 predicted next tokens at that layer with their softmax probabilities. Cells matching the final layer's top-1 prediction are highlighted in green — watch the answer crystallize as you scan down.

Newlines shown as .

Once upon a

L top 1 top 2 top 3 top 4 top 5
0 a1.000 0.000 <pad>0.000 <eos>0.000 <bos>0.000
1 a1.000 an0.000 A0.000 the0.000 <pad>0.000
2 a1.000 an0.000 A0.000 0.000 the0.000
3 a1.000 an0.000 A0.000 a0.000 the0.000
4 a1.000 an0.000 متعلقه0.000 A0.000 another0.000
5 a1.000 an0.000 RenderAtEndOf0.000 ftant0.000 richTextPanel0.000
6 a1.000 an0.000 Eine0.000 a0.000 very0.000
7 a1.000 ,0.000 the0.000 The0.000 an0.000
8 a1.000 0.000 the0.000 The0.000 ,0.000
9 a1.000 0.000 ,0.000 the0.000 an0.000
10 a0.998 0.002 0.000 ,0.000 0.000
11 a0.986 0.014 0.000 ,0.000 0.000
12 a0.622 0.377 0.000 we0.000 0.000
13 a0.905 0.095 0.000 we0.000 0.000
14 a0.817 0.182 0.000 0.000 we0.000
15 a0.995 0.005 0.000 happy0.000 few0.000
16 a0.614 ffions0.329 ratulations0.039 hematical0.004 ^(@)0.004
17 0.548 a0.377 time0.051 little0.008 few0.002
18 ratulations0.142 certain0.097 hematical0.097 unately0.097 ecute0.036
19 few0.910 time0.040 day0.017 couple0.015 warm0.006
20 few0.661 time0.130 very0.090 little0.090 day0.020
21 time1.000 time0.000 tyme0.000 few0.000 little0.000
22 time1.000 time0.000 Time0.000 times0.000 TIME0.000
23 time1.000 time0.000 Time0.000 times0.000 TIME0.000
24 time1.000 time0.000 Time0.000 times0.000 TIME0.000
25 time1.000 Time0.000 times0.000 time0.000 0.000
26 time0.998 day0.000 long0.000 dream0.000 midnight0.000

Roses are red, violets are

L top 1 top 2 top 3 top 4 top 5
0 are1.000 is0.000 <pad>0.000 <eos>0.000 <bos>0.000
1 are1.000 is0.000 were0.000 Are0.000 are0.000
2 are1.000 <bos>0.000 is0.000 were0.000 Are0.000
3 are1.000 is0.000 were0.000 ARE0.000 was0.000
4 are1.000 is0.000 were0.000 <bos>0.000 ARE0.000
5 are1.000 were0.000 is0.000 also0.000 ARE0.000
6 <bos>1.000 are0.000 were0.000 is0.000 ARE0.000
7 are0.997 is0.001 were0.001 lenker0.000 تضيفلها0.000
8 are0.845 lenker0.069 were0.054 saites0.026 is0.004
9 are0.727 were0.267 also0.005 is0.000 going0.000
10 were0.888 are0.094 lenker0.013 فريبيس0.004 iſter0.001
11 were0.721 are0.265 lenker0.013 ]")]0.001 iſter0.000
12 lenker0.994 were0.005 iſter0.000 are0.000 iettivo0.000
13 lenker0.489 فريبيس0.231 were0.204 iſter0.046 LogFactory0.007
14 were0.537 lenker0.326 gonna0.044 iſter0.030 LoggerFactory0.024
15 lenker0.999 فريبيس0.001 gonna0.000 تضيفلها0.000 صوتيه0.000
16 lenker1.000 فريبيس0.000 تضيفلها0.000 gonna0.000 IntoConstraints0.000
17 lenker1.000 فريبيس0.000 tvguidetime0.000 gonna0.000 ousand0.000
18 lenker0.999 blue0.000 gonna0.000 too0.000 red0.000
19 purple0.773 blue0.221 lenker0.004 color0.000 red0.000
20 purple0.923 blue0.076 Purple0.001 lenker0.000 purple0.000
21 blue0.940 purple0.060 violet0.000 Purple0.000 lenker0.000
22 blue0.995 purple0.005 blue0.000 green0.000 violet0.000
23 blue1.000 purple0.000 Blue0.000 violet0.000 blue0.000
24 blue1.000 purple0.000 Blue0.000 white0.000 blue0.000
25 blue1.000 Blue0.000 purple0.000 black0.000 also0.000
26 blue0.997 purple0.001 violet0.000 black0.000 Blue0.000

The quick brown fox jumps over the lazy

L top 1 top 2 top 3 top 4 top 5
0 lazy1.000 lazy0.000 laziness0.000 Lazy0.000 <pad>0.000
1 lazy1.000 lazy0.000 Lazy0.000 Lazy0.000 laziness0.000
2 lazy1.000 lazy0.000 Lazy0.000 laziness0.000 Lazy0.000
3 lazy1.000 lazy0.000 Lazy0.000 Lazy0.000 laziness0.000
4 lazy1.000 lazy0.000 Lazy0.000 Lazy0.000 laziness0.000
5 lazy1.000 Lazy0.000 lazy0.000 Lazy0.000 laziness0.000
6 lazy1.000 Lazy0.000 lazy0.000 Lazy0.000 lazily0.000
7 lazy0.999 Lazy0.001 lazy0.000 Lazy0.000 lazily0.000
8 lazy0.993 Lazy0.007 lazy0.000 Lazy0.000 lazily0.000
9 lazy0.991 Lazy0.009 lazy0.000 Lazy0.000 lazily0.000
10 lazy0.940 Lazy0.060 lazy0.000 Lazy0.000 lazily0.000
11 lazy0.953 Lazy0.047 lazy0.000 Lazy0.000 lazily0.000
12 Theſe0.479 Lazy0.290 lazy0.226 lazy0.002 Lazy0.002
13 lazy0.548 Lazy0.427 Lazy0.013 Theſe0.010 lazy0.002
14 Theſe0.994 faſt0.003 pleaſure0.002 Lazy0.000 houſe0.000
15 Theſe0.990 faſt0.007 pleaſure0.001 Lazy0.001 lazy0.000
16 Theſe1.000 faſt0.000 pleaſure0.000 houſe0.000 Jefus0.000
17 Theſe1.000 lazy0.000 pleaſure0.000 faſt0.000 Lazy0.000
18 Theſe1.000 faſt0.000 pleaſure0.000 Jefus0.000 lazy0.000
19 Theſe1.000 pleaſure0.000 Lazy0.000 lazy0.000 faſt0.000
20 Theſe0.961 lazy0.037 lazy0.001 houſe0.001 Lazy0.000
21 Theſe0.987 dog0.011 houſe0.000 ſmall0.000 sleeping0.000
22 dog0.899 Theſe0.095 dogs0.005 faſt0.000 dog0.000
23 dog0.996 dogs0.003 cat0.000 dog0.000 sleeping0.000
24 dog0.999 white0.000 cat0.000 0.000 red0.000
25 dog0.999 dogs0.001 0.000 0.000 red0.000
26 dog0.975 dogs0.008 brown0.004 red0.001 0.001

Romeo and

L top 1 top 2 top 3 top 4 top 5
0 and1.000 0.000 ,0.000 (0.000 <pad>0.000
1 and1.000 and0.000 or0.000 ,0.000 And0.000
2 and1.000 <bos>0.000 and0.000 AND0.000 &0.000
3 and1.000 and0.000 And0.000 AND0.000 0.000
4 and1.000 <bos>0.000 and0.000 AND0.000 And0.000
5 and1.000 and0.000 LookAnd0.000 \&0.000 AND0.000
6 <bos>1.000 and0.000 and0.000 0.000 LookAnd0.000
7 and1.000 LookAnd0.000 and0.000 0.000 RunWith0.000
8 and0.997 printStackTrace0.001 liſh0.001 aand0.000 ratulations0.000
9 and0.982 liſh0.010 그리고0.002 ratulations0.002 and0.001
10 and0.960 достатки0.020 and0.004 liſh0.003 RunWith0.001
11 and0.982 достатки0.012 liſh0.001 ratulations0.001 RunWith0.000
12 and0.792 достатки0.095 ratulations0.035 ffions0.014 liſh0.013
13 and0.906 достатки0.035 ffions0.021 liſh0.013 ſs0.010
14 liſh0.644 достатки0.127 and0.099 ffions0.068 ſs0.008
15 достатки0.530 liſh0.364 ffions0.049 and0.023 ratulations0.009
16 liſh0.409 ofition0.219 ffions0.150 ratulations0.133 достатки0.055
17 and0.665 достатки0.168 liſh0.033 ofition0.029 the0.020
18 and0.353 his0.167 the0.147 0.079 dorfer0.054
19 the0.335 ofition0.296 liſh0.085 ffions0.040 I0.028
20 tershire0.556 Romeo0.059 ofition0.052 ++]=0.046 Romeo0.040
21 ftant0.303 tershire0.236 Juliet0.162 other0.098 dorfer0.046
22 Juliet0.783 &___0.039 rieux0.030 ftant0.030 tershire0.024
23 Juliet0.884 his0.050 Juliet0.027 rieux0.013 tershire0.007
24 Juliet0.545 I0.201 the0.138 his0.107 Juliet0.005
25 Juliet0.999 the0.001 juli0.000 I0.000 his0.000
26 Juliet1.000 Juliette0.000 juli0.000 Julia0.000 Julie0.000

Mary had a little

L top 1 top 2 top 3 top 4 top 5
0 little1.000 little0.000 <pad>0.000 <eos>0.000 <bos>0.000
1 little1.000 little0.000 Little0.000 LITTLE0.000 Little0.000
2 little1.000 little0.000 Little0.000 LITTLE0.000 Little0.000
3 little1.000 little0.000 LITTLE0.000 Little0.000 Little0.000
4 little1.000 little0.000 Little0.000 LITTLE0.000 LITTLE0.000
5 little1.000 little0.000 LITTLE0.000 Little0.000 LITTLE0.000
6 little1.000 little0.000 LITTLE0.000 Little0.000 Little0.000
7 little0.999 little0.001 LITTLE0.000 LITTLE0.000 Little0.000
8 little0.998 little0.002 LITTLE0.000 LITTLE0.000 Little0.000
9 little0.993 little0.007 LITTLE0.000 LITTLE0.000 Little0.000
10 little0.915 little0.075 LITTLE0.008 Little0.001 LITTLE0.001
11 <>",0.597 little0.362 little0.020 LITTLE0.008 ftance0.005
12 <>",0.895 ftance0.057 neſs0.027 little0.013 Theſe0.003
13 ftance0.303 <>",0.303 neſs0.184 little0.143 !") 0.022
14 ftance0.695 little0.050 ftant0.050 !") 0.027 neſs0.021
15 فريبيس0.628 Distribuzione0.124 صوتيه0.066 <>",0.058 spare0.035
16 LITTLE0.774 setupUi0.082 فريبيس0.023 <>",0.023 iſt0.021
17 little0.794 LITTLE0.122 classnames0.074 little0.005 bit0.003
18 LITTLE0.553 little0.430 bit0.008 little0.004 LITTLE0.003
19 bit0.967 little0.018 LITTLE0.011 Little0.003 LITTLE0.000
20 little0.447 bit0.239 LITTLE0.145 Little0.100 baby0.042
21 little0.804 LITTLE0.085 bit0.066 tiny0.019 Little0.015
22 little0.708 baby0.123 bit0.096 LITTLE0.058 girl0.005
23 little0.759 baby0.169 bit0.049 tiny0.016 girl0.005
24 0.590 little0.217 bit0.091 baby0.080 ,0.007
25 lamb0.954 0.020 baby0.011 0.003 little0.003
26 lamb0.998 Lamb0.002 baby0.000 brother0.000 sister0.000

ABC, easy as 1, 2,

L top 1 top 2 top 3 top 4 top 5
0 ,1.000 0.000 .0.000 <pad>0.000 <eos>0.000
1 ,1.000 .0.000 0.000 ↵↵0.000 ,0.000
2 ,1.000 <bos>0.000 .,0.000 ,0.000 0.000
3 kasarigan0.999 .$,0.001 BeginContext0.000 parsedMessage0.000 MigrationBuilder0.000
4 kasarigan0.464 виправивши0.362 beginnetje0.171 AssemblyCulture0.001 MigrationBuilder0.001
5 kasarigan0.440 sanitaires0.087 ,<0.077 ,-,0.053 urma0.041
6 <bos>1.000 and0.000 beginnetje0.000 ,0.000 ٬0.000
7 beginnetje0.586 følgelig0.168 ViewFeatures0.079 DockStyle0.062 IntoConstraints0.048
8 rapides0.807 beginnetje0.096 ViewFeatures0.052 parsedMessage0.013 AssemblyCulture0.007
9 IntoConstraints0.530 rapides0.322 følgelig0.038 (",",0.021 beginnetje0.018
10 RenderAtEndOf0.787 utafitiHapana0.155 essayé0.009 rapides0.008 ImageContext0.008
11 Abt0.184 sometimes0.111 and0.068 parfois0.068 stepping0.060
12 rapides0.711 جغرافيا0.096 magique0.019 obé0.019 pleins0.017
13 and0.631 rapides0.141 pleins0.046 we0.036 obé0.015
14 and0.926 or0.025 we0.013 parfois0.007 yes0.006
15 and0.272 maybe0.187 dorfer0.088 같이0.069 的样子0.054
16 and0.392 rapides0.210 tranquille0.099 sanitaires0.088 sauvages0.077
17 and0.909 0.085 or0.001 rapides0.001 sanitaires0.001
18 0.996 and0.003 maybe0.000 sometimes0.000 sanitaires0.000
19 0.999 and0.000 maybe0.000 sometimes0.000 please0.000
20 0.998 maybe0.002 yes0.000 and0.000 sometimes0.000
21 1.000 three0.000 maybe0.000 30.000 third0.000
22 1.000 three0.000 and0.000 trois0.000 or0.000
23 1.000 three0.000 easy0.000 trois0.000 and0.000
24 1.000 three0.000 and0.000 or0.000 30.000
25 1.000 three0.000 30.000 and0.000 Three0.000
26 0.984 and0.004 three0.004 30.003 ABC0.000

Water freezes at zero degrees

L top 1 top 2 top 3 top 4 top 5
0 degrees1.000 Degrees0.000 degrees0.000 degree0.000 <pad>0.000
1 degrees1.000 Degrees0.000 degrees0.000 degree0.000 degree0.000
2 degrees1.000 Degrees0.000 degree0.000 degrees0.000 Degrees0.000
3 degrees1.000 Degrees0.000 degrees0.000 degree0.000 Degrees0.000
4 degrees1.000 Degrees0.000 degrees0.000 degree0.000 Degrees0.000
5 degrees1.000 Degrees0.000 degrees0.000 degree0.000 Degrees0.000
6 degrees1.000 Degrees0.000 degrees0.000 <bos>0.000 degree0.000
7 degrees1.000 degrees0.000 Degrees0.000 degree0.000 degree0.000
8 degrees0.999 degrees0.000 Degrees0.000 Degrees0.000 degree0.000
9 degrees0.994 degrees0.003 Degrees0.001 myſelf0.001 Fahrenheit0.000
10 degrees0.757 degrees0.103 myſelf0.080 againſt0.029 ſte0.014
11 itſelf0.427 myſelf0.427 pleaſure0.074 againſt0.058 Monfieur0.004
12 itſelf0.695 myſelf0.256 Monfieur0.035 pleaſure0.006 Efq0.005
13 degrees0.900 degrees0.045 ſte0.035 pleaſure0.008 againſt0.007
14 againſt0.738 ſte0.145 pleaſure0.078 degrees0.022 degrees0.004
15 from0.418 ſte0.224 and0.106 or0.064 degrees0.044
16 Fahrenheit0.641 degrees0.184 degrees0.041 fahren0.036 and0.025
17 againſt0.804 becauſe0.051 Jefus0.051 myſelf0.015 degrees0.011
18 temperature0.391 becauſe0.185 Celsius0.163 againſt0.087 purpoſe0.041
19 temperature0.545 and0.424 Celsius0.011 because0.008 Fahrenheit0.003
20 and0.941 .0.022 Celsius0.020 fahren0.006 temperature0.002
21 and0.684 Celsius0.173 Fahrenheit0.056 fahren0.050 .0.018
22 and0.914 .0.058 Celsius0.019 on0.004 in0.002
23 .0.803 and0.179 in0.007 ,0.007 on0.003
24 and0.559 .0.339 ,0.059 in0.028 0.010
25 and0.522 Celsius0.317 .0.150 ,0.010 on0.001
26 Celsius0.998 celsius0.001 Fahrenheit0.000 and0.000 cel0.000

The Eiffel Tower is in

L top 1 top 2 top 3 top 4 top 5
0 in1.000 0.000 ,0.000 (0.000 <pad>0.000
1 in1.000 In0.000 in0.000 IN0.000 In0.000
2 in1.000 <bos>0.000 In0.000 IN0.000 In0.000
3 in1.000 In0.000 IN0.000 in0.000 on0.000
4 in1.000 at0.000 IN0.000 in0.000 on0.000
5 in1.000 IN0.000 in0.000 In0.000 In0.000
6 <bos>0.881 in0.119 on0.000 the0.000 IN0.000
7 in1.000 IN0.000 on0.000 in0.000 In0.000
8 in1.000 a0.000 on0.000 In0.000 the0.000
9 in1.000 on0.000 a0.000 at0.000 In0.000
10 in1.000 on0.000 a0.000 In0.000 very0.000
11 in1.000 on0.000 a0.000 all0.000 In0.000
12 in1.000 all0.000 a0.000 at0.000 very0.000
13 in1.000 a0.000 all0.000 very0.000 on0.000
14 in1.000 a0.000 very0.000 all0.000 the0.000
15 in1.000 on0.000 the0.000 a0.000 at0.000
16 in0.999 a0.001 the0.001 all0.000 on0.000
17 in0.865 the0.117 a0.014 all0.004 at0.000
18 in0.652 a0.240 the0.088 0.020 all0.000
19 the0.663 a0.313 0.020 all0.002 in0.001
20 the0.825 a0.143 0.025 all0.007 in0.000
21 the0.966 a0.029 all0.003 0.001 front0.000
22 the0.992 a0.007 fact0.001 all0.000 its0.000
23 the0.994 a0.003 Paris0.002 all0.000 full0.000
24 the0.995 a0.004 full0.000 0.000 all0.000
25 the0.970 Paris0.023 full0.004 fact0.001 France0.001
26 Paris0.851 the0.148 a0.000 France0.000 full0.000

def fibonacci(n): if n <= 1: return

L top 1 top 2 top 3 top 4 top 5
0 return1.000 return0.000 <pad>0.000 <eos>0.000 <bos>0.000
1 return1.000 return0.000 Return0.000 returns0.000 Return0.000
2 return1.000 return0.000 Return0.000 returns0.000 RETURN0.000
3 return1.000 return0.000 Return0.000 RETURN0.000 returns0.000
4 return1.000 return0.000 <bos>0.000 Return0.000 returns0.000
5 return1.000 return0.000 Return0.000 RETURN0.000 returns0.000
6 return1.000 return0.000 <bos>0.000 Return0.000 returns0.000
7 return1.000 return0.000 Return0.000 RETURN0.000 returns0.000
8 return1.000 return0.000 Return0.000 returns0.000 RETURN0.000
9 return1.000 return0.000 ContentAsync0.000 nahilalakip0.000 returning0.000
10 return1.000 return0.000 GOTREF0.000 returns0.000 Return0.000
11 return1.000 return0.000 iffion0.000 VIRONMENT0.000 goutte0.000
12 return0.995 mourir0.002 iffion0.001 विश्वसनीयता0.000 ſmall0.000
13 return0.450 विश्वसनीयता0.350 mourir0.188 honte0.002 étanche0.002
14 विश्वसनीयता0.585 nahilalakip0.355 IsContent0.033 return0.023 writerow0.001
15 विश्वसनीयता0.591 nahilalakip0.358 IsContent0.020 enumi0.011 return0.008
16 nahilalakip0.915 extAlignment0.028 honte0.024 IsContent0.012 AssemblyCulture0.009
17 ViewImports0.956 EconPapers0.022 الاطلاع0.011 religieux0.004 valable0.002
18 viewDidLoad0.480 IsContent0.200 médec0.121 numerusform0.039 новништво0.027
19 pleaſure0.733 ſtate0.163 AssemblyCulture0.060 nahilalakip0.015 Бахар0.009
20 AssemblyCulture0.532 ſtate0.323 pleaſure0.072 purpoſe0.021 spesies0.018
21 AssemblyCulture0.998 spesies0.002 CreateTagHelper0.000 RegressionTest0.000 IsContent0.000
22 AssemblyCulture1.000 spesies0.000 GOTREF0.000 awtextra0.000 nourriture0.000
23 AssemblyCulture0.994 HasFactory0.004 RenderAtEndOf0.001 spesies0.001 awtextra0.000
24 n1.000 0.000 N0.000 [0.000 None0.000
25 n0.932 0.068 [0.000 0.000 "0.000
26 n0.823 0.152 [0.007 (0.004 (0.002

import numpy as

L top 1 top 2 top 3 top 4 top 5
0 as1.000 0.000 in0.000 ,0.000 (0.000
1 as1.000 As0.000 as0.000 AS0.000 sebagai0.000
2 as1.000 As0.000 as0.000 AS0.000 As0.000
3 as1.000 As0.000 as0.000 AS0.000 readAs0.000
4 as1.000 agaimana0.000 AssemblyCulture0.000 Kariera0.000 AS0.000
5 ValueStyle0.703 autorytatywna0.202 as0.058 الاطلاع0.035 CreateTagHelper0.001
6 as0.594 agaimana0.219 autorytatywna0.103 <bos>0.034 estimés0.026
7 nahilalakip0.913 snippetHide0.035 autorytatywna0.028 ValueStyle0.017 estimés0.006
8 estimés0.315 rêver0.245 snippetHide0.131 nahilalakip0.131 lenker0.055
9 nahilalakip0.405 as0.245 agaimana0.131 rêver0.080 rachtet0.070
10 as0.958 rêver0.029 agaimana0.003 nélk0.003 スポンサーリンク0.001
11 as0.972 rêver0.023 créativité0.003 スポンサーリンク0.001 nélk0.000
12 as0.979 lenker0.011 créativité0.007 rachtet0.001 suivants0.001
13 as0.592 rachtet0.091 suivants0.049 médicaux0.049 créativité0.038
14 as0.609 XmlAccessorType0.254 rachtet0.030 créativité0.014 rêver0.014
15 as0.387 suivantes0.142 suivants0.142 ISHOP0.111 rêver0.041
16 laude0.927 lenker0.052 ISHOP0.010 rêver0.002 rachtet0.002
17 laude0.765 lenker0.219 rêver0.005 suivantes0.002 chrétienne0.001
18 lenker0.764 laude0.117 כן0.063 rêver0.030 ISHOP0.005
19 np0.749 numpy0.042 UOUS0.033 ious0.018 lenker0.014
20 np0.977 numpy0.014 num0.003 yaszt0.001 pecially0.001
21 np0.945 lenker0.037 numpy0.017 yaszt0.000 AssemblyCulture0.000
22 np0.996 🏻‍♀️0.002 numpy0.001 IUrlHelper0.000 Normdatei0.000
23 np0.968 🏻‍♀️0.023 IUrlHelper0.004 numpy0.004 Normdatei0.000
24 ^(@)0.667 np0.245 Theſe0.070 NUMX0.005 myſelf0.003
25 np1.000 numpy0.000 NP0.000 py0.000 N0.000
26 np0.999 n0.000 N0.000 numpy0.000 0.000