Properties of the sample
MNIST - handwritten digits/Properties of the sample
Sections
All pixels
The training sample has a size of 60,000. We shall take every eighth event from the sample to make a more convenient size of 7,500. First load the training sample $A$ (see History and HistoryRepa),
from NISTDev import *
(uu,hrtr) = nistTrainIO()
digit = VarStr("digit")
vv = uvars(uu)
vvl = sset([digit])
vvk = vv - vvl
len(hrvars(hrtr))
785
hrsize(hrtr)
60000
Now take a subset,
hr = hrev([i for i in range(hrsize(hrtr)) if i % 8 == 0],hrtr)
hrsize(hr)
7500
The system is $U$. The sample substrate variables are $V = \mathrm{vars}(A)$, the label variables are $V_{\mathrm{l}} = \{\mathrm{digit}\}$, and the query variables form the remainder, $V_{\mathrm{k}} = V \setminus V_{\mathrm{l}}$.
The query variable valency is 256, $\{|U_w| : w \in V_{\mathrm{k}}\} = \{256\}$,
sset([vol(uu,sset([w])) for w in vvk])
# {256}
The label variable valency is $|U_{\mathrm{digit}}| = 10$,
vol(uu,vvl)
10
rpln(aall(araa(uu,hrred(hr,vvl))))
# ({(digit, 0)}, 743 % 1)
# ({(digit, 1)}, 822 % 1)
# ({(digit, 2)}, 738 % 1)
# ({(digit, 3)}, 745 % 1)
# ({(digit, 4)}, 746 % 1)
# ({(digit, 5)}, 720 % 1)
# ({(digit, 6)}, 707 % 1)
# ({(digit, 7)}, 767 % 1)
# ({(digit, 8)}, 754 % 1)
# ({(digit, 9)}, 758 % 1)
We can image any event by creating a bitmap. There are several utility functions in NISTDev for creating and manipulating bitmaps from a HistoryRepa. For example, this generates bitmaps of the first 25 events placed in a row:
hr1 = hrhrred(hr,vvk)
file = "NIST.bmp"
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,256,hrev([i],hr1))) for i in range(25)]))
The entire history can be averaged together, $\hat{A}\%V_{\mathrm{k}}$,
bmwrite(file,bmborder(1,hrbm(28,3,256,hr1)))
The pixel variables can be bucketed to reduce the valency from 256 to 2, by partitioning at 128,
(uu,hrtr) = nistTrainBucketedIO(2)
digit = VarStr("digit")
vv = uvars(uu)
vvl = sset([digit])
vvk = vv - vvl
hrsize(hrtr)
60000
hr = hrev([i for i in range(hrsize(hrtr)) if i % 8 == 0],hrtr)
hrsize(hr)
7500
The query variable valency is 2, $\{|U_w| : w \in V_{\mathrm{k}}\} = \{2\}$,
sset([vol(uu,sset([w])) for w in vvk])
# {2}
The first 25 events placed in a row now look slightly coarser,
hr1 = hrhrred(hr,vvk)
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,2,hrev([i],hr1))) for i in range(25)]))
Again, the entire history can be averaged together, $\hat{A}\%V_{\mathrm{k}}$,
hrbmav = hrbm(28,3,2,hr1)
bmwrite(file,bmborder(1,hrbmav))
We can show the average image for each digit, $\{\hat{B}\%V_{\mathrm{k}} : S \in V_{\mathrm{l}}^{\mathrm{CS}},~B = A * \{S\}^{\mathrm{U}}\}$,
bmwrite(file,bmhstack([bmborder(1,hrbm(28,2,2,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for ss in cart(uu,vvl)]))
Now consider how highly aligned variables might be grouped together. See Entropy and alignment. We will use the tupler to group together highly aligned variables in the substrate, $V$.
Create a shuffled sample, $A_{\mathrm{r}}$,
hrr = historyRepasShuffle_u(hr,1)
hrsize(hrr)
7500
bmwrite(file,bmborder(1,hrbm(28,3,2,hrhrred(hrr,vvk))))
The average shuffle looks exactly the same as the average sample, but individual events are, in fact, randomised,
hrr1 = hrhrred(hrr,vvk)
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,2,hrev([i],hrr1))) for i in range(25)]))
Now optimise the shuffle content alignment with the tuple set builder, $I_{P,U,\mathrm{B,ns,me}}$, \[ \{(\mathrm{algn}(A\%K)-\mathrm{algn}(A_{\mathrm{r}}\%K),~K) : ((K,\cdot,\cdot),\cdot) \in I_{P,U,\mathrm{B,ns,me}}^{ * }((V,~\emptyset,~A,~A_{\mathrm{r}}))\} \]
def buildtup(xmax,omax,bmax,uu,vv,xx,xxrr):
return list(reversed(list(sset([(algn(rraa(uu,hrred(xx,kk))) - algn(rraa(uu,hrred(xxrr,kk))), kk) for ((kk,_),_) in parametersSystemsBuilderTupleNoSumlayerMultiEffectiveRepa_ui(xmax,omax,bmax,1,uu,vv,fudEmpty(),xx,hrhx(xx),xxrr,hrhx(xxrr))[0]]))))
rpln(buildtup(2**2,10,10,uu,vv,hr,hrr))
# (2291.6372825038998, {<13,9>, <14,9>})
# (2233.4541392531537, {<14,9>, <15,9>})
# (2188.0496248784257, {<14,8>, <15,8>})
# (2172.5493410184863, {<12,9>, <13,9>})
# (2138.315236820694, {<23,11>, <23,12>})
# (2134.5397923793207, {<23,12>, <23,13>})
# (2132.7826762610566, {<12,10>, <13,10>})
# (2130.8430218366993, {<15,9>, <16,9>})
# (2123.8248034136413, {<17,20>, <18,20>})
# (2122.7658721083135, {<14,10>, <15,10>})
ll = buildtup(2**12,10,10,uu,vv,hr,hrr)
rpln(ll)
# (23407.929708607575, {<10,10>, <10,11>, <11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <14,9>, <14,10>})
# (23349.464303283552, {<11,9>, <11,10>, <12,9>, <12,10>, <13,8>, <13,9>, <13,10>, <14,8>, <14,9>, <14,10>, <15,9>, <15,10>})
# (23297.792788153452, {<11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <13,11>, <14,9>, <14,10>, <14,11>})
# (23263.852635704003, {<10,9>, <10,10>, <10,11>, <11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <14,9>})
# (23260.13861789893, {<11,9>, <11,10>, <12,9>, <12,10>, <13,9>, <13,10>, <14,8>, <14,9>, <14,10>, <15,8>, <15,9>, <15,10>})
# (23212.088352065173, {<10,10>, <10,11>, <11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <13,11>, <14,9>})
# (23193.60656502565, {<11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <13,11>, <14,9>, <14,10>, <15,9>})
# (23190.114339541982, {<10,10>, <11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <13,11>, <14,9>, <14,10>})
# (23154.901612698726, {<10,10>, <10,11>, <11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <14,9>, <15,9>})
# (23153.444862459703, {<10,11>, <11,9>, <11,10>, <11,11>, <12,9>, <12,10>, <12,11>, <13,9>, <13,10>, <13,11>, <14,9>, <14,10>})
Let us image the most highly aligned tuple, \[ Q_1~=~\{\mathrm{<10,10>},\dots,~\mathrm{<14,10>}\} \]
def qqhr(d,uu,vv,qq):
return aahr(uu,single(llss([(v, ValInt(d-1)) for v in qq] + [(v, ValInt(0)) for v in (vv-qq)]),1))
qq1 = ll[0][1]
bmwrite(file,bmborder(1,hrbm(28,3,2,qqhr(2,uu,vvk,qq1))))
Note that the coordinates <x,y> are defined so that the x coordinate increases from top row 1 to bottom row 28, and the y coordinate runs from left column 1 to right column 28.
Showing it overlaid on the average,
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq1)))))
Showing it overlaid on the average for each digit,
bmwrite(file,bmhstack([bmborder(1,bmmax(hrbm(28,2,2,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk)),0,0,hrbm(28,2,2,qqhr(2,uu,vvk,qq1)))) for ss in cart(uu,vvl)]))
We can see that this substrate tuple appears in the top loop where a three would differ from an eight.
Now let’s image the first 20 states ordered by size descending, $\mathrm{top}(20)(A\%Q_1)$,
pp = list(reversed(list(sset([(b,a) for (a,b) in aall(araa(uu,hrred(hr,qq1)))]))))[:20]
[int(a) for (a,b) in pp]
# [2656, 317, 175, 146, 123, 114, 111, 94, 92, 90, 89, 85, 83, 73, 72, 72, 68, 65, 63, 60]
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,2,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for (_,ss) in pp]))
The first state is the largest slice. In this case all of the pixels of the tuple are background, so it includes all images where this region is blank. The remaining states all have at least one foreground pixel in this region, for which there are images for most digits except one.
Now optimise again having removed the top tuple from the substrate, and \[ \{(\mathrm{algn}(A\%K)-\mathrm{algn}(A_{\mathrm{r}}\%K),~K) : ((K,\cdot,\cdot),\cdot) \in I_{P,U,\mathrm{B,ns,me}}^{ * }((V \setminus Q_1,~\emptyset,~A,~A_{\mathrm{r}}))\} \]
ll = buildtup(2**12,10,10,uu,vv-qq1,hr,hrr)
rpln(ll)
# (22974.379106386157, {<21,10>, <21,11>, <22,10>, <22,11>, <22,12>, <23,10>, <23,11>, <23,12>, <23,13>, <24,11>, <24,12>, <24,13>})
# (22967.80715755579, {<21,10>, <21,11>, <21,12>, <21,13>, <22,10>, <22,11>, <22,12>, <22,13>, <23,10>, <23,11>, <23,12>, <23,13>})
# (22857.47611313194, {<21,10>, <21,11>, <22,10>, <22,11>, <22,12>, <22,13>, <23,10>, <23,11>, <23,12>, <23,13>, <24,12>, <24,13>})
# (22853.47433214035, {<22,9>, <22,10>, <22,11>, <22,12>, <22,13>, <23,10>, <23,11>, <23,12>, <23,13>, <24,11>, <24,12>, <24,13>})
# (22839.70753834977, {<21,10>, <21,11>, <22,9>, <22,10>, <22,11>, <22,12>, <23,10>, <23,11>, <23,12>, <23,13>, <24,12>, <24,13>})
# (22837.73237196119, {<22,10>, <22,11>, <22,12>, <23,10>, <23,11>, <23,12>, <23,13>, <23,14>, <24,11>, <24,12>, <24,13>, <24,14>})
# (22826.585288632137, {<21,10>, <21,11>, <21,12>, <22,10>, <22,11>, <22,12>, <23,10>, <23,11>, <23,12>, <23,13>, <24,12>, <24,13>})
# (22820.297998690636, {<21,10>, <22,10>, <22,11>, <22,12>, <22,13>, <23,10>, <23,11>, <23,12>, <23,13>, <24,11>, <24,12>, <24,13>})
# (22814.524101103874, {<22,10>, <22,11>, <22,12>, <22,13>, <23,10>, <23,11>, <23,12>, <23,13>, <23,14>, <24,12>, <24,13>, <24,14>})
# (22796.601568398462, {<21,10>, <21,11>, <21,12>, <22,9>, <22,10>, <22,11>, <22,12>, <22,13>, <23,10>, <23,11>, <23,12>, <23,13>})
qq2 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq2)))))
This second substrate tuple is nearly as highly aligned as the first. Again it is located where a three would differ from an eight, but now appears in the bottom loop.
Again, let’s image the first 20 states ordered by size descending,
pp = list(reversed(list(sset([(b,a) for (a,b) in aall(araa(uu,hrred(hr,qq2)))]))))[:20]
[int(a) for (a,b) in pp]
# [2651, 274, 168, 161, 123, 122, 116, 103, 102, 88, 88, 83, 82, 76, 75, 72, 69, 67, 66, 65]
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,2,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for (_,ss) in pp]))
The first state is the largest slice. In this case also, all of the pixels of the tuple are background, so it includes all images where this region is blank. It resembles a non-oblique nine.
Continuing on,
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2,hr,hrr)
rpln(ll)
# (23104.900949518127, {<17,20>, <18,19>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>})
# (22960.349186777927, {<17,20>, <17,21>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>})
# (22840.443227031716, {<17,19>, <17,20>, <18,19>, <18,20>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>})
# (22837.063199354783, {<17,19>, <18,19>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>})
# (22791.67231895468, {<17,20>, <18,19>, <18,20>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>, <22,19>})
# (22789.48599062861, {<17,20>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>, <22,19>})
# (22772.26799923459, {<17,20>, <17,21>, <18,19>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <21,19>, <21,20>})
# (22723.030521171495, {<17,20>, <18,19>, <18,20>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>, <21,21>})
# (22712.286638556525, {<17,19>, <17,20>, <18,19>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <21,19>, <21,20>})
# (22706.68420497635, {<17,20>, <18,20>, <18,21>, <19,19>, <19,20>, <19,21>, <20,19>, <20,20>, <20,21>, <21,19>, <21,20>, <21,21>})
qq3 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq3)))))
Again, the third substrate tuple is as highly aligned as the first two. It is located where a five would differ from a two in the bottom loop.
Imaging the first 20 states ordered by size descending,
pp = list(reversed(list(sset([(b,a) for (a,b) in aall(araa(uu,hrred(hr,qq3)))]))))[:20]
[int(a) for (a,b) in pp]
# [2693, 249, 244, 160, 136, 120, 93, 93, 89, 80, 79, 78, 76, 75, 71, 70, 69, 65, 64, 59]
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,2,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for (_,ss) in pp]))
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3,hr,hrr)
rpln(ll)
# (17356.38849645686, {digit, <13,14>, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,14>, <15,15>})
# (17314.851785642062, {digit, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,14>, <15,15>, <15,16>})
# (17305.333682298817, {digit, <13,14>, <13,15>, <14,13>, <14,14>, <14,15>, <15,13>, <15,14>, <15,15>})
# (17259.502357359706, {digit, <12,16>, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,14>, <15,15>})
# (17240.66817747341, {digit, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,13>, <15,14>, <15,15>})
# (17081.937598002434, {digit, <13,14>, <13,15>, <14,14>, <14,15>, <15,14>, <15,15>, <16,14>, <16,15>})
# (17070.499385308365, {digit, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,14>, <15,15>, <16,14>})
# (17055.786924287586, {digit, <12,15>, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,14>, <15,15>})
# (17044.885096219266, {digit, <13,15>, <14,14>, <14,15>, <15,14>, <15,15>, <16,13>, <16,14>, <16,15>})
# (17039.324314549714, {digit, <13,15>, <13,16>, <14,14>, <14,15>, <14,16>, <15,14>, <15,15>, <16,15>})
Now, in the fourth tuple we see that digit has appeared.
qq4 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq4-vvl)))))
This fourth substrate tuple is not as highly aligned as the first three. It is located within the top loop at the bottom.
Imaging the first 20 states ordered by size descending,
pp = list(reversed(list(sset([(b,a) for (a,b) in aall(araa(uu,hrred(hr,qq4)))]))))[:20]
sat = statesVarsValue
[(int(a), sat(ss,digit)) for (a,ss) in pp]
# [(653, 0), (478, 7), (322, 4), (305, 1), (288, 2), (250, 9), (208, 6), (197, 8), (169, 3), (139, 5), (127, 1), (89, 1), (88, 4), (71, 3), (70, 5), (67, 1), (63, 1), (62, 5), (60, 7), (57, 5)]
bmwrite(file,bmhstack([bmborder(1,hrbm(28,1,2,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for (_,ss) in pp]))
In this case, the images are each associcated with a digit value.
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4,hr,hrr)
rpln(ll)
# (21583.42514813407, {<10,19>, <11,19>, <11,20>, <12,18>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <15,19>})
# (21565.369214092865, {<10,19>, <11,19>, <12,18>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <14,20>, <15,19>})
# (21515.941010458548, {<10,19>, <11,19>, <11,20>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <14,20>, <15,19>})
# (21481.863697596393, {<10,19>, <11,18>, <11,19>, <12,18>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <15,19>})
# (21416.80548183016, {<10,19>, <11,18>, <11,19>, <11,20>, <12,18>, <12,19>, <12,20>, <13,18>, <13,19>, <14,18>, <14,19>, <15,19>})
# (21353.65435870437, {<10,19>, <11,19>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <14,20>, <15,19>, <15,20>})
# (21316.354271132936, {<10,19>, <11,19>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <14,20>, <15,18>, <15,19>})
# (21307.252146369683, {<10,19>, <11,18>, <11,19>, <12,18>, <12,19>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <14,20>, <15,19>})
# (21297.206920226832, {<10,19>, <10,20>, <11,19>, <11,20>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <15,19>})
# (21283.91748126134, {<10,19>, <11,19>, <12,18>, <12,19>, <12,20>, <13,18>, <13,19>, <13,20>, <14,18>, <14,19>, <15,18>, <15,19>})
qq5 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq5)))))
The fifth substrate tuple, qq5, has slightly lower alignment than the first three, but higher than the fourth, qq4.
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5,hr,hrr)
rpln(ll)
# (21761.47641406216, {<13,8>, <14,8>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>, <18,9>})
# (21643.640258377673, {<13,8>, <14,8>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>, <18,8>})
# (21603.015512050486, {<13,8>, <14,8>, <15,7>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>})
# (21574.38400300072, {<13,8>, <14,8>, <15,8>, <15,9>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>, <18,8>, <18,9>})
# (21552.30151733259, {<12,8>, <13,8>, <14,8>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>})
# (21509.49520647218, {<13,8>, <14,8>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <18,8>, <18,9>})
# (21508.588547909407, {<13,8>, <14,8>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <16,11>, <17,8>, <17,9>, <17,10>})
# (21501.01142613017, {<13,8>, <14,8>, <15,8>, <15,9>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>, <18,9>, <18,10>})
# (21459.632400840906, {<13,8>, <14,8>, <15,7>, <15,8>, <15,9>, <15,10>, <16,7>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>})
# (21453.038356586992, {<13,8>, <14,8>, <15,8>, <15,9>, <15,10>, <16,8>, <16,9>, <16,10>, <17,8>, <17,9>, <17,10>, <18,10>})
qq6 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq6)))))
The sixth substrate tuple, qq6, resembles the first substrate tuple, qq1. When adding them together we obtain,
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq1|qq6)))))
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6,hr,hrr)
rpln(ll)
# (22489.420460751047, {<18,15>, <18,16>, <19,14>, <19,15>, <19,16>, <20,13>, <20,14>, <20,15>, <20,16>, <21,13>, <21,14>, <21,15>})
# (22432.222910781104, {<18,13>, <18,14>, <18,15>, <19,13>, <19,14>, <19,15>, <20,13>, <20,14>, <20,15>, <21,13>, <21,14>, <21,15>})
# (22383.671016607914, {<18,14>, <18,15>, <19,13>, <19,14>, <19,15>, <20,12>, <20,13>, <20,14>, <20,15>, <21,13>, <21,14>, <21,15>})
# (22365.905164590025, {<18,14>, <18,15>, <18,16>, <19,14>, <19,15>, <19,16>, <20,14>, <20,15>, <20,16>, <21,14>, <21,15>, <21,16>})
# (22264.56837985155, {<18,14>, <18,15>, <19,13>, <19,14>, <19,15>, <20,13>, <20,14>, <20,15>, <21,12>, <21,13>, <21,14>, <21,15>})
# (22223.312134400352, {<18,14>, <18,15>, <18,16>, <19,14>, <19,15>, <19,16>, <20,13>, <20,14>, <20,15>, <20,16>, <21,14>, <21,15>})
# (22213.161553055274, {<18,15>, <18,16>, <19,14>, <19,15>, <19,16>, <20,14>, <20,15>, <20,16>, <21,13>, <21,14>, <21,15>, <21,16>})
# (22179.686139721343, {<18,14>, <18,15>, <18,16>, <19,14>, <19,15>, <19,16>, <20,14>, <20,15>, <20,16>, <21,13>, <21,14>, <21,15>})
# (22173.74295615703, {<17,15>, <18,14>, <18,15>, <19,13>, <19,14>, <19,15>, <20,13>, <20,14>, <20,15>, <21,13>, <21,14>, <21,15>})
# (22142.74951703074, {<18,15>, <18,16>, <19,13>, <19,14>, <19,15>, <19,16>, <20,13>, <20,14>, <20,15>, <20,16>, <21,14>, <21,15>})
qq7 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq7)))))
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7,hr,hrr)
rpln(ll)
# (20855.132804664812, {<15,17>, <15,18>, <16,17>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>})
# (20840.095303537077, {<15,18>, <16,17>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>, <20,17>})
# (20630.83117399637, {<16,17>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>, <20,17>, <20,18>})
# (20601.044419759968, {<15,18>, <16,17>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>, <20,18>})
# (20556.711049208014, {<15,18>, <16,17>, <16,18>, <16,19>, <16,20>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>})
# (20516.735456510894, {<15,18>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>, <20,17>, <20,18>})
# (20454.344857458327, {<16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>, <20,17>, <20,18>, <21,17>})
# (20436.09923262436, {<15,18>, <16,17>, <16,18>, <16,19>, <17,16>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>})
# (20366.847226964914, {<2,20>, <15,18>, <16,17>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>})
# (20365.055467495687, {<4,6>, <15,18>, <16,17>, <16,18>, <16,19>, <17,17>, <17,18>, <17,19>, <18,17>, <18,18>, <19,17>, <19,18>})
qq8 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq8)))))
Continuing on,
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8,hr,hrr)
rpln(ll)
# (21638.18626128333, {<16,12>, <16,13>, <16,14>, <16,15>, <17,12>, <17,13>, <17,14>, <17,15>, <18,12>, <18,13>, <18,14>, <19,13>})
# (21609.984725045386, {<16,12>, <16,13>, <16,14>, <16,15>, <17,12>, <17,13>, <17,14>, <18,12>, <18,13>, <18,14>, <19,12>, <19,13>})
# (21592.67009164659, {<16,12>, <16,13>, <16,14>, <16,15>, <17,12>, <17,13>, <17,14>, <17,15>, <18,12>, <18,13>, <18,14>, <19,12>})
# (21528.97842413271, {<16,12>, <16,13>, <16,14>, <16,15>, <17,11>, <17,12>, <17,13>, <17,14>, <17,15>, <18,12>, <18,13>, <18,14>})
# (21512.334769560475, {<16,12>, <16,13>, <16,14>, <16,15>, <17,12>, <17,13>, <17,14>, <17,15>, <18,11>, <18,12>, <18,13>, <18,14>})
# (21456.25679233699, {<16,12>, <16,13>, <16,14>, <16,15>, <17,11>, <17,12>, <17,13>, <17,14>, <18,11>, <18,12>, <18,13>, <18,14>})
# (21403.439001992254, {<16,11>, <16,12>, <16,13>, <16,14>, <16,15>, <17,11>, <17,12>, <17,13>, <17,14>, <18,11>, <18,12>, <18,13>})
# (21367.71290302565, {<16,11>, <16,12>, <16,13>, <16,14>, <16,15>, <17,11>, <17,12>, <17,13>, <17,14>, <18,12>, <18,13>, <18,14>})
# (21347.439969852006, {<16,11>, <16,12>, <16,13>, <16,14>, <16,15>, <17,12>, <17,13>, <17,14>, <17,15>, <18,12>, <18,13>, <18,14>})
# (21340.052110554607, {<16,12>, <16,13>, <16,14>, <16,15>, <17,11>, <17,12>, <17,13>, <17,14>, <17,15>, <18,11>, <18,12>, <18,13>})
qq9 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq9)))))
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9,hr,hrr)
rpln(ll)
# (21135.007196751125, {<6,12>, <6,13>, <6,14>, <6,15>, <7,11>, <7,12>, <7,13>, <7,14>, <7,15>, <8,11>, <8,12>, <8,13>})
# (21022.871958359097, {<6,12>, <6,13>, <6,14>, <6,15>, <7,12>, <7,13>, <7,14>, <7,15>, <8,12>, <8,13>, <8,14>, <8,15>})
# (21008.886890636255, {<6,12>, <6,13>, <6,14>, <6,15>, <7,11>, <7,12>, <7,13>, <7,14>, <7,15>, <8,12>, <8,13>, <8,14>})
# (20979.21306589527, {<6,12>, <6,13>, <6,14>, <6,15>, <7,12>, <7,13>, <7,14>, <7,15>, <8,11>, <8,12>, <8,13>, <8,14>})
# (20917.158803059552, {<6,13>, <6,14>, <6,15>, <7,11>, <7,12>, <7,13>, <7,14>, <7,15>, <8,11>, <8,12>, <8,13>, <8,14>})
# (20848.651218395073, {<6,12>, <6,13>, <6,14>, <6,15>, <6,16>, <7,12>, <7,13>, <7,14>, <7,15>, <8,12>, <8,13>, <8,14>})
# (20801.77063803292, {<6,11>, <6,12>, <6,13>, <6,14>, <6,15>, <7,12>, <7,13>, <7,14>, <7,15>, <8,12>, <8,13>, <8,14>})
# (20763.408676788615, {<6,11>, <6,12>, <6,13>, <6,14>, <6,15>, <7,11>, <7,12>, <7,13>, <7,14>, <7,15>, <8,12>, <8,13>})
# (20728.801724287594, {<6,13>, <6,14>, <6,15>, <7,12>, <7,13>, <7,14>, <7,15>, <8,11>, <8,12>, <8,13>, <8,14>, <8,15>})
# (20723.745830955988, {<6,13>, <6,14>, <6,15>, <6,16>, <7,12>, <7,13>, <7,14>, <7,15>, <8,12>, <8,13>, <8,14>, <8,15>})
qq10 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq10)))))
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10,hr,hrr)
rpln(ll)
# (21002.90496698302, {<12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <15,13>, <16,11>})
# (20985.425239605152, {<11,13>, <12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <15,13>})
# (20958.335251870125, {<11,12>, <11,13>, <12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>})
# (20922.084025382144, {<11,12>, <12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <15,13>})
# (20736.362772044602, {<12,12>, <12,13>, <12,14>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <15,13>})
# (20599.927693008645, {<11,12>, <11,13>, <12,12>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <15,13>})
# (20550.560663871554, {<11,13>, <12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <16,11>})
# (20476.139671750916, {<11,12>, <12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <16,11>})
# (20444.256797367347, {<11,12>, <12,12>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>, <15,13>, <16,11>})
# (20420.52061794775, {<11,13>, <11,14>, <12,12>, <12,13>, <13,11>, <13,12>, <13,13>, <14,11>, <14,12>, <14,13>, <15,11>, <15,12>})
qq11 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq11)))))
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11,hr,hrr)
rpln(ll)
# (20262.481208318823, {<7,18>, <7,19>, <7,20>, <8,18>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>})
# (20110.149527143985, {<7,19>, <7,20>, <7,21>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>, <11,21>})
# (20085.92248401114, {<7,18>, <7,19>, <7,20>, <8,18>, <8,19>, <8,20>, <8,21>, <9,18>, <9,19>, <9,20>, <9,21>, <10,20>})
# (20062.67766533998, {<7,18>, <7,19>, <7,20>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>, <11,21>})
# (20058.36273021924, {<7,19>, <7,20>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>, <11,21>, <12,21>})
# (19990.490983666587, {<7,18>, <7,19>, <7,20>, <7,21>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>})
# (19970.40011596435, {<7,19>, <7,20>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <9,22>, <10,20>, <10,21>, <11,21>})
# (19962.661872371667, {<7,19>, <7,20>, <8,19>, <8,20>, <8,21>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>, <10,22>, <11,21>})
# (19958.417007575656, {<7,19>, <7,20>, <8,18>, <8,19>, <8,20>, <8,21>, <9,18>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>})
# (19940.15639291197, {<7,18>, <7,19>, <8,18>, <8,19>, <8,20>, <8,21>, <9,18>, <9,19>, <9,20>, <9,21>, <10,20>, <10,21>})
qq12 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq12)))))
Having removed 12 tuples, there is plenty of alignment in the remaining substrate. The union of the 12 tuples is
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,(qq1|qq2|qq3|qq4|qq5|qq6|qq7|qq8|qq9|qq10|qq11|qq12)-vvl)))))
We can continue in this way,
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11-qq12,hr,hrr)
qq13 = ll[0][1]
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11-qq12-qq13,hr,hrr)
qq14 = ll[0][1]
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11-qq12-qq13-qq14,hr,hrr)
qq15 = ll[0][1]
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11-qq12-qq13-qq14-qq15,hr,hrr)
qq16 = ll[0][1]
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11-qq12-qq13-qq14-qq15-qq16,hr,hrr)
qq17 = ll[0][1]
ll = buildtup(2**12,10,10,uu,vv-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9-qq10-qq11-qq12-qq13-qq14-qq15-qq16-qq17,hr,hrr)
rpln(ll)
# (18202.790918012604, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,11>, <25,12>, <25,13>, <25,14>, <25,15>})
# (18184.08594121058, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,12>, <25,13>, <25,14>, <25,15>, <25,16>})
# (18144.412940493505, {<21,16>, <22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,12>, <25,13>, <25,14>, <25,15>})
# (18016.48051264869, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,12>, <25,13>, <25,14>, <25,15>, <26,14>})
# (18011.880832053263, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,12>, <25,13>, <25,14>, <25,15>, <26,15>})
# (17962.517647791818, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,12>, <25,13>, <25,14>, <25,15>, <26,12>})
# (17945.29579780344, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,12>, <25,13>, <25,14>, <25,15>, <26,13>})
# (17926.086378929896, {<21,16>, <22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,13>, <25,14>, <25,15>, <25,16>})
# (17921.076683220715, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,13>, <25,14>, <25,15>, <25,16>, <25,17>})
# (17875.589239621342, {<22,13>, <22,14>, <22,15>, <23,14>, <23,15>, <24,14>, <24,15>, <25,13>, <25,14>, <25,15>, <25,16>, <26,15>})
qq18 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,qq18)))))
The union of the first 18 tuples is
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(28,3,2,qqhr(2,uu,vvk,(qq1|qq2|qq3|qq4|qq5|qq6|qq7|qq8|qq9|qq10|qq11|qq12|qq13|qq14|qq15|qq16|qq17|qq18)-vvl)))))
Averaged pixels
The pixel variables can be averaged as well as bucketed to reduce the dimension from 28x28 to 9x9. Here 3x3 square regions are bucketed to have 8 values. (Note that the load requires around 2.5 GB memory.)
(uu,hrtr) = nistTrainBucketedAveragedIO(8,9,0)
digit = VarStr("digit")
vv = uvars(uu)
vvl = sset([digit])
vvk = vv - vvl
hr = hrev([i for i in range(hrsize(hrtr)) if i % 8 == 0],hrtr)
hrsize(hr)
7500
The query variable valency is 8, $\{|U_w| : w \in V_{\mathrm{k}}\} = \{8\}$,
sset([vol(uu,sset([w])) for w in vvk])
# {8}
The first 25 events placed in a row now look blurrier,
hr1 = hrhrred(hr,vvk)
bmwrite(file,bmhstack([bmborder(1,hrbm(9,3,8,hrev([i],hr1))) for i in range(25)]))
Again, the entire history can be averaged together, $\hat{A}\%V_{\mathrm{k}}$,
hrbmav = hrbm(9,3*3,8,hr1)
bmwrite(file,bmborder(1,hrbmav))
We can show the average image for each digit, $\{\hat{B}\%V_{\mathrm{k}} : S \in V_{\mathrm{l}}^{\mathrm{CS}},~B = A * \{S\}^{\mathrm{U}}\}$,
bmwrite(file,bmhstack([bmborder(1,hrbm(9,2*3,8,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for ss in cart(uu,vvl)]))
Again, consider how highly aligned variables might be grouped together. Create a shuffled sample, $A_{\mathrm{r}}$,
hrr = historyRepasShuffle_u(hr,1)
hrsize(hrr)
7500
Now optimise the shuffle content alignment with the tuple set builder, $I_{P,U,\mathrm{B,ns,me}}$, \[ \{(\mathrm{algn}(A\%K)-\mathrm{algn}(A_{\mathrm{r}}\%K),~K) : ((K,\cdot,\cdot),\cdot) \in I_{P,U,\mathrm{B,ns,me}}^{ * }((V,~\emptyset,~A,~A_{\mathrm{r}}))\} \]
def buildtup(xmax,omax,bmax,uu,vv,xx,xxrr):
return list(reversed(list(sset([(algn(rraa(uu,hrred(xx,kk))) - algn(rraa(uu,hrred(xxrr,kk))), kk) for ((kk,_),_) in parametersSystemsBuilderTupleNoSumlayerMultiEffectiveRepa_ui(xmax,omax,bmax,1,uu,vv,fudEmpty(),xx,hrhx(xx),xxrr,hrhx(xxrr))[0]]))))
rpln(buildtup(8**2,10,10,uu,vvk,hr,hrr))
# (1778.3439485839262, {<5,3>, <6,3>})
# (1546.4243938068685, {<5,8>, <6,8>})
# (1508.297779876615, {<4,3>, <5,3>})
# (1452.2331641622513, {<6,7>, <7,7>})
# (1410.4253862658807, {<8,3>, <8,4>})
# (1236.2427473031858, {<7,7>, <8,7>})
# (1192.1885900411507, {<6,8>, <7,8>})
# (1180.8147592081878, {<5,7>, <6,7>})
# (1174.0988978699388, {<4,8>, <5,8>})
# (1171.3702751695928, {<8,4>, <8,5>})
ll = buildtup(8**4,10,10,uu,vvk,hr,hrr)
rpln(ll)
# (5682.351019334106, {<5,3>, <5,8>, <6,3>, <6,8>})
# (5552.604687655745, {<4,3>, <5,3>, <5,8>, <6,3>})
# (5337.095505255653, {<5,3>, <5,8>, <6,3>, <7,3>})
# (5274.019275638675, {<4,8>, <5,3>, <5,8>, <6,3>})
# (5260.718020426779, {<4,3>, <5,3>, <6,3>, <6,8>})
# (5164.163585010243, {<4,3>, <5,3>, <6,3>, <7,3>})
# (5093.746288077142, {<5,8>, <6,3>, <6,8>, <7,3>})
# (5059.289354936685, {<5,3>, <6,3>, <6,8>, <7,3>})
# (5028.322281669007, {<4,3>, <5,3>, <5,7>, <6,3>})
# (5026.611555818628, {<5,8>, <6,3>, <6,8>, <7,8>})
ll = buildtup(8**5,10,10,uu,vvk,hr,hrr)
rpln(ll)
# (8050.553866086997, {<4,3>, <5,3>, <5,8>, <6,3>, <6,8>})
# (7998.912572413858, {<5,3>, <5,8>, <6,3>, <6,8>, <7,3>})
# (7820.9715551441805, {<4,8>, <5,3>, <5,8>, <6,3>, <6,8>})
# (7800.100864466516, {<5,3>, <5,8>, <6,3>, <6,8>, <7,8>})
# (7731.101242843724, {<4,3>, <4,8>, <5,3>, <5,8>, <6,3>})
# (7621.316722082192, {<4,8>, <5,3>, <5,8>, <6,3>, <7,3>})
# (7539.52093791958, {<5,3>, <5,7>, <5,8>, <6,3>, <6,8>})
# (7520.485299578861, {<4,3>, <5,3>, <5,8>, <6,3>, <7,3>})
# (7495.3678382320395, {<5,8>, <6,3>, <6,8>, <7,3>, <7,8>})
# (7430.563038948512, {<4,3>, <5,3>, <5,7>, <5,8>, <6,3>})
Let us image the most highly aligned tuple,
def qqhr(d,uu,vv,qq):
return aahr(uu,single(llss([(v, ValInt(d-1)) for v in qq] + [(v, ValInt(0)) for v in (vv-qq)]),1))
qq1 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(9,3*3,8,qqhr(8,uu,vvk,qq1)))))
The first tuple is split into two clusters on the left and right of the averaged image.
Now let’s image the first 20 states ordered by size descending, $\mathrm{top}(20)(A\%Q_1)$,
pp = list(reversed(list(sset([(b,a) for (a,b) in aall(araa(uu,hrred(hr,qq1)))]))))[:20]
[int(a) for (a,b) in pp]
# [3165, 177, 154, 125, 108, 95, 94, 87, 72, 69, 49, 44, 42, 41, 40, 35, 33, 31, 27, 27]
bmwrite(file,bmhstack([bmborder(1,hrbm(9,3,8,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for (_,ss) in pp]))
Now optimise again having removed the top tuple from the substrate,
ll = buildtup(8**5,10,10,uu,vvk-qq1,hr,hrr)
rpln(ll)
# (6552.725968278681, {<2,6>, <7,3>, <7,4>, <8,3>, <8,4>})
# (6470.476739248241, {<3,3>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6360.00856836614, {<3,3>, <3,4>, <7,7>, <7,8>, <8,7>})
# (6351.090494764196, {<5,7>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6310.23267732935, {<2,4>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6267.855911086894, {<3,7>, <7,3>, <7,4>, <8,3>, <8,4>})
# (6250.538787034926, {<6,7>, <7,3>, <7,7>, <7,8>, <8,7>})
# (6245.4011219204585, {<3,4>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6126.752957233486, {<6,7>, <7,7>, <7,8>, <8,7>, <8,8>})
# (6092.669300136113, {<2,6>, <3,7>, <7,3>, <8,3>, <8,4>})
qq2 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(9,3*3,8,qqhr(8,uu,vvk,qq2)))))
Again, let’s image the first 20 states ordered by size descending,
pp = list(reversed(list(sset([(b,a) for (a,b) in aall(araa(uu,hrred(hr,qq2)))]))))[:20]
[int(a) for (a,b) in pp]
# [2034, 242, 131, 127, 115, 82, 82, 75, 71, 55, 55, 45, 41, 34, 29, 23, 22, 22, 20, 19]
bmwrite(file,bmhstack([bmborder(1,hrbm(9,3,8,hrhrred(hrhrsel(hr,aahr(uu,single(ss,1))),vvk))) for (_,ss) in pp]))
Continuing on,
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2,hr,hrr)
rpln(ll)
# (6470.476739248241, {<3,3>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6360.00856836614, {<3,3>, <3,4>, <7,7>, <7,8>, <8,7>})
# (6351.090494764196, {<5,7>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6310.23267732935, {<2,4>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6245.4011219204585, {<3,4>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6126.752957233486, {<6,7>, <7,7>, <7,8>, <8,7>, <8,8>})
# (6057.676763678517, {<2,5>, <6,7>, <7,7>, <7,8>, <8,7>})
# (6030.4634853367115, {<3,3>, <3,4>, <6,7>, <7,7>, <8,7>})
# (6024.011956005892, {<2,4>, <3,3>, <7,7>, <7,8>, <8,7>})
# (5876.455495421929, {<2,4>, <3,4>, <7,7>, <7,8>, <8,7>})
qq3 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(9,3*3,8,qqhr(8,uu,vvk,qq3)))))
We can continue in this way,
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3,hr,hrr)
rpln(ll)
qq4 = ll[0][1]
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3-qq4,hr,hrr)
rpln(ll)
qq5 = ll[0][1]
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3-qq4-qq5,hr,hrr)
rpln(ll)
qq6 = ll[0][1]
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3-qq4-qq5-qq6,hr,hrr)
rpln(ll)
qq7 = ll[0][1]
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3-qq4-qq5-qq6-qq7,hr,hrr)
rpln(ll)
# (2148.123502836578, {<3,6>, <4,6>, <5,6>, <6,6>, <9,4>})
# (2138.15403044073, {<3,6>, <4,6>, <5,6>, <6,6>, <9,3>})
# (2099.5540951074927, {<3,2>, <4,2>, <4,6>, <5,6>, <6,6>})
# (2089.041993971011, {<3,6>, <4,6>, <5,1>, <5,6>, <6,6>})
# (2088.5029974702784, {<3,6>, <4,6>, <5,6>, <6,6>, <9,9>})
# (2088.215315397827, {<3,6>, <4,6>, <5,6>, <6,6>, <7,1>})
# (2087.5041497117636, {<3,1>, <3,6>, <4,6>, <5,6>, <6,6>})
# (2087.070183093523, {<3,6>, <4,6>, <5,6>, <6,6>, <9,2>})
# (2087.052164588021, {<3,6>, <4,6>, <5,6>, <6,6>, <8,1>})
# (2086.070665216217, {<3,2>, <3,6>, <4,6>, <5,6>, <6,6>})
qq8 = ll[0][1]
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(9,3*3,8,qqhr(8,uu,vvk,qq8)))))
Now the alignment has decreased significantly.
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8,hr,hrr)
rpln(ll)
qq9 = ll[0][1]
ll = buildtup(8**5,10,10,uu,vvk-qq1-qq2-qq3-qq4-qq5-qq6-qq7-qq8-qq9,hr,hrr)
rpln(ll)
(1091.9729017068748, {<2,7>, <2,8>, <4,5>, <7,2>, <8,2>})
(989.9015106783045, {<2,7>, <2,8>, <7,2>, <8,2>, <9,3>})
(975.724964808549, {<2,7>, <2,8>, <5,9>, <7,2>, <8,2>})
(962.8047501522524, {<2,7>, <2,8>, <2,9>, <7,2>, <8,2>})
(962.3862594156308, {<2,7>, <2,8>, <4,9>, <7,2>, <8,2>})
(958.7063695781835, {<2,7>, <2,8>, <7,2>, <8,2>, <9,2>})
(936.4720997172553, {<2,7>, <2,8>, <7,2>, <8,2>, <9,6>})
(935.6995871323525, {<2,7>, <2,8>, <3,9>, <7,2>, <8,2>})
(932.8303721981865, {<1,7>, <2,7>, <2,8>, <7,2>, <8,2>})
(924.2819900327377, {<2,7>, <2,8>, <7,2>, <8,1>, <8,2>})
qq10 = ll[0][1]
Now, after 10 tuples, the alignments are much decreased and the union is
bmwrite(file,bmborder(1,bmmax(hrbmav,0,0,hrbm(9,3*3,8,qqhr(8,uu,vvk,qq1|qq2|qq3|qq4|qq5|qq6|qq7|qq8|qq9|qq10)))))
