Soon after Jim has released a nice bouquet of arm7 compiles, i've conducted some quick tests in order to verify whether these upgraded binaries really bring additional strength. As always, i need to decide whether they are eligible for replacing their elder ARM5 brothers in Rapidroid.
We all know that arm7 should, in theory, perform better than arm5 but does the practice meet the theory every time? Neuh!
I didn't write anything since years but i'm not far from programming principles. Let's just state many unexpected parameters can effect the outcome. The most remarkable one is the source code itself. An arm7 compile can significantly overcome an arm5 compile only if the source code includes algorithms which would benefit from the differences between arm5 and 7. Otherwise, there's no reason for the performance to be different.
You want me to be clearer? Let's take the example of the most basic arithmetic formula: You would have a source code to multiply any user input number by 10. You would prepare the c++ code and compile it for arm5, then for arm7. Both binaries would probably do the job in exactly the same time, no matter what target architecture is choosen.
Coming back to recent arm7 releases by Jim Ablett, i wanted to compare the performance of both targets the quickest possible way. For this, i usually run the 3 minutes benchmark from initial position and compare the kNps together with the search depth reached by the engine.
I assume here that any binary with more depth and with higher kNpsthan the other should be superior.
In case the same depth applies to both builds but the kNps favors one of them, i decide according to the ratio between the kNps. If the ratio is significant enough like more than 10%, the higher kNps shows the better build.
If above criterias don't allow a selection, we must have two builds with very close depth and kNps scores. In this case a minimum of 100 head-to-head games at 180+2 may help decide. Or not...
If still no visible gap, the last thing to do is to conduct two gauntlets in parallel using same hardware and software conditions against another engine of similar strength.
Following this method, i've tested some JA compiles on Rockchip 1.4 Ghz quadcore processor and i've come to below verdicts (All scores given as arm5 followed by arm7):
One thing that may have mattered is that i didn't use the newest arm5 compiles that JA releases with arm7, instead i kept using the older arm5 releases rated in Rapidroid. Unexpected results below may also be related to the efficiency of the compiles from different date.
* Bison 9.11: Old ARM5 is better than new ARM7.
The benchmark gave 89 kNps vs 85 at depth 17 vs 17. Head to head 100 games result is 55.5 vs 44.5 favors arm5 by 38 ELO. Then i've done twin gauntlets vs Mini Rodent 1.0. Arm5 scored 52-48 (+14 ELO) while Arm7 scored 44.5-55.5 (-38 ELO). All 3 matches favored arm5 by ~45 ELO.
* Crafty 24.1: Old ARM5 is better than new ARM7.
362 kNps at depth 21 drops to 347 kNps at depth 21. No need to analyze more because v25.0 is already released.
* Crafty 25.0: ARM7 is clearly better.
332 kNps at depth 20 is way lower than 427 kNps at depth 20. Depths are the same but arm7 showed shorter time to depth performance thanks to higher nps.
* Discocheck 5.2.1: Old ARM5 is better than new ARM7.
Arm7 is disappointing. 144kNps @ D18 drops to 134 kNps @ D18. Arm-7 is slower considering the depths step up slightly faster with ARM5.
* ExChess 7.71b: ARM7 is clearly better.
387 kNps @ D24 rises to 555 kNps @ D25
* Fridolin 2.0.0: No verdict yet. I must do twin gauntlets vs other engines or keep ARM5.
1192 kNps @ D18 are close to 1142 kNps @ D18
* Hakkapeliitta 3.0: Old ARM5 is clearly better.
There's something terribly wrong with Arm7 compile because 439 kNps @ D23 vs 62 kNps @ D18 is more than a usual gap. Arm7 is very very slow.
* Phalanx XXIV: ARM7 is slightly better
154 kNps @ D15 rises to 170 kNps @ D15
* RedQueen 1.1.97: ARM7 is clearly better
174 kNps @ D21 rises to 198 kNps @ D22
* Scorpio 2.7.7: ARM7 is clearly better
The result is to blame on SMP function which still doesn't work for arm5. Hash setting of scorpio.ini works but arm5 build can't use 4 threads. Arm7 wins easily. 215 kNps @ D21 vs 786 kNps @ D24 !!! Parallel search rocks!
* Senpai 1.0: Old ARM5 is clearly better.
294 kNps @ D22 drops to 263 kNps @ D21. Arm7 compile may have suffered from something.
And finally, given the number of surpises noticed, i've finally wanted to check Stockfish 121215 which will probably take back its crown from Komodo 9.3 in Rapidroid. This time ARM7 overcomes ARM5 easily: 441 kNps @ D23 rises to 645 kNps @ D23. This result clearly confirms ARM7 is much better.
Now, i guess Jim will take a deeper look to discover what happened to Bison, Hakkapeliitta, DiscoCheck and Senpai. Meanwhile i will keep the relevant ARM5 builds running in Rapidroid.
Others? Migrating from ARM5 to ARM7 seems healthy. This means: Many gauntlets and reprises await for me.
Others? Migrating from ARM5 to ARM7 seems healthy. This means: Many gauntlets and reprises await for me.
No comments:
Post a Comment