I'm afraid i'll never manage to understand what is the purpose of testing engines at extremely low time controls like 15 secs/game and issuing results of shoot outs kind of "compile of the day" vs "compile of the day" continously.
One may get more samples in given duration and quickly obtain some rankings or some kind of results stating that engine X is better than engine Y by Z ELO points in the conditions used. Right.
So what's next? Is there yet a concrete answer to what level of accuracy does such experiment can lead us?
I have two permanent reasons to defend that bullets and even blitz levels are far behind the real world. I believe rapid chess is the optmal simulation acceptable by a serious tester, no matter whether the developers of the top engines tend to verify ELO gains obtained with new patches, for instance based on thousands of games at 60"+0.05", the "LTC" definiton of the Stockfish test network, for LONG TIME CONTROL.
Time is relative but i relatively suggest that one minute is not long at all.
My reasons are:
1) A suitable time control must be "watchable" by human eyes and should not extract games faster than humans can follow live. 15 seconds x 2 sides = 30 seconds for a game is really a lightning in the sky. Don't we need to see what's going on? If i can't watch something happening, i'd better forget about it!
2) Surrounding components such as hard disks and memory are still slow compared to the processor speeds. Impact of relatively slower components are always suspected to deviate results.
I've been active in Stockfish test network for a while for hundreds of hours, testing their patches. My latest opinion is that what's called LTC there, is exactly what i'd been teached to call "bullet" time control for years. Impatience and hunger for quick samples matters here. But real life differs often and sometimes it's too late to discover that on extreme hardware with FIDE time controls things may change unexpectedly. Therefore you may lose the crown to a reptile. Surprise! You are TCEC'ed :-)
My verdict and my commitment is to keep 15 sec/move or 15 mins/game as recommended setting and 10 sec/move or 10 min/game as minimum acceptable time control for all my experiments. As practiced in Rapidroid...
And about other lists based on blitz or bullet time controls, i can simply congratulate for the effort but my respect is not a sign of confirmation.
To my taste, i still prefer less samples based on the simulation of a "watchable" chess game which requires no less than a Rapid chess time control, the way it's defined by FIDE. Shortly said: "Each side can use a base time for the whole game from 10 minutes to 60 minutes, be it without increment or with an increment per move to add in such a way that the total of the base time and 60 increments does not exceed 60 minutes".
That rule looks totally healthy and 600"+0" is the minimum acceptable, at least for me. Therefore if my tourneys take time, so be it. I'd better wait.