We take advantage of a recently published open source implementation of the AES
protected with a mix of countermeasures against side-channel attacks to discuss both the challenges in protecting COTS devices against such attacks
and the limitations of closed source security evaluations. The target implementation has been proposed by the French ANSSI (Agence Nationale de la Sécurité des
Systèmes d'Information) to stimulate research
on the design and evaluation of side-channel secure implementations.
It combines additive and multiplicative secret sharings
into an affine masking scheme that is additionally mixed with a shuffled execution.
Its preliminary leakage assessment did not detect data dependencies with up to
100,000 measurements. We first exhibit the gap between such a preliminary leakage assessment and advanced attacks by exhibiting how a countermeasures' dissection exploiting a mix of dimensionality reduction, multivariate information extraction
and key enumeration can recover the full key with less than 2,000 measurements.
We then discuss the relevance of open source evaluations to analyze
such implementations efficiently, by exhibiting that certain steps of the
attack are hard to automate without implementation knowledge
(even with machine learning tools), while performing them
manually is trivial. Our findings are not
due to design flaws but from the general difficulty to prevent side-channel attacks in COTS devices with limited noise. We anticipate that
high security on such devices requires significantly more shares.