In the last years, we have observed active and proficuous research in autonomous and self-adaptive systems (SASs). The Software Engineering for Adaptive and Self-Managing Systems (SEAMS) has proposed two roadmap papers and there are various survey papers aiming at identifying the underlying research gaps and providing a taxonomy of SASs. Over the years, SASs are increasingly becoming “smarter” to be able to adapt and learn how to handle and manage new and unexpected events with autonomy. However, the precise meaning of “making a system smarter” is not always obvious, and, more pragmatically, it is not straightforward to decide how to concretely operate to achieve the ambition. Making a system smarter might involve various system’s abilities, such as configurability, autonomy, adaptability, perception, cognitive, and interaction with other systems and humans, to mention a few. These abilities can have various levels of relevance in specific systems. Then, it is important to understand in which direction it is worth and useful to invest to make systems smarter, and determining how to improve it and the extent of improvement for each specific direction.

To mitigate this research gap, this paper proposes an evaluation framework for autonomous systems, called LENS - evaLuation framEwork for autoNomous Systems. It is an instrument that can be used for (i) making an assessment of a system under the lens of abilities related to adaptation and smartness, (ii) identifying the possible directions of improvement, and (iii) making a re-assessment when the improvement has been performed. LENS stimulates reasoning to determine which abilities are worth enhancing in a system, and which levels within an ability are suitable and optimal for a system (thus rejecting the idea that higher levels are always better). Then, it will enable the planning of improvement steps and also define Key Performance Indicators (KPIs) to measure improvements in making systems smarter.

Given the high variability in the various domains in which autonomous systems are and can be used, LENS is defined in abstract terms and instantiated to a specific and important class of medical devices, i.e., Programmable Electronic Medical Systems (PEMS). The instantiation, called LENS_PEMS, is validated in terms of applicability, i.e., how it is applicable to real PEMS, generalizability, i.e., to what extent LENS_PEMS is generalizable to the PEMS class of systems, and usefulness, i.e., how it is useful in making an assessment and identifying possible directions of improvement towards smartness.