UI State Comparison Using Screenshots

Art der Arbeit:

Master Thesis

Betreuer:

Groß, Tobias
Lehrstuhl für Informatik 1 (IT-Sicherheitsinfrastrukturen)
E-Mail: tobias.gross@cs.fau.de

Beschreibung der Arbeit:

A huge amount of data like hard disk dumps and memory dumps are needed in digital forensic research and science to evaluate new technologies and to teach forensic methods. Mostly, this data has to be produced in a artificial way, since often real IT systems cannot be used because of sensitive personal data.
For the artificial creation of this data, IT systems like Android Smartphones or Windows PCs have to be used automated. This can be implemented through several automation frameworks, but the outcome is often fragile and not very reliable. To improve the automation we want to evaluate the outcome for each automated UI action.
In this thesis we want to investigate if UI screenshots of Android Smartphones or Windows PCs can be used to evaluate the outcome of an UI action and to decide if the action conducted successfully or not. Therefore we need to use computer vision methods (f.e. the OpenCV Library [1]) to detect text elements and primitive geometrical shapes to describe a screenshot. Afterwards this can be used to compare different screenshots in a fuzzy manner and to score the similarity between screenshots.
Using this developed method and screenshots for the expected UI action outcome, we can decide wether the automated action was conducted successfully or not.
1

Schlagwörter:

Android Forensics

Bearbeitungszustand:

Die Arbeit ist bereits abgeschlossen.
Bearbeiter:	Ufuk Altiparmakoglu
Abgegeben am:	30.11.2020