In order to switch things up a bit from all the formulas, programming and PCB building.
I decided to create a template for the functions of the device that I will be testing.
This template includes 3 sections:
Functions = Pass/Fail outcome
Measuremets = Accuracy %/Working %
Questionnaire = Questions for users on what they think about the watch, ease of use etc.
The 3 sections apply to different parts of the device, and the testing process is different for each part/functionality.
The following is the whole template:
FUNCTIONS (PASS/FAIL):
1) Sensor output.
2) Correct voice line output.
3) Correct voice line output with specific values (example 0.007 kg/m^3).
4) Voice recognition.
5) Tilt activation.
6) Battery charging mechanism.
MEASURABLES
Sensor values:
1) Compare them against calibrated sensors. Plot a graph to check the difference visually.
2) Add filtering systems. Check the values with and without the filtering systems.
Audio:
1) Check device volume with different speakers. Compare the results.
2) Check the best speaker in multiple environments with different noise levels to check how easy it is for the user to hear/understand.
Voice recognition:
1) Train the system with multiple microphones, check the differences in recognition accuracy.
2) Check the recognition accuracy with different scenarios having different noise levels.
3) Check the accuracy when using words with similar syllables/vowels. Example the word “Five” is similar to “Hive”, “Strive”, “Dive”, “Thrive”.
IMU Activity AI:
1) Compare the given activity to the actual person activity.
2) Check how long in ms it takes to understand the activity has started or ended.
IMU Step Counter AI:
1) Check accuracy for steps compared to actual measured steps + other fitness device measuring steps.
Rainfall Linear Regressor ML:
1) Compare model output to historically recorded outputs. Check difference %. Create graphs.
Whether Decision Tree ML:
1) Compare model output to historically recorded outputs. Check classification difference. Create table showing differences with RIGHT/WRONG, and total % of accuracy.
QUESTIONNAIRE:
1) Do you find the tilt activation easy to use?
2) Did you find the voice commands appropriate?
3) Did you find the volume of the smartwatch high enough?
4) Did you find the audio quality good?
5) Was your accent/voice pitch properly recognised by the system?
6) Do you find the watch comfortable on the wrist? Is it too big or too heavy?
Once I gather all the data from my device, based on the testing methods shown in the aforementioned template, I will be using Excel and/or MatPlotLib in Python to plot different types of graphs in order to more easily understand the data.
The following is a MatPlotLib example:
The 2 documents attached to this blog contain the testing template, as well as a MatPlotLib tutorial for graphs and histogram plotting.