Implementing Self-Configuring Graphical User Interfaces though Machine Learning
Part 2 of 4
At this point we have the bare minimum theory we need. We can now move to the implementation on Microsoft Windows 11. Though I have not tested it, the code should work fine on Linux. Just see if you need to change the directory nomenclature. Rest of the piece should sail through. Leave a note if it doesn’t, I will modify and send a Linux implementation.
What did we use?
Just two Python packages.
tkinter
- This is the heart of the entire GUI implementation. It is too huge to sum up. Please refer tkinter — Python interface to Tcl/Tk — Python 3.10.7 documentation
os
- To help interface with the Window operating system.
Here are some random interactions I did:
The Process and the Code
Here is the process. This aligns with the inline code comments.
Here is the screen shot of the code color coded for readability. Comments are inline. I strongly suggest you read these at least once. The actual code fragment is right below the screenshots.
What you will note after you have run the code:
- As you play with the code and capture your UI activity, you will see overlapping points or lines, after you have clicked long enough. This doesn’t affect the outcome we are targeting. In fact, it reiterates that your UI actions are confined to well-defined coordinates.
- If you capture your actions regularly, you will note that overtime, your clicking pattern and movement is nearly the same and you traverse a well-known set of points. The intuition of this implementation will kick-in at that point.
- I used tkinter as application UI context and you might not want that. You may want a full-screen context with no window or canvas visible to your end-user and yet the code should capture these actions. This requires a list of apps you use, permissions from them and a hook into their graphical contexts/layouts. It poses several challenges — security for one, and anti-virus software preventing you from reading/writing into an application binary. I suggest you stick to the original thinking that is — apply this approach to an application in which you have a code-level control on the geometry and positioning of the elements, right inside your IDE.
- You can modify code easily to add more action such as mouse middle click, right click, mouse hovering and more. Refer tkinter documentation on the event handlers. A whole universe awaits you.
- You can compare UI actions of each of the end-users and see who exhibits commonality in their screen travel paths. With enough data, you can club such users and gather more insights.
- Clicking arbitrarily for fun is OK, but it will create some standard deviations in the outcome causing bias or unexplained outlier. This is because you are clicking without a specific use case or a user journey that could require you to click the mouse or tap the screen in a certain path. Nonetheless, it is a blueprint of your UI interactions.
- Leave you with a thought — are your UI actions random?
What is coming next?
We have set the stage to capture user behavior. In the final 2 episodes, we will put this raw material into machine learning implementation and then see how we can use it to configure the user interface — automatically.
import tkinter as tk
from tkinter import Canvas
import os#
def write_events( type, c_x, c_y ):
with open(filename + “.csv”, “a”) as the_file:
to_write2 = f”{user_name},{type},{c_x},{c_y}\n”# write to the file
the_file.write(to_write2)
#
return#
def left_click( event ):
# sorry I took the easy way out by using Global variables
# using classes would have made code longer, and less palatable
global click_count, prev_x, prev_y# capture the event coordinates
x, y = event.x, event.y# click counter, important for labelling the sequence of the UI events
click_count = click_count + 1# the label shown when user clicks
the_label = f”Sample {click_count}. ({event.x}, {event.y})”# paint on the screen at the coordinate where click occurred
C.create_text(event.x, event.y, text = the_label, fill = “grey”)# draw the line to show chain of user behaviour
if prev_x != 0 and prev_y != 0:
# draw line from the last point and plot small marker in the vicinity
# the coordinates that go into the file remain unchanged
C.create_line(prev_x, prev_y, x, y, fill = “light grey”, smooth = 1)
C.create_rectangle(x, y, x + 4, y + 2, fill = “red”, outline = “grey”)# store the last click for line drawing
prev_x = x
prev_y = y# generate features
write_events(“left_clicked”, event.x, event.y)
#
return#
def key( event ):
# exit the code the moment any key is pressed
exit( )#
# main() of sorts
# — — — — — — — — -# Use the code filename a
filename = str(__file__).split(“/”)[ -1 ].split(“.”)[ 0 ]
file_expr = filename + “.csv”
user_name = os.getlogin( )# delete the file if one exists already
# to capture fresh coordinates
# this is important as how you use mouse
# differs as work through the day
if os.path.exists(file_expr):
os.remove(file_expr)# we can use anything to write files
# I chose open() function,
# note that a stands for append mode
with open(filename + “.csv”, “a”) as the_file:
# USERNAME will make a distinction as to whose behaviour is captured
# ACTION captures what the user did — for example left-clicked or moved the mouse
# X and Y are coordinates where ACTION occurred
to_write = (f”USERNAME,ACTION,X,Y” + “\n”)
the_file.write(to_write)# make a root handle to the tkinter toolkit
root = tk.Tk( )
## — — — — — — — — — -
click_count = 0
prev_x = 0
prev_y = 0# create a canvas
C = Canvas(root, bg = “white”, height = 1024, width = 768)# pack the Canvas with the main Window = root
C.pack( )
# this binds only the left button. Change this to <Button> if
# you want any of the mouse keys to capture the events
root.bind(‘<Button-1>’, left_click)# Helps terminate the program when mouse is engaged!
root.bind(‘<Key>’, key)#
root.mainloop( )
Disclaimer: All copyrights and trademarks belong to their respective companies and owners. The purpose of this paper is educational only and the views herein are my own.
