Week 1 – Lecture Summary
Mobile and Ubiquitous Computing 2020/21
Sandy Gould, School of Computer Science University of Birmingham
Overview
Here I will introduce, in broad terms:
– History of computing
– Mark Weiser and Xerox PARC
– Mobile computing
– Android
– Systems, experience and sensors
Important concepts
This week I am introducing broad concepts related to mobile and ubiquitous computing. Some of these concepts will be recurring themes throughout the module. It is therefore essential that you have assimilated these concepts to help structure your learning in subsequent weeks.
History of computing
The size, shape, computational power and power usage of computers has changed hugely over the decades. In the early days of computing, computers were large, room-sized machines. This was the mainframe era. If an organisation had a digital computer (‘computer’ was originally a job title for a human) then they’d probably only have one. People would have to share time on this machine. They were often programmed using punched pieces of cardboard. Access to these machines was highly regulated with specialist staff responsible for running code and collating output. Significant expertise would be required to program and control these machines. Most people in an organisation did not interact with them. Individuals would not have had their own machines at work. People did not have digital computers at home. There would be one machine with many users.
Later, as technology developed, machines
became small enough and cheap enough for
lots of people to have their own personal
computer. This is the era of desktop
computing. People without technical expertise
now have a computer and it has been made
easier to operate because of the development
of graphical user interfaces. People no longer
had to share time on machines. They had their
own computing power that they could use as
much or as little as they wanted. These computers were not cheap by any means, but their size and cost also meant people also had them in their homes. There is now one machine with one user.
Finally, we arrive in the era of ubiquitous computing. Now computing devices are cheap enough to be used disposably. Everyday things like fridges and heating systems are now digital and networked. We carry personal computing devices with us all day every day: in our pockets, in our wrists and even under our skin. Computers are everywhere, computing is everywhere. There are now multiple, dozens, of machines for every user. These devices may not even have the kinds of screens and input methods that we’re using to seeing in ‘computers’. We may be interacting with information systems without even knowing it (e.g., through sensors and machine learning). This is the era in which we are currently living, with more and more devices every year.
So, over time, we have progressed from a small number of very large and expensive computers that were shared by lots of people, through an era of smaller computers where each person had their own, to an era where each person might have lots and lots of computers with them at any one moment. The way that interactive systems are designed in each of these eras is, by necessity, very different. In this module we focus on the ubiquitous era.
Mark Weiser and Xerox PARC
Mark Weiser wrote some of the most important and influential texts in ubiquitous computing. Much of his research took place at Xerox PARC, a research centre in California where many of the technologies we use every day were first imagined and developed. During the 1980s Mark Weiser and teams at Xerox PARC were working on portable machines and the idea of digital information being accessible from everywhere.
They realised that in the future there would be vastly more digital devices and that these would all need to be connected somehow. Weiser and colleagues developed the idea of the tabs, pads and boards. Tabs were the smallest devices and represented small pieces of information. If more information was needed, then a task could be transferred to a pad. If it needed sharing with a large group, then a board if it needed to be
shared with a group. One of the interesting things about this idea is that these devices would not be personal to particular people. Instead, they could be found around the environment, picked-up and used by anyone, just like Post-It notes, notebooks or whiteboards.
They envisaged radio and infrared communication allowing devices to talk. We don’t see much infrared used nowadays, but just about everything as a radio in it. Weiser also imagined digital technology ‘disappearing’, indeed he is most widely quoted for saying that the “[…] most profound technologies are those that disappear”. However, the kinds of interactions that Weiser was talking about didn’t mean that the devices would be invisible to people. Instead, they would disappear in the same way that street signs disappear into our environment. Weiser’s Scientific American article, The Computer for the 21st Century (Weiser, 1991) is still very influential.
Ubiquitous computing has not (and may not ever) reach Weiser’s vision. Look at the technology launched at CES 2020 and you’ll see devices that don’t seem like they will ‘disappear’ into our environment like street signs or furniture. We are moving towards Weiser’s vision though; sensing and machine learning will increasingly mean that technology, for better or worse, is increasingly able to disappear.
In “The Computer for the 21st Century” Mark Weiser realises the potential ethical challenges of ubiquitous computing. However, these issues are treated as a technology problem that can be solved with the competent application of security techniques. As we shall see in future weeks, Weiser underestimated the potential for these technologies to compromise our privacy.
Mobile computing
The biggest success of ubiquitous computing to date has been the mobile phone. Literally hundreds of millions of these devices are manufactured every year. In just the third quarter of 2019, 358m of these devices were sold. You probably carry one with you at all times. You might even sleep with it under your pillow.
Modern phones are exceptionally powerful compared to the first smartphones developed more than a decade ago. They have a vast array of sensors, from gyroscopes to cameras to microphones to barometers. This makes them almost like portable laboratories. But they aren’t magic – all this sensing requires power, network connections are not always reliable, and engineers use all the power of these devices (with, e.g., machine learning) as quickly as it is added.
iPhone (2007)
iPhone 12 Pro (2020)
CPU: Six-core @ 3.1/1.8GHz RAM: 6GB RAM
Screen: 6.1”, 1170×2532 Network: 5G, 2,000 Mbps
CPU: 412 MHz ARM 11
RAM: 128MB RAM
Screen: 3.5”, 320×480 Network: EDGE (2G) 135kbps
The power and capability of mobile phones means they are distinct from the full array of ubiquitous computing technologies, which must often make do without screens or large batteries. In many ways a phone is not that different to a laptop computer. Designing for future ubicomp devices means designing for devices that, potentially, have no screens or obviously discernible input methods. Designing interactions for these new contexts is a big challenge.
Have a think about it. How many devices with screens are there at the place where you live? How many internet connected devices? How many digital devices are there, ones that contain one or more microprocessors? Think about microwaves, dishwashers, ovens, clocks.
Android
The most popular operating system for mobile phones is, by far, Android. The first devices running Android were released in 2008 and since then it has found its way into television streaming sticks, e-readers, videogame consoles and cars. The current version of Android is Android 11. Development is primarily in Java, although Google are in creasingly moving toward the Kotlin language for development. Development of Android apps usually takes place in Android Studio, which runs on macOS, Windows and on Linux.
As part of this module, you will be learning to work with the Android API. The Java that you’ll need to write is not too difficult, but navigating the API is difficult, particularly given the number of changes that have been made to it over the years.
Research in ubiquitous computing
John Krumm (Krumm, 2009) identifies three themes in research involving ubicomp technologies. Systems, experience and sensors. We will cover these aspects in more detail in future weeks. For now an overview will suffice.
Systems research is concerned with the design, engineering and implementation of ubicomp systems. It considers how trade-offs around resource constraints (e.g., computational power vs energy consumption) might be resolved. It looks at how a panoply of devices all producing data can be connected. How do we manage such a wide variety of networking technologies? How can devices discover each other and share information when we don’t know what digital context these devices will be used in? This is the challenge of a dynamic heterogenous execution environment.
There is also the challenge of the dynamic heterogenous environment in which these devices actually operate. Where will they be placed in the home? Will they be used outside? It may be impossible to understand how people appropriate these kinds of ubiquitous devices. Devices, therefore, need to be design in a way that means they can function in as many physical environments as possible.
When designing ubicomp systems we also have to think about the ‘knowledge’ that such systems can build. What can be reliably sensed? What can a system reliably ‘know’? What can it reliably infer about a given scenario? Having to deal with dynamic heterogenous environments and dynamic heterogenous execution environments there is always the prospect of these systems breaking down and failing to work as intended. The idea of seamfulness (Chalmers & MacColl, 2003) is that as designers of ubicomp systems we need to prepare for these failures. We can deal with them pessimistically (i.e., only show what we are certain of); optimistically (i.e., pretend that everything is fine); cautiously (i.e., present any uncertainty to people); or opportunistically (i.e., make the best of the problem introducing something interesting or entertaining).
Finally, systems should, to the maximum possible extent, make use of off-the-shelf components. As few components as necessary should be used. This helps to keep costs low and helps make development and debugging easier.
After we’ve built these systems then we have experiences. In this kind of research we are interested in how people interact with these ubiquitous technologies. We can use field studies to investigate these technologies in the context of use by understanding how things worked before technology was introduced, whether the new technology actually works, and then how the new technology affects people’s behaviour. We can collect all kinds of data to work this out; telemetry; questionnaires; experience sampling reports; diaries; interviews. We will focus on these methods later in the module.
The final research theme is sensors. This research theme is concerned with how machines can sense the world and use this sense data to understand the world they exist in. Context creation is a real challenge for machines – as people we take the ability to adapt to our surroundings for granted. To build an idea of context, machines need to be able to understand their execution environment (i.e., the software and hardware world they exist in) but also their physical environment and, most challenging of all, the human/social environment. Machines can react to context by building complex if/else rules. Alternatively, and increasingly, Machine Learning can
be used to develop probabilistic accounts of a given context that can be used to attempt to classify a given context.
References
Chalmers, M., & MacColl, I. (2003). Seamful and seamless design in ubiquitous computing. Workshop at the Crossroads: The Interaction of HCI and Systems Issues in UbiComp, 17. https://www.researchgate.net/profile/Matthew_Chalmers/publication/228551086_Seamful_an d_seamless_design_in_ubiquitous_computing/links/0c9605188da3cf3173000000.pdf
Krumm, J. (Ed.). (2009). Ubiquitous Computing (1 edition). Chapman and Hall/CRC. Weiser, M. (1991). The Computer for the 21st Century. Scientific American, 265(3), 94–104.
https://doi.org/10.1038/scientificamerican0991-94