Personal Assistant System: Use of Computer Technologies to Facilitate Elderly People with Assisted Living

Presenter: Jennifer C. Hou ‡ Collaboration with Linda Ball*, Stanley Birge*, Marco Caccamo ‡ , Carl Gunter ‡ , Karrie G. Karahalios ‡ , Narasimhan Nitya # , Lui Sha ‡, Yang Yu # ‡ Dept of Computer Science University of Illinois at Urbana Champaign *College of Medicine, Washington University in Saint Louis # Pervasive Platforms and Architecture Lab, Motorola Labs [email protected], 217-265-6329

Population Aging

  Aging of the baby boomer has become a social and economical issue.

In the United States alone, the number of people over age 65 is expected to hit 70 million by 2030, almost doubling from 35 million in 2000.



Table compiled by the U.S. Administration on Aging based on data from the U.S. Census Bureau.

Percentage of People of 65+ and 85+

  People over age 65 are expected to constitute 20% of the population in 2030.

Similar increases are expected worldwide.



Table compiled by the U.S. Administration on Aging based on data from the U.S. Census Bureau.

Similar Expenses Worldwide

2002

SOURCE:

United Nations ▪ “Population Aging ▪ 2002”

Similar Expenses Worldwide

2030

SOURCE:

United Nations ▪ “Population Aging ▪ 2002”

Consequences Are …..

 Along with the increase of elderly people population, the expenditures of the United States for health care will project to rise to 15.9% of the GDP ($2.6 trillion) by 2010. -- Health care industry study, Digital Foresight  Many elderly people will stay at home, rather than being consigned to expensive retirement homes.

  Even today, only 10% of elderly people of age 65-85 and 25% of those of age >= 85 are institutionalized.

Many elderly people choose to stay at home also for privacy/dignity issues.

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work  Systems Architecture      Wireless communications Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?

 Pilot studies Where Do We Go From Here?

Why Do We Care About Senior Care

  Because your parents are the next in line to be qualified as seniors, and you are next to the next in line.

Can advances in sensing, object localization, wireless communications technologies   enable elderly people to regain their capability of independent living?

make possible unobtrusive supervision of basic needs of frail elderly and thereby replicate services of on-site health care providers?

We believe the answers are Yes!

We Believe Technology Can Help

 Time Driven Reminders of Daily Activities: Home PC     Serves as the intelligence.

Sends reminder messages to wireless-enabled appliances.

Closes the loop with HCI/ localization techniques.

Takes action in the lack of response: A reminder can be sent more times, after which a designated on-site personnel or a healthcare provider is notified.

Jennifer, It is 8:30am.

Time to take your Insulin injection before breakfast.

Overview of PAS

Assisted Living Hub (ALH) ALSP Clinician Service Gateway  Medical Device A drop-box architecture in which  A security-enhanced, assisted living device called for You (AMY) (ALH).

Authentication Manager co-exists with a home PC, called the Assisted-Living Hub   The ALH is equipped with multiple wireless interfaces and serves as the local intelligence. It also communicates with the Assisted Living Service Provider (ALSP).

Through web interfaces, ALSP allows healthcare providers and clinicians to retrieve/analyze data and give instructions.

A More Technical View of PAS

Home Environment

ALH

IEEE 802.11

WLAN Internet ALSP Monitoring Service • Easy to deploy.

• Few entities that are subject to security attacks.

• Numerous applications can be built on top of it.

Clinicians

What Applications Are Most Critical

Factors Contributing to Loss of Independence and Institutionalization*

% of Residents % of Residents as Need With Need Primary Cause ________________________________________________________________________ Needs prompting to take medications 95 42 Risk of injury due to falls 42 17 Unable to get up after a fall 20 17 Monitoring of vital signs too labor intensive 12 20 Needs physical assistance with Activities of Daily Living 90 67 Needs prompting to toilet on a schedule 67 17 Needs prompting to go to meals 33 10 Needs prompting to bathe 75 0 Gets lost in apartment 17 0 May wander out of facility 12 10 Needs monitoring of blood sugar frequently 20 8 Needs monitoring of weight daily/weekly 25 0

* Respondents include 8 geriatricians, 10 nurse administrators of assisted living facilities, and 6 home health clinicians at Washington University in Saint Louis

Application I: Time-based Reminder Services

 PAS can help reminding residents of daily activities.

   ALH obtains from the ALSP updated prescription and appointment records of a resident.

When it is time for the resident to carry out their time driven routines, the ALH locates active wireless-enable devices and sends reminder messages to one or more devices that are in the proximity of the resident.

Whether or not these routines are followed as advised is detected in a non-intrusive manner by exploiting sensor localization techniques.

Assisted Living Hub (ALH) ALSP Clinician Service Gateway Medical Device



Application II: Monitoring of Physiological Functions

A number of physiological functions critical to maintaining homeostasis for different medical conditions can be measured by Bluetooth-enabled medical devices, transmitted to the ALH and then to the ALSP to be evalauted by healthcare providers.

 Measures will have a prescribed desired range and deviations from that range will generate an alert from the ALSP to the health care provider.

 This enables prompt intervention before the situation deterioates to a point requiring hospitalization and may simply be in the form of additional instructions to the resident.

ALSP Home Environment IEEE 802.11

WLAN Internet Monitoring Service

Application III: Non-intrusive Monitoring of Daily Activities and locations  Detection of early warning signs for depression and/or other chronic diseases:  The location of a resident and the duration in which he/she stays at that location are profiled by Motes-based (or Ubisense-based) sensing and tracking in a privacy-preserving manner.

 Movement profiles are transmitted to the ALSP and analyzed (if necessary) for  Early warning signs for severe depression (e.g., not taking medicine, not eating, staying in bed for very long time)  Preventive measures (behavior changes) for chronic diseases commonly seen in elderly people (such as Parkinson’s disease and/or Alzheimer’s disease).

Application IV: Fall Detection

   Residents wear sensors equipped with accelerometers (with fall detection algorithms that detect falls with the combination of speed and orientation changes).

In the case of fall detection, the sensor device beeps and an alert message is sent to the ALSP and to a designated healthcare provider.

In the case of false alarm, the resident can press a button and disable the message sending.

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work 

Systems Architecture

     Wireless communications Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?

 Pilot studies Where Do We Go From Here?

Software Infrastructure Needed

 A software infrastructure that integrates sensing, communication, and event/information management.

 Understand, analyze, structure and control the complex interactions across the layers of computing, communication and sensing along the dimensions of robustness, reliability, QoS, security and privacy.

Reminders/social interaction Monitoring Emergency event processing Telemedicine

Interface mgmt Events/data management

Dependability Evolvability

Communication Sensing

Real-time Security & Privacy

Socket s = new

Networked Software Architecture

Socket(myaslp.com:90) ServerSocket ss = new ServerSocket(90); ss.accept();

Active Application A J2ME API for Accessing BT Nodes and Internet TCP/IP Stack Fixed ALH OS BT Stack Under Normal Situation When the Gateway is Available ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Vital Sign Meters 802.11

IP Routing Service 802.11

MAC Wireline MAC Gateway Cellphone Network + Internet

System Architecture of the ALH

OS Layer in the ALH Architecture

 The OS layer   Is equipped with various communication stacks and corresponding platform-dependent APIs Depending on the family of peripherals to support, other stacks and APIs may include Bluetooth, Zigbee, and Infrared.

Middleware Layer in the ALH Architecture

 Middleware   Device monitoring daemons: monitor the join/leave of peripheral devices in the environment and registers/de-registers the devices in the Device Registry Services .

The register-de-register process includes creating/destroying proxy or singular proxy stub objects for the device.

 The proxy/proxy stub with the device.

allows applications to be built upon well-known device APIs instead of vendor specific APIs/semantics. It also allows off the-shelf devices to be integrated into the PAS system, as long as the vendor provides the semantics specifications on how to communicate

Middleware Layer in the ALH Architecture

  Device registry service: maintains a database of peripherals available in the environment. Each entry of the database is a proxy (or proxy stub) object created for a specific peripheral device.

Applications will query this service for appropriate proxies. When an appropriate proxy is found, a clone of the proxy migrates to the application.

Middleware Layer in the ALH Architecture

  The Unified Peripheral Communication APIs   Abstracts different network stack programming interfaces into a consistent paradigm, which basically follows java.net.* APIs. Allows users to specify (in XML) their QoS demands when establishing communication links, and the middleware returns approved QoS guarantees. This allows proxies to be developed independently of specific network stack programming interfaces.

Middleware Layer in the ALH Architecture

 Internet Heartbeat Daemon:   periodically checks the availability of Internet access through the gateway router. When the gateway router fails/recovers, this daemon activates/deactivates the Bluetooth cell phone to access to the ALSP.

 ALH Main Daemon: the ALH.

is in charge of managing (start, suspend, stop, restart etc.) all the application daemons and middleware daemons on

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work  Systems Architecture     

Wireless communications and networking

Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?

 Pilot studies Where Do We Go From Here?

Peripheral Bluetooth Network earplug TV

Networked Software Architecture

ALSP ALH Internet Gateway WLAN (e.g. IEEE 802.11) Internet-capable Medical Meters

Socket s = new

Networked Software Architecture

Socket(myaslp.com:90) ServerSocket ss = new ServerSocket(90); ss.accept();

Active Application A J2ME API for Accessing BT Nodes and Internet TCP/IP Stack Fixed ALH OS BT Stack Under Normal Situation When the Gateway is Available ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Vital Sign Meters 802.11

IP Routing Service 802.11

MAC Wireline MAC Gateway Internet

ServerSocket ss = new ServerSocket(90); ss.accept()

Networked Software Architecture

Socket s = new Socket(128.174.11.11:90)

Passive Application A J2ME API for Accessing BT Nodes and Internet TCP/IP Stack Fixed ALH OS BT Stack PC IP: 128.174.11.11

Under Normal Situation When the Gateway is Available ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Dumb Dev Cellphone Network + Internet 802.11

IP Routing Service 802.11

MAC Wireline MAC Gateway

What If the Internet Gateway Is Down?

 One major deficiency of PAS is that all the traffic is transported through the gateway AMY, which becomes unavailable when it fails or when the resident is away from home.  To enhance robustness and ubiquity of PAS, we have used cell phones (transparent to users) as both a backup AMY and the local intelligence for data aggregation and acquisition.  we have leveraged the programming capability of Motorola EZX platforms (mainly the A780, E680 and E680i phones) to incorporate the following novel features that are not currently present in any cell phones:   Enabling cell phones to serve as local data storage/fusion intelligence.

Enabling cell phones as a delivery endpoint for reminder messages

Networked Software Architecture

Peripheral Bluetooth Network mobile medical TV devices earplug ALH cellphone ALSP Internet WLAN (e.g. IEEE 802.11) Internet-capable Medical Meters GPRS network or WiFi Base Station When the Gateway Available is Not

Socket s = new Socket(myaslp.com:90)

Active Application A J2ME + Bluetooth Dial-Up Networking Profile (BT-DUN) for Internet Access Fixed ALH OS BT Stack The cellphone modem is activated Cellphone DUN Modem (service provided by most BT enabled cellphones) APIs (TAPI/NAPI, File system, etc.) Service Layer (Network, Connectivity, Telephony, etc) BT Stack Cellphone OS GPRS

ServerSocket ss = new ServerSocket(90); ss.accept();

ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Bluetooth Medical Meter 802.11

IP Routing Service 802.11

MAC Wireline MAC A. L. Device Cellphone Network + Internet

ServerSocket ss = new ServerSocket(90); ss.accept()

Active Application A J2ME + Bluetooth Dial-Up Networking Profile (BT-DUN) for Internet Access Fixed Butler OS BT Stack The cellphone modem is activated Cellphone DUN Modem (service provided by most BT enabled cellphones) APIs (TAPI/NAPI, File system, etc.) Service Layer (Network, Connectivity, Telephony, etc) BT Stack Cellphone OS GPRS

Socket s = new Socket(128.174.11.

12 :90)

ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Bluetooth Medical Meter 802.11

IP Routing Service 802.11

MAC Wireline MAC Gateway Cellphone Network + Internet

Networked Software Architecture

Peripheral Bluetooth Network mobile medical devices earplug Cellphone ALSP GPRS or WiFi Base Station Internet WLAN (e.g. IEEE 802.11) Internet-capable Medical Meters When cell phone takes the roll of ALH

Socket s = new Socket(myaslp.com:90)

Active Application A Fixed Butler OS J2ME BT Stack The cellphone is used as a smart device Application 1 (Listening to PC) QoS based Application 2 (Sending data to ALSP Server) APIs (TAPI/NAPI, File system, etc.) Service Layer (Network, Connectivity, Telephony, etc) BT Stack Cellphone OS GPRS Or WiFi

ServerSocket ss = new ServerSocket(90); ss.accept();

ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Bluetooth Medical Meter 802.11

IP Routing Service 802.11

MAC Wireline MAC A. L. Device Cellphone Network + Internet

Socket s = new

Networked Software Architecture

Socket(myaslp.com:90) ServerSocket ss = new ServerSocket(90); ss.accept();

Active Application A J2ME API for Accessing BT Nodes and Internet GPRS Or WiFi Fixed ALH OS BT Stack When cell phone takes the roll of ALH ALSP Server TCP/IP Comm. API TCP/IP Stack ALSP Server OS Bluetooth Vital Sign Meters GPRS network or 802.11

IP Routing Service Base Station Internet

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work  Systems Architecture      Wireless communications and networking

Tracking and localization

Security and privacy Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?

 Pilot studies Where Do We Go From Here?

Real-Time Tracking and Localization

  We have built a system that uses a combination of ultrasonic and RFID technologies as the underlying sensing mechanism for real-time tracking of both human and objects. Human Tracking:    Ultrasonic transmitters, called beacons and denoted by B i , occasionally sends out an ultrasonic pulse and a short RF signal at the same time. Due to the speed difference between an RF signal and an ultrasonic signal, the listener denoted by L j on the resident can infer its distance from the beacon using Time Difference Of Arrivals of the two signals. This distance measurement can be reported to the ALH for real time tracking of the use (marked as (2) and (3)).

Real-Time Tracking and Localization

 Object tracking    The RFID (radio frequency identification) reader attached on the wristband can read RFID tags of objects touched by the user. The objects can be tracked, whenever the user touches and/or carries them.

The whereabouts of the objects are logged.

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work  Systems Architecture      Wireless communications and networking Tracking and localization

Security and privacy

Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?

 Pilot studies Where Do We Go From Here?

How Medical Data Can Be Securely Transmitted ?

Pub AS /Priv AS Pub C /Priv C

SOAP Envelope

Header Medical Information Pub C K F AS Sign. on Msg.

AP Sign. on Key AP PW AP K F

[Michael et al. 06]

How Medical Data Can Be Securely Transmitted ?

Pub AS /Priv AS Pub C /Priv C

SOAP Envelope

Header

SOAP Envelope

Header Medical Information Pub AS Sign. on Msg.

Sign. on Key AP Medical Information Pub C K F Sign. on Msg.

AP PW AP K F

How Medical Data Can Be Securely Transmitted ?

Pub AS /Priv AS Pub C /Priv C PW AP Pub C K F C Sign. on Msg.

AS Sign. on Key AS K F

How Medical Data Can Be Securely Transmitted ?

Pub AS /Priv AS Pub C /Priv C PW AP Medical Information Pub C K C K F C Sign. on Msg.

AS Sign. on Key AS K F

Implementation

Assisted Person Medical Device AL Hub Message Generator Encrypt Data Add Signatures/ Encrypt Envelope ALSP Company ALSP Server Clinicians / Family or Friends / Assisted Person Computer DB Access Message Generator Decrypt Envelope/ Verify Signatures Add Signatures/ Encrypt Envelope Decrypt Envelope/ Verify Signatures Message Generator Add Signatures/ Encrypt Envelope User Interface

Drop-box Prototype

AL Messages SAAJ Decrypt Data Decrypt Envelope/ Verify Signatures AL Security Engines XMLSec JCE

Apache Axis (WSDD / SOAP, XML-RPC / Java RPC, EJB, HTTP / WSDL Tools)

Implementation

 H/W     Device: Digital Pulse Oximeter AL Hub: A labtop, Windows ALSP: A desktop, Linux Clinician’s computer: A labtop, Windows  S/W     Language: Java Database: MySQL Web service: Apache web server, tomcat, axis 1.4

Libraries: XML-SEC, SAAJ, avetanaBluetooth, etc.

Security tokens

 The following security tokens are set up      Doctor’s public key certificate The ALSP’s public key certificate A secret key shared among AP and AP’s family members or friends AP’s and family/friends’ password to ALSP Access control matrix (ALSP), DataID table (AL Hub) G Doc , G AS , K AP

, U*; V*

,

P PA , P FA

,

ACL

AP , Δ AP • • • G Doc : Doctors office’s public key certificate.

G AS : The ALSP's public key certificate.

U*; V*

: URL pointers to the ALSP's public key certificate.

U*; V*

are the ones which AP and clinician have, respectively.

• K AP : A secret key shared the AP and her family members.

•

P PA ; P FA

: AP's and Family's secret passwords (respectively) shared with the ALSP.

•

ACL

AP : An access control list for the AP's records at the ALSP.

• Δ AP : A mapping table from a serial number of a medical device to DataID

1) 2) 3) 4) 5) 6) 7)

Bootstrapping

Clinician sets up relationship with ALSP / Exchange URL pointers Assisted person (AP) subscribes to an ISP / Set up a gateway for WPA network AP visits clinician. Clinician recommends a list of devices, hubs, and ALSPs. AP buys devices and hub and adds them to home network AP subscribes to ALSP. ALSP sends a URL pointer and a password token. AP loads them into the hub.

AP creates a shared “family key” for family members’ and friends’ access to data on the ALSP. Family members register passwords to ALSP.

Hub generates Δ AP , a set of DataID. A dataID is mapped to a device serial dataID, URL pointer) With the recipient information (clinicians and family members), ALSP creates

 

AP .

      

G doc , G AS , K AP

, U*, V*

,

P PA , P FA

,

ACL

AP , Δ AP Clinician Family AP ALSP

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work  Systems Architecture      Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?



Pilot studies

Where Do We Go From Here?

Pilot Study at Nazareth Living Center*

   With the help of geriatricians at Washington University in Saint Louis, we have carried out a 3-week pilot study at the Nazareth Living Center for Assisted Living in June-July 2006. This facility houses 110 well-educated, predominantly female residents, whose average age was 88. Of 30 residents who attended a presentation, 14 agreed to participate.

After administering a standard cognitive assessment, two residents were consented to test the PAS prototypes (application 1 and applicatoin2) and ten residents to carry/wear a placeholder device for a period of two weeks.

* Nazareth Living Center is a Catholic, nonprofit skilled nursing and assisted living community, sponsored by the Sisters of St. Joseph of Carondelet.

Pilot Study at Nazareth Living Center

  Residents found PAS useful and were willing to wear the sensing apparatus:   The two residents using the PAS prototype found it to be quite useful.

Residents not chosen to use the working prototype expressed their desire to use the working version as well.

Residents lack in confidence in PAS when it did not work properly:    Wireless connectivity is made possible by a low-end Linksys WRT54G wireless router. This, coupled with concrete walls between residents' rooms and the nurse station, led to intermittent connectivity. Residents, as a result, were not confident in relying solely on PAS for medical monitoring. To address this problem for the time being, we will add a wireless repeater (i.e., an additional Linksys WRT54G router configured in the client-mode and forwarding all packets to another, master router) between residents' rooms and the nurse station.

Pilot Study at Nazareth Living Center

 Nurses/caretakers desire interfaces that provide security/privacy .

 With a high resident-to-nurse ratio, nurses were usually very busy and could not ensure that the information being displayed would not be viewed by unauthorized personnel. The need for privacy should be addressed by designing adequate access control to PAS.

 Nurses inquired whether or not medical data could be securely transmitted via wireless technology. This implies they also had concerns about PAS security.

Overview of the Talk

     Why Do We Care About Senior Care?

Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work  Systems Architecture      Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Do Elderly People Accept This Technology?

 Pilot studies Where Do We Go From Here?

Where Do We Go From Here?

 Technology enhancement with respect to software robustness/reliability, security/privacy, and HCI.

 Extensive, hypothesis-driven clinical trials with Washington University in Saint Louis and Buehler Center on Aging, Health & Society, Northwest University   The average cost of nursing home stay is about $8,300/month * .

Can PAS delay transfers of elderly people to skilled nursing facilities (SNFs) and improve the quality of their lives (by preserving independence)?

Delaying institutionalization by simply one month would cover the cost of deploying PAS.

* Phibbs CS, Holtz J-EC, Goldstein MK et al. The effect of geriatrics evaluation and management on nursing home use and health care costs: Results from a randomized trial. Med Care 2006;44:91-5.

Concluding Remark

 With most of the PIs being baby boomers, it is our sincere hope that we will be the first generation to live independently and with dignity with the help of PAS when we age, and the project will benefit the healthcare system and the society as a whole.