Passive Data Kit

What is Passive Data?

In your daily life, your mobile devices, web browsers, and desktop computers are generating and gathering data about you to provide many of the services and amenities of modern life. Your mobile phone uses satellites and network towers to estimate your current location to help you find nearby places to work and play. Your web browser maintains a history of your journey through the web to help you get to your favorite sites quickly. If you're a fitness fan, you may use a wearable device to track your daily activities in order to meet your fitness goals and motivate yourself to work a little bit harder.

All of this information is Passive Data that is generated without your direct intervention and largely out of your sphere of attention.

About Passive Data Kit

Passive Data Kit (PDK) is a software development framework that helps create systems that collect passive data and are simple, robust, and responsible:

Simple: You should be able to focus on your own system's unique value proposition and not worry about solving problems that have been addressed many times before. PDK encapsulates years of experience across many passive data applications to provide simple APIs and structures to help build your unique solution.
Robust: Passive data is only useful when it is consistent and complete. You should not have to re-engineering your system every time a vendor makes a change to the platform that interrupts your data collection unexpectedly. PDK detects problematic environments (both within and outside the device) and provides solutions to let you focus on your unique users and product.
Responsible: Passive data is sensitive and does reveal much about ourselves as individuals in our daily lives. PDK provides system creators a full set of tools to assist with legal and compliance obligations, user-facing interfaces to disclose and educate users about their data, and other components to help you deploy systems that your users understand and trust.

The PDK framework consists of

Reusable software components for gathering data in mobile, desktop (coming soon), and web environments.
A Django-based application server for collecting data from PDK clients deployed in the field.

PDK's intended audience is the software developer creating highly targeted systems that tackle domain-specific challenges with systems tailored to maximize the user experience. As such, PDK is a component that is imported into existing software projects that provides data collection, visualization, and transmission services leaving you to focus on meeting the challenges specific to your unique context. PDK is not¹ a standalone app or other independent system that non-software developers can download and deploy to begin collecting passive data without writing a line of code.

PDK is distributed under the Apache License, Version 2.0, allowing the framework to be deployed in open-source, educational, research, and commercial contexts.

¹ While a standalone PDK-based data collection system is on our eventual agenda, we recommend the AWARE Framework if you need to start collecting passive data and do not have the necessary software development resources.

Supported Platforms

PDK has been deployed in the field on the following user-facing platforms:

Android (GitHub)
iOS (GitHub)
Web browsers and web-hybrid apps (GitHub link coming soon)

In the cloud, PDK is a Django application that may be imported into an existing installation:

Django (GitHub)

Currently, PDK is only distributed via the Git repositories above, but support for platform-specific distribution methods (PIP, Gradle, CocoaPods) is on [our roadmap].

Origin & History

PDK is the creation of Audacious Software, a Chicago software development company founded by Chris Karr, who is primarily responsible for the development and direction of the project. The framework is a core component of Audacious Software's consulting business and has been deployed in educational, research, and healthcare contexts on behalf of a variety of clients. (See [Notable Deployments] below.)

Karr created PDK to address a problematic gap in passive data collection tools: the space between custom applications that reinvented the data collection "wheel" and standalone systems that discouraged focused user experiences and required additional user-hostile steps to get up and running. Developers creating apps that include passive data collection as a feature consistently encountered challenges unique to this form of data collection:

Technical changes to underlying platforms, such as Android's increasingly conservative power management, would render otherwise working apps broken when underlying technical assumptions changed. Policy changes on those platforms, such as the increased use of explicit user-granted permissions, created new user interface requirements for obtaining permissions from the users.
Competition between device and software vendors lead to inconsistent and contradictory implementations of "standards" that are often difficult to identify during development and testing. For example, in order to maximize their scores on measures such as device battery times, vendors will implement their own power management schemes that system developers only discover in the field when a user experiences problems on an effected device.
Reliable and consistent data collection is challenging in the field. Not only do developers have to understand their own domains, they also must be proficient in underlying platform-specific details that affect their systems when their apps are not running front-and-center. Deploying systems to different platforms compound this problem - you may be able to rely on one mechanism to gather data in a timely manner that doesn't work the same way on another platform. On iOS, regular push notifications may be a core component of a reliable data collection strategy, while on Android, equivalent notifications may be ignored or deferred by the host device.
"The Field" itself is complex and challenging. Waiting to transmit gathered data until a WLAN connection is available is a solid strategy. It's also an incomplete strategy given that there are any number of WLAN connections that do not connect to your cloud. (Recall the click-through agreements more often accompanying public WiFi networks.) When users find themselves in power- and connectivity-constrained environments, how should the system anticipate and address these contexts?

In his previous role as the developer of Purple Robot (a standalone Android system for gathering passive data) at Northwestern University's Center for Behavioral Intervention and Technologies (CBITs), Karr also dealt with the challenges of trying to shoehorn a standalone system into a variety of use cases:

How does a standalone system fit into an otherwise targeted custom system when the user experience of the customized solution has a very specific design language and user expectations that could not be anticipated during the initial creation?
In research contexts, subjects are often compensated for setting up the system with the assistance of staff employed to maximize the quality of data gathered from the field. In a standalone context, this often meant one app intended for the user, and another app installed to collect and transmit the data. While you can pay users to install both apps for the purposes of the study, this is a non-starter when a system is trying to make the jump from proof-of-concept to a commercially-viable system seeking wider adoption than the original research studies.
If the standalone system implements a system that allows the deployer to implement a customized user interface through some declarative mechanism, will those interfaces ever come close to the polished experience provided by native application development frameworks. At what point does working within the design constraints of the standalone system exceed the effort that would have been spent working from a custom system from the beginning?
Can you deploy a standalone system with your own in a commercial context without running into licensing issues? Will you be paying a tax on your success to the standalone system's vendor, in a commercial context? At what point will you replace the standalone component with something that is more harmonized with your business or organizational goals?

When Karr relaunched Audacious Software in 2016, he already had a pipeline of passive data collection projects and began creating PDK to be used with those clients to improve the efficiency of completing those projects. (Faster deployments are positively correlated with client satisfaction and financial success.) Standalone systems like Purple Robot would not meet the needs of the company's commercial clients, and he wanted to be unconstrained with respect to the user experiences provided to end-users. PDK occupies that "sweet spot" between competing client needs ("quick and dirty" and "commercially viable"), and effectively serves as the embodied institutional knowledge and experience of making passive data systems that work as well as possible.

In addition to the operational concerns addressed above, Karr also wanted to make it easier for software developers to be ethical in their treatment and use of data generated by end-users. In this sense, PDK is also designed to serve as a functional reference implementation that helps developers inform and disclose exactly what data is being gathered in order to begin rebuilding trust in these kinds of systems after a decade of abuse commercial and non-commercial system creators alike. This mission has only increased in importance as the law has caught up with the ethics, and system creators are now obligated to disclose and offer control over data collection under regulations such as the GDPR and HIPAA.

Passive Data Kit represents the cumulation and embodiment of two decades of passive data system construction from Karr's earliest mobile GIS systems created as a Princeton CS undergraduate, to his masters thesis work with [Pennyworth], to his Purple Robot work at CBITs, to his current work with research and commercial clients around the world.

Notable Deployments

IMAGE

BlūBot (2016 to 2017)

Developers at Emory University and Audacious Software created a proof-of-concept using Passive Data Kit to demonstrate the feasibility of gathering a variety of data from mobile devices, evaluating the data, and generating mood estimates, entirely on a mobile device using the latest research in smartphone mental health models.

The PDK server received the predictions from the field and integrated with an electronic medical record (EMR) to provide mental health providers the latest snapshot of a user’s mental health, informed by their behavior as observed by the BlūBot mobile app.

IMAGE

Sleepsight (2017 to Present)

Developers at Kings College London and Audacious Software created a mobile app that used Passive Data Kit to gather a variety of passive data from the mobile device and wearables to created real-time snapshots of subjects’ environment. The app also used PDK to administer a daily diary to aid researchers in creating predictive models to help patients identify and prepare for the onset of a schizophrenia episode.

In addition to serving as the central collection point for data from the field, the PDK server also served as the central data repository, providing a variety of authorized researchers access to environmental and sensor data in order to support their mental health research efforts.

IMAGE

DIAMANTE (2018 to Present)

A team from UC Berkeley, UCSF, and Audacious Software created a cross-platform mobile app that gathered fitness information from a variety of sources (including commercial fitness trackers) to support a reinforcement learner that used the fitness behavior to determine optimal strategies for messaging subjects to encourage a healthy lifestyle.

In this deployment, PDK was included as an add-on module to the HealthySMS system, allowing PDK to extend an existing messaging system to include subjects’ passive behavioral data to implement an adaptive messaging strategy that would have been otherwise impossible.

IMAGE

Fresh Comics, The Pnakotic Atlas, Santa's Map to Christmas (2016 to Present)

Audacious Software integrated PDK into existing entertainment apps serving a wide consumer audience in order to better understand the contexts and times when users were most engaged with these experiences.

Audacious Software continues to use intelligence gathered from PDK to refine and improve these apps in order to inform and delight audiences around the world.

Commercial Support

Passive data gathering systems is a cornerstone of Audacious Software's services.

If you are interested in having Audacious Software design, build, and deploy a PDK-enabled system, please send an e-mail to inquiries@audacious-software.com with the following details:

What data you are interested in capturing? (Note that this is not only limited to passive data - please specify any surveys or questionnaires as well.)
What is your project timeline? What are any relevant deadlines?
How is this project funded? Is it a fixed-cost research project funded by a grant or something else?
Do you have an internal team effort that this work will be supplementing, or do you need Audacious Software to provide the complete solution?
Do you have any externally-imposed constraints, such as the domicile of the cloud server storing the data? To what extent will you be gathering personally-identifiable information (PII) or personal health information (PHI) that may require additional documentation and agreements such as a Business Associates Agreement?

When we receive your inquiry, we will send a follow-up e-mail with any additional questions as well as arrange a time to discuss your project further.