Julia Schwarz

PhD Student, Carnegie Mellon University

Co-founder of Qeexo

I build software that leverages probabilistic modeling, computer vision and machine learning to improve user interfaces. I have worked in the domains of touch and free-space input, and developed a radically new user interface architecture for handling inputs with uncertainty. More generally, I love solving puzzles and thinking about how computers can best augment human intellect.

Please see my curriculum vitae for a summary of my professional and academic experience. You can also find me on Google Scholar, GitHub, Twitter, StackOverflow, and LinkedIn.

Publications my output as a PhD student

Air+Touch: Interweaving Touch & In-Air Gestures

Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, Scott E. Hudson
In Proceedings of UIST '14
Anthony and I developed a class of interaction techniques that interweave touch events with in-air gestures, demonstrating how touch and in-air gestures are highly complementary. This was a great partnership; I led the technical implementation and project logistics (development of finger tracking algorithm & API using depth-sensing camera), while Anthony focused on making demos, design, and paper writing.

Probabilistic Palm Rejection Using Spatiotemporal Touch Features and Iterative Classification

Julia Schwarz, Robert Xiao, Jennifer Mankoff, Scott E. Hudson and Chris Harrison
In Proceedings of CHI ‘14
Presents a new algorithm for discriminating unintentional palm inputs from intentional pen inputs. This algorithm improves over existing palm rejection methods, reducing accidental palm inputs to 0.016 per pen stroke, while correctly passing 98% of stylus inputs.

Replacing 'Wave to Engage' on Xbox One by Combining Body Pose, Gaze and Motion to Determine Intention to Interact

Julia Schwarz, Charles Marais, Tommer Leyvand, Scott E. Hudson, Jennifer Mankoff
In Proceedings of CHI ‘14
I performed this work while working as a researcher for Microsoft Xbox on the team responsible for shipping Kinect. The results helped to replace the ‘wave to engage’ gesture on Xbox One. It presents a new algorithm for determining when to pay attention to a user’s actions, and when to ignore them.

TouchTools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction

Chris Harrison, Robert Xiao, Julia Schwarz and Scott E. Hudson
In Proceedings of CHI ‘14
The core idea behind TouchTools is to draw upon user familiarity with tools from the real world and bring them to interactive surfaces. To invoke a tool, users replicate a tool’s real-world grasp and press it to the screen as though it were physically present.

Phone as a Pixel: Enabling Ad-Hoc, Large Scale Displays using Mobile Devices

Julia Schwarz, David Klionsky, Chris Harrison, Paul Dietz, and Andy Wilson
In Proceedings of CHI ‘14
A system for building crowd sized displays using personal mobile phones. A camera watches the crowd and looks for flashing patterns identifying each individual phone. Once phones are identified, they can display a unique color or animation based on their location.

TapSense: Enhancing Finger Interaction on Touch Surfaces

Chris Harrison, Julia Schwarz, and Scott E. Hudson
In Proceedings of UIST '11
I helped develop a touchscreen that detects not only touch location, but touch type. This is accomplished by performing acoustic analysis when a finger (or other material) strikes the screen. The technology is now being commercialized by the CMU spinoff Qeexo.

Monte Carlo Methods for Managing Interactive State, Action and Feedback Under Uncertainty

Julia Schwarz, Jennifer Mankoff, Scott E. Hudson, Andy Wilson
In Proceedings of UIST '11
The second of two papers presenting a radically new input dispatch system for handling inputs with uncertainty (e.g. voice, touch, gesture). Tracks the likelihood of interface alternatives by deterministically simulating hundreds of input alternatives and aggregating results. This allows develpers to build uncertain interfaces without having to think probabilistically.

Augmenting Web Pages and Search Results to Support Credibility Assessment

Julia Schwarz and Meredith Ringel-Morris
In Proceedings of CHI '11
How do you decide what to believe online? It turns out that showing aggregate statistics such as number of viewers (particularly expert viewers e.g. doctors, lawyers, etc.) significantly helps people asses the credibility of web page content. This is especially helpful when viewing search results.

A Robust and Flexible Framework for Handling Inputs with Uncertainty

Julia Schwarz, Scott E. Hudson, and Jennifer Mankoff
In Proceedings of UIST '10
The first of two papers presenting a radically new input dispatch system for handling inputs with uncertainty (e.g. voice, touch, gesture). Presents a general input dispatch architecture where interface elements are queried regarding their suitability to handle an event.

Cord Input: An Intuitive, High-Accuracy, Multi-Degree-of-Freedom Input Method for Mobile Devices

Julia Schwarz, Chris Harrison, Scott E. Hudson, and Jennifer Mankoff
In Proceedings of CHI '10
I built a cord which detected twist, pull, and touch, with the intention of turning a simple headphone cord into an input device. My first research publication as a PhD student.

Reflections of Everyday Activities in Spending Data

Julia Schwarz, Jennifer Mankoff, Scott Matthews
In Proceedings of CHI '09
Your purchases reveal your whereabouts, your habits, preferences, and even your carbon footprint! This project aims to identify what credit companies already infer about your activies.

Notable Side Projects output from evenings and weekends

Headshot: A Windows Phone Application that Uses Computer Vision to Help You Take Selfies

Julia Schwarz
Windows Phone
Headshot is an application that helps people take better self-photos by detecting faces telling users how to adjust their phone. I developed this application while an intern at Microsoft, where it won a company-wide application development contest. The app has been featured in Windows Phone App Store, with over 225,000 downloads as of June 2014.

A Face Detection Library for Windows Phone 7

Julia Schwarz
Windows Phone
A C# implementation of Viola-Jones, able to load OpenCV XML files. First face detection library available for Windows Phone. I wrote this library while working on Headshot.

Window Manipulation Using the Microsoft Touch Mouse

Julia Schwarz and David Klionsky
Windows
An application enabling multitouch gestures on a Microsoft Touch mouse to move and resize windows. This application was built for the 2011 UIST Student Innovation Contest where it won first place in the 'Most Useful' category.

Inhabited Web

Julia Schwarz, Chris Harrison and Bryan Pendleton
Web
A small visualization next to a browser's scroll bar. Triangles are used to represent users and their current scroll position, allowing people to see where others are congregating. This application won the Yahoo HackU contest at Carnegie Mellon in 2009.

For more prior work, please see the archives, which contain older sites and projects (before 2010).

My other skills include juggling, skiing , and raft-building.