Julia Schwarz
I build software that leverages probabilistic modeling, signal processing, and machine learning to improve user interfaces.
I received a BS from University of Washington, and a PhD from Carnegie Mellon University. At Carnegie Mellon I co-founded Qeexo, where I led the team that developed FingerSense, currently shipping on over 300 million Huawei devices.
In 2015 I moved back home to work on the HoloLens team at Microsoft. At Microsoft, I brought instinctual interactions to HoloLens 2, allowing users to directly interact with holograms using their hands.
For more information, please see my résumé.
You can find me on GitHub, Twitter, StackOverflow, and LinkedIn.
Publications my output as a PhD student
PhD Thesis: A Monte Carlo Approach to Probabilistic User Interfaces
Julia Schwarz
PhD Thesis, Carnegie Mellon University
A user interface architecture for handling probabilisitc or uncertain inputs. Uses monte carlo methods to track alternate input interpretations
An Architecture for Generating Interactive Feedback in Probabilistic User Interfaces
Julia Schwarz, Jennifer Mankoff, Scott E. Hudson
In Proceedings of CHI '15, Honorable Mention for Best Paper
New user interface architecture for handling probabilistic input. Simulates all possible interfaces and fuses them together in a meaningful way.
Air+Touch
Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, Scott E. Hudson
In Proceedings of UIST '14
A collection of interaction techniques that interweave touch events with in-air gestures.
Probabilistic Palm Rejection Using Spatiotemporal Touch Features and Iterative Classification
Julia Schwarz, Robert Xiao, Jennifer Mankoff, Scott E. Hudson and Chris Harrison
In Proceedings of CHI ‘14
New palm rejection algorithm that improves over existing palm rejection methods, reducing accidental palm inputs to 0.016 per pen stroke, while correctly passing 98% of stylus inputs.
Replacing 'Wave to Engage' on Xbox One by Combining Body Pose, Gaze and Motion to Determine Intention to Interact
Julia Schwarz, Charles Marais, Tommer Leyvand, Scott E. Hudson, Jennifer Mankoff
In Proceedings of CHI ‘14
A new algorithm for determining when to pay attention to a user’s actions, and when to ignore them. This work helped to replace the ‘wave to engage’ gesture on Xbox One.
TouchTools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction
Chris Harrison, Robert Xiao, Julia Schwarz and Scott E. Hudson
In Proceedings of CHI ‘14
TouchTools draws upon user familiarity with tools from the real world and bring them to interactive surfaces. To invoke a tool, users replicate a tool’s real-world grasp and press it to the screen as though it were physically present.
Phone as a Pixel: Enabling Ad-Hoc, Large Scale Displays using Mobile Devices
Julia Schwarz, David Klionsky, Chris Harrison, Paul Dietz, and Andy Wilson
In Proceedings of CHI ‘14
Use mobile phones to build crowd-sized displays. A camera watches the crowd and looks for flashing patterns identifying each individual phone. Once phones are identified, they can display a unique color or animation based on their location.
TapSense: Enhancing Finger Interaction on Touch Surfaces
Chris Harrison, Julia Schwarz, and Scott E. Hudson
In Proceedings of UIST '11
A touchscreen that detects not only touch location, but touch type. This is accomplished by performing acoustic analysis when a finger strikes the screen. The technology is now being commercialized at Qeexo.
Monte Carlo Methods for Managing Interactive State, Action and Feedback Under Uncertainty
Julia Schwarz, Jennifer Mankoff, Scott E. Hudson, Andy Wilson
In Proceedings of UIST '11
New user interface architecture that tracks probabilistic interface state bysimulating hundreds of input alternatives and aggregating results.
Augmenting Web Pages and Search Results to Support Credibility Assessment
Julia Schwarz and Meredith Ringel-Morris
In Proceedings of CHI '11
How do you decide what to believe online? It turns out that showing aggregate statistics such as number of viewers (particularly expert viewers e.g. doctors, lawyers, etc.) helps, especially when viewing search results.
A Robust and Flexible Framework for Handling Inputs with Uncertainty
Julia Schwarz, Scott E. Hudson, and Jennifer Mankoff
In Proceedings of UIST '10
Presents a general probabilistic user interface architecture where interface elements are queried regarding their suitability to handle an event.
Cord Input: An Intuitive, High-Accuracy, Multi-Degree-of-Freedom Input Method for Mobile Devices
Julia Schwarz, Chris Harrison, Scott E. Hudson, and Jennifer Mankoff
In Proceedings of CHI '10
I built a cord which detected twist, pull, and touch, with the intention of turning a simple headphone cord into an input device.
Reflections of Everyday Activities in Spending Data
Julia Schwarz, Jennifer Mankoff, Scott Matthews
In Proceedings of CHI '09
Your purchases reveal your whereabouts, your habits, preferences, and even your carbon footprint!
Notable Side Projects output from evenings and weekends
Headshot: A Windows Phone Application that Uses Computer Vision to Help You Take Selfies
Julia Schwarz
Windows Phone
Headshot helps you take selfies by detecting your face in the camera preview and telling you how to adjust your camera.
This application won a company-wide development contest at Microsoft and has been featured in Windows Phone App Store. It has over 225,000 downloads as of June 2014.
A Face Detection Library for Windows Phone 7
Julia Schwarz
Windows Phone
A C# implementation of Viola-Jones, able to load OpenCV XML files. First face detection library available for Windows Phone. I wrote this library while working on Headshot.
Window Manipulation Using the Microsoft Touch Mouse
Julia Schwarz and David Klionsky
Windows
An application enabling multitouch gestures on a Microsoft Touch mouse to move and resize windows. This application was built for the 2011 UIST Student Innovation Contest where it won first place in the 'Most Useful' category.
Inhabited Web
Julia Schwarz, Chris Harrison and Bryan Pendleton
Web
A small visualization next to a browser's scroll bar. Triangles are used to represent users and their current scroll position, allowing people to see where others are congregating. This application won the Yahoo HackU contest at Carnegie Mellon in 2009.
My other skills include juggling, skiing , and raft-building.