Homepage of Fabrice Matulic
 |
I am a Senior Researcher at Preferred Networks Inc., Japan, and Assistant Professor at the University of Waterloo, Canada, in Human-Computer Interaction (HCI), Extended Reality (XR) and applied AI. I have over 20 years of international experience leading research projects across academia and industry in six countries, resulting in several award-winning publications at top-tier HCI conferences and patents. My current work explores novel interactive experiences and techniques that leverage the power of AI, mainly in XR, but also for interactive surfaces and embodied interaction. |
 |
  |
HCI for Machine Learning Preparing and preprocessing source data to train neural networks can require considerable manual labour and expertise. We create intuitive user interfaces and techniques to facilitate some of those tasks, including data labelling and augmentation. Related Publications
|
 |
Dexterous Finger Gestures to Manipulate Mobile Phones This research explores single-handed "dexterous gestures" for manipulating a mobile phone using fine motor skills of fingers. We consider four dexterous manipulations: shift, spin, rotate, and flip, which we analyse in three user studies. We provide design guidelines to map gestures to interactions and show how they can be used in applications. Related Publication
|
|
  |
Pen+Touch+Midair Hybrid Two-Hand Interaction in desktop VR We explore a design space for hybrid bimanual pen and touch input extended to midair interaction in desktop-based virtual reality, specifically, asymmetric interaction patterns combining the pen with the other hand when interacting in the same “space” (either surface or midair), across both spaces, and with cross-space transitions (from surface to midair and vice versa). We concretely investigate those interactions and associated gestures with three testbed applications for 3D modelling, volumetric rendering, and terrain editing. Related Publications
|
|
 |
Typealike: Near-Keyboard Hand Postures for Expanded Laptop Interaction Typealike is a style of hand postures close to natural typing poses that allow users to quickly trigger commands on a laptop computer. The hand postures are detected using deep learning classification of images captured by the laptop's webcam reflected through a downward-facing mirror. Related Publication
|
|
  |
Mobile Phones as VR Controllers with Above-Screen Mirrors to Capture and Track Hands for Visualisation in VR Smartphones can be used as VR controllers but since the user cannot see the phone or their hands when wearing the headset, precise touch input is difficult. We address this problem by attaching one or two mirrors above the phone screen such that the front-facing camera captures the hand through reflection. With a single mirror the camera feed can be shown directly as a texture on the screen of the phone model in VR to help the user aim precisely with their fingers. With two mirrors capturing the hand from two different angles, we can track the 3D position of fingertips using deep learning. Related Publications
|
|
 |
PenSight: Enhancing Pen Interaction via a Pen-Top Camera PenSight is a novel concept to enhance pen interaction on tablets using a fisheye-lens camera attached to the top of the pen and facing downwards. Thus, the camera can "see" the user's hands and the surrounding environment. Using deep learning, we can detect different hand postures and tablet grips for quick action triggers and capturing off-tablet content such as surrounding documents. Related Publications
|
|
 |
Elicitation of Alternative Pen-Holding Postures for Quick Action Triggers with Suitability for EMG Armband Detection In this project we study what alternative ways of gripping a digital pen people might choose to trigger actions and shortcuts in applications (e.g. while holding the pen, extend the pinkie to invoke a menu). We also investigate how well we can recognise these different pen-holding postures using data collected from an EMG armband and deep learning. Related Publications
|
|
 |
Human-Robot Interaction for Personal Robots Smart domestic robots are poised to revolutionise the way household chores and everyday tasks are carried out in the home of the future. Thanks to the recent boom of deep learning and "artificial intelligence", machines are able to autonomously perform increasingly complex tasks. But no matter how smart these robots may be or become, humans still need to engage with them and it is paramount that such interactions occur smoothly and safely. Our research efforts in human-robot interaction aim to not only better support end users (customers) when operating robots in their home, but also facilitate the programming and training of these machines by engineers, technicians and developers. Related Projects and Publications
|
|
 |
ColourAIze: AI-Driven Colourisation of Paper Drawings with Interactive Projection System ColourAIze is an interactive system that analyses black and white drawings on paper, automatically determines realistic colour fills using AI and projects those colours onto the paper within the line art. In addition to selecting between multiple colouring styles, the user can specify local colour preferences to the AI via simple stylus strokes in desired areas of the drawing. This allows users to immediately and directly view potential colour fills for paper sketches or published black and white artwork such as comics. Related Publication
|
|
 |
Single-Hand Pen and Touch Input Using Variations of Pen-Holding Grips This work investigates the use of different pen-holding grips while writing and drawing on a tablet to trigger various actions, including changing the pen function (e.g. to select, scroll, search) and calling in-place menus. The postures are recognised when the hand contacts the surface using a deep convolutional neural network applied on the raw touch input data (the capactitive image of the tablet). The feasibility of this approach is confirmed by two user evaluations. Related Publications
|
|
 
|
 |
HybridPointing for Touch: Switching Between Absolute and Relative Pointing on Large Touch Screens CursorTap is a multitouch selection technique to efficiently reach both near and distant targets on large wall displays using hybrid absolute and relative pointing. The user switches to relative mode with three-fingers of one hand while using the other hand to control a cursor, similar to a touchpad. Related Publication
|
 |
Barehand Mid-air Mode-Switching Techniques in VR This work presents an empirical comparison of bare hand, mid-air mode-switching techniques suitable for virtual reality (VR). Specifically, we look at what kind of hand/finger postures can efficiently change the type of operation performed by the same action of the dominant hand (e.g. from moving a virtual object with a finger translation to scaling or copying it). We consider common finger and hand motions such as pinching fingers, turning and waving the hand(s)) as switching techniques. Our results provide guidance to researchers and practitioners when choosing or designing bare hand, mid-air mode-switching techniques in VR. Related Publication
|
|
 |
Multiray: Multi-Finger Raycasting for Large Vertical Displays Multiray is a concept that extends single raycasting for interacting with distant vertical displays to multi-finger raycasting, that is, each finger projects a ray onto the remote display. In particular, with multirays, patterns of ray intersections created by hand postures can form 2D geometric shapes to trigger actions and perform direct manipulations that go beyond single-point selections. Related Publication
|
|
 |
Experimental Analysis of Mode Switching Techniques in Touch-based User Interfaces This project looks at the performance of switching between different functions or modes for touch input (the possibility to rapidly change the output produced by the same touch action). Six techniques are evaluated in sitting and standing conditions: long press, non-dominant hand, two-fingers, hard press, knuckle, and thumb-on-finger. Our work addresses the lack of empirical evidence on the efficiency of touch mode-switching techniques and provides guidance to practitioners and researchers when designing new mode-switching methods. Related Publication
|
|
 |
 |
Hand and Finger Posture-Based Calling and Control of Tabletop Widgets Tabletop interaction can be enriched by considering whole hands as input instead of only fingertips. In this work, we propose a straightforward, easily reproducible computer vision algorithm to recognise hand contact shapes from the raw touch contact image. The technique is able to discard resting arms and supports dynamic properties such as finger movement and hover. The algorithm is used to trigger, parameterise, and dynamically control menu and tool widgets. Related Publication
|
 |
Embodied Interactions for Novel Immersive Presentational Experiences This project is about enhancing live multimedia presentations by integrating presenters in their presentation content as interactive avatars. Using multimodal input, especially body gestures, presenters control those embedded avatars through which they can interact with the virtual presentation environment in a fine-grained fashion, i.e. they are able to manipulate individual presentation elements and data as virtual props. The goal of this endeavour is to create novel immersive presentational experiences for live stage performances (talks, lectures etc.) as well as for remote conferencing in more confined areas such as offices and meeting rooms. Related Publication
|
|
 |
Smart Ubiquitous Projection: Discovering Adequate Surfaces for the Projection of Adaptive Content In this work, we revisit the concept of ubuiquitous projection, where instead of considering every physical surface and object as a display, we seek to determine areas that are suitable for the projection and interaction with digital information. We achieve this using mobile projector-cameras units (procams) and a computer vision technique to automatically detect rectangular surface regions with properties that are desirable for projection (uniform, pale, non-reflective, planar etc.). In a next step, we explore body-based interactions to adaptively lay out content in those recognised areas. Related Publication
|
|
 |
BodyLenses – Embodied Magic Lenses and Personal Territories for Wall Displays
Magic lenses are popular tools to provide locally altered views of visual data. In this work, we introduce the concept of BodyLenses, special kinds of magic lenses for wall displays that are mainly controlled by body interactions. Using body position, arm gestures, distance to the display and classic multitouch on the screen, we show how parameters such as lens position, shape, function and tool selection can be dynamically and intuitively modified by users. Related Publication
|
|
 |
 |
Sensing Techniques for Tablet+Stylus Interaction
Using a special grip- and motion-sensitive stylus and a grip-sensitive tablet, we explore a range of novel pen and touch interactions including detecting how the user holds the pen and the tablet, distinguishing between the pen-holding hand and the bare hand, discarding touches caused by resting palms while writing (palm rejection) and a number of contextual gestures resulting from the detection of those different postures. Related Publication
|
 |
 |
Handheld Devices as Eyes-Free Touch Toolboxes for Pen-Based Interactive Whiteboards
In this project, we investigate how smartphones can be used as portable quick-access toolboxes held by the non-dominant hand to provide assistive touch commands for pen-driven whiteboard tasks. In particular, we consider an eyes-free UI design, which allows users to operate the handheld device in a blind manner, i.e. without having to look at it, thereby allowing them to concentrate on the pen task. Related Publication
|
 |
Pen-Based Spatial Queries on Interactive Maps
In this work, we present and evaluate a set of pen-based techniques to annotate maps on tablets or interactive tabletops and selectively convert those annotations into spatial queries allowing users to search for points of interests within explicitly or implicitly specified scopes, e.g. look for restaurants, hotels etc. within circled areas or along sketched paths or calculated routes. Related Publication
|
|
  |
Document Engineering on Pen and Touch Tabletops
Digital tabletops operated using hybrid pen and touch input provide rich interaction possibilities. As interactive workdesks within the office of the future, they stand to support knowledge workers in a number of productivity tasks, many of which are likely to involve documents. This project aims to leverage the potential of those systems to support document-centric activities, especially the editing and authoring of documents. In particular, the practicality of post-WIMP designs based on bimanual gestures is explored. Related Publications
The active reading application was also featured twice on national Swiss TV (SF 1):
Also check this UML diagram creation tool some of my students developed in a lab project: Youtube video |
|
 |
Properties of Pen and Touch Input
Combined bimanual pen and touch input is a relatively new interaction paradigm with promising prospects. Its properties are not yet well understood and hence merit to be studied. This project experimentally investigates and reports on some important issues of pen and touch input on horizontal surfaces, including aspects of speed, accuracy and coordination. Related Publication
|
|
 |
Adaptive Web-Page Layout for Large Screens
The vast majority of web pages adapt very poorly to large displays, especially widescreens. We propose techniques to produce and evaluate adaptive web pages using web standards (and especially features of HTML5 and CSS3). We address issues such as multi-column layouts, scale-dependent element selection and positioning, font size, line lengths etc. Related Publications
|
|
 |
Automatic Extraction of Visually Salient Pages of Large Documents
This technique attempts to automatically select a given number of pages from a document that visually "stand out" with a view to including them in a document list with thumbnails of sample pages (e.g. a catalogue or an online book store). The algorithm considers a set of low-level features such as element block sizes and tone saliency to determine pages that are more likely to attract attention. A smoothing function is available to inject some level of spread in the culling process. Related Publications
|
|
 |
 |
Touch Scan-n-Search: A Touchscreen Interface To Retrieve Online Versions of Scanned Documents
This system tackles the problem of finding online content based on paper documents through an intuitive touchscreen interface designed for modern scanners and multifunction printers. Touch Scan-n-Search allows the user to select elements of a scanned document (e.g. a newspaper article) and seamlessly connect to common web search services in order to retrieve the online version of the document along with related content. This is achieved by automatically extracting keyphrases from text elements in the document (obtained by OCR) and creating tappable GUI widgets to allow the user to control and fine-tune the search requests. Related Publications
|
 |
SmartPublisher - Document Creation on Pen-Based Systems Via Document Element Reuse
SmartPublisher is a powerful, all-in-one application for pen-based devices with which users can quickly and intuitively create new documents by reusing individual image and text elements acquired from analogue and/or digital documents. The application is especially targeted at scanning devices with touch screen operating panels or tablet PCs connected to them (e.g. modern multifunction printers with large touch screen displays), as one of its main purposes is reuse of material obtained from scanned paper documents. Related Publications
|
|
 |
Document Layout Recognition and Template Matching
This system allows the user to draw rough frames with a stylus or use a scanned drawing to create placeholders for content to be inserted in (e.g. photos for a photo album). Based on the hand-drawn shapes, queries to search for matching templates can also be issued. The user can then select an appropriate template and automatically map content to its placeholders. Related Publications
|
|
 |
Advanced UI for Efficient Document Element Transfer on High-End Multifunction Printers
This application designed for pen-operated multifunction printers integrates 2 modules that support users to send and share elements of scanned documents. The advanced scan2E-Mail function allows users to send only the desired portion of the scanned document as well as extracted text directly in the E-Mail body. The size and compression level of the sent content can also be adapted to the target recipient device (e.g. a mobile phone). The second module gives users the possibility to gather and send document elements to their work PC. The contents appear in a sidebar from which they can be dragged and dropped into desktop applications such as a word processor. Related Publications
|
|
 |
Printing Web Pages with Embedded Videos
Attempting to print a web page with embedded multimedia content with a standard web browser will at best return a printout with a single frame in lieu of the video. This technique, meant as a browser plugin, extracts a number of relevant frames to be included in the printout to recover some of the lost context of the video. The result is a document containing strips of representative movie frames at the location of the video or at the end of the document. Related Publication
|
|
(click and use arrow keys to turn pages) |
Interactive Animated Document Icons
Interactive Animated Document Icons or IADIs are full documents rendered in thumbnail size to be integrated in document lists or file browsers. Pages of an IADI can be "turned" following a user-defined trigger (mouse hover, wheel or keyboard). A magnifying function is also available to zoom the pages for quick previews. Related Publications
|
|
 |
Automatic Document Compiler
The goal of this project is to provide a comprehensive solution to gather and aggregate content relevant to the user from heterogeneous sources (e.g. news articles about a particular topic) and compile the elements into a single coherent document with an appropriate layout. Several criteria are considered to define the constraints used to produce the layout: user preferences, target medium constraints, aesthetic layout rules and semantic similarity between items. Related Publications
|
|
 |
Document Element Extraction and Search
This work deals with the extraction of document components from existing office documents in order to populate a database of reusable elements. Those elements can then be retrieved via an ad hoc web interface (using keywords, but also content-based searching) and inserted into new documents. Related Publications
|
