Jingjie Zheng

Software Engineer & Researcher
jingjie@acm.org


Who?

I am a Software Engineer at Google Canada. I build full-stack systems to make emails dynamic and interactive and help people better connect in large organizations. In my 20% time, I invent and evaluate interaction techniques that bring computers with more expressivity, efficiency, and reliability. Prior to my current position, I studied Human-Computer Interaction (HCI) at University of Waterloo with Daniel Vogel.

Research


Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements (EMNLP 2020)

Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan

We proposed widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces.


Reducing Spatial Jitter caused by Asynchronous Input and Output Rates (UIST 2020)

Axel Antoine, Mathieu Nancel, Ella Ge, Jingjie Zheng, Navid Zolghadr, Géry Casiez

We characterized, modelled, and proposed solutions to reduce spatial jitter caused by asynchronous input and output rates on mobile devices.


Using Bayes' Theorem for Command Input (CHI 2020)

Suwen Zhu, Yoonsang Kim, Jingjie Zheng, Jennifer Yi Luo, Ryan Qin, Liuping Wang, Xiangmin Fan, Feng Tian, Xiaojun Bi

We formalized Bayes' theorem as a generic guiding principle for deciding targets in command input, developed three models for estimating prior and likelihood probabilities, and carried out experiments to demonstrate the effectiveness of this formalization.


i'sFree: Eyes-Free Gesture Typing (CHI 2019)

Suwen Zhu, Jingjie Zheng, Shumin Zhai, Xiaojun Bi

We proposed i'sFree, an interaction technique that enables eyes-free gesture typing on a distant display from a touch-enabled remote control.


HotStrokes: Word-Gesture Shortcuts on a Trackpad (CHI 2019)

Wenzhe Cui, Jingjie Zheng, Blaine Lewis, Daniel Vogel, Xiaojun Bi

We proposed HotStrokes to enable entering shortcut commands by holding a modifier key, then gesture typing a command name on a laptop trackpad as if on an imaginary virtual keyboard.


FingerArc & FingerChord (UIST 2018)

Jingjie Zheng, Blaine Lewis, Jeff Avery, Daniel Vogel

We conducted interviews with expert computer users, identified key issues that hinder the wide use of keyboard shortcuts, and proposed FingerArc and FingerChord to support novice to expert transitions.


M3 Gesture Menu (CHI 2018)

Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, Shumin Zhai

We addressed the adoption challenges of marking menus in practice by designing, implementing, and evaluating M3 Gesture Menu (M3), a re-imagination of marking menus targeted at mobile interfaces.


Finger-Aware Shortcuts (CHI 2016)

Jingjie Zheng, Daniel Vogel

We leveraged computer vision and machine learning techniques to identify the hand, finger, and posture on a laptop keyboard when a key is being pressed to enable more shortcut commands.


Knee Image Registration (SIViP)

Jingjie Zheng, Zhenyan Ji, Kuangdi Yu, Qin An, Zhiming Guo, Zuyi Wu

We proposed a feature-based solution to align the human knee images in the modalities of Computed Tomography (CT) and Magnetic Resonance (MR) to construct 3D models for disease diagnosis and stress analysis.

Conference Papers

  • [C8] Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan. 2020. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP '20). Association for Computational Linguistics, 5495–5510. DOI:https://arxiv.org/abs/2010.04295

  • [C7] Axel Antoine, Mathieu Nancel, Ella Ge, Jingjie Zheng, Navid Zolghadr, and Géry Casiez. 2020. Modeling and Reducing Spatial Jitter caused by Asynchronous Input and Output Rates. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST '20). Association for Computing Machinery, New York, NY, USA, 869–881. DOI:https://doi.org/10.1145/3379337.3415833

  • [C6] Suwen Zhu, Yoonsang Kim, Jingjie Zheng, Jennifer Yi Luo, Ryan Qin, Liuping Wang, Xiangmin Fan, Feng Tian, and Xiaojun Bi. 2020. Using Bayes’ Theorem for Command Input: Principle, Models, and Applications. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–15. DOI:https://doi.org/10.1145/3313831.3376771

  • [C5] Suwen Zhu, Jingjie Zheng, Shumin Zhai, and Xiaojun Bi. 2019. I’sFree: Eyes-Free Gesture Typing via a Touch-Enabled Remote Control. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 448, 1–12. DOI:https://doi.org/10.1145/3290605.3300678

  • [C4] Wenzhe Cui, Jingjie Zheng, Blaine Lewis, Daniel Vogel, and Xiaojun Bi. 2019. HotStrokes: Word-Gesture Shortcuts on a Trackpad. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 165, 1–13. DOI:https://doi.org/10.1145/3290605.3300395

  • [C3] Jingjie Zheng, Blaine Lewis, Jeff Avery, and Daniel Vogel. 2018. FingerArc and FingerChord: Supporting Novice to Expert Transitions with Guided Finger-Aware Shortcuts. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). Association for Computing Machinery, New York, NY, USA, 347–363. DOI:https://doi.org/10.1145/3242587.3242589

  • [C2] Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, and Shumin Zhai. 2018. M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, NY, USA, Paper 249, 1–14. DOI:https://doi.org/10.1145/3173574.3173823

  • [C1] Jingjie Zheng and Daniel Vogel. 2016. Finger-Aware Shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 4274–4285. DOI:https://doi.org/10.1145/2858036.2858355

Journal Articles

  • [J1] Jingjie Zheng, Zhenyan Ji, Kuangdi Yu, Qin An, Zhiming Guo, and Zuyi Wu. 2015. A feature-based solution for 3D registration of CT and MRI images of human knee. SIViP 9, 1815–1824 (2015). DOI:https://doi.org/10.1007/s11760-014-0660-5

Theses