Jingjie Zheng
Who?
I am a Software Engineer at Google, where I architect the future of FinOps & Billing experiences for Google Cloud. In my spare time, I design and evaluate techniques to make computing more expressive, efficient, intelligent, and reliable. Previously, I studied Human-Computer Interaction (HCI) at University of Waterloo with Daniel Vogel. This web page primarily shows my research portfolio.
Research
Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements (EMNLP 2020)
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan
We proposed widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces.
Reducing Spatial Jitter caused by Asynchronous Input and Output Rates (UIST 2020)
Axel Antoine, Mathieu Nancel, Ella Ge, Jingjie Zheng, Navid Zolghadr, Géry Casiez
We characterized, modelled, and proposed solutions to reduce spatial jitter caused by asynchronous input and output rates on mobile devices.
Using Bayes' Theorem for Command Input (CHI 2020)
Suwen Zhu, Yoonsang Kim, Jingjie Zheng, Jennifer Yi Luo, Ryan Qin, Liuping Wang, Xiangmin Fan, Feng Tian, Xiaojun Bi
We formalized Bayes' theorem as a generic guiding principle for deciding targets in command input, developed three models for estimating prior and likelihood probabilities, and carried out experiments to demonstrate the effectiveness of this formalization.
i'sFree: Eyes-Free Gesture Typing (CHI 2019)
Suwen Zhu, Jingjie Zheng, Shumin Zhai, Xiaojun Bi
We proposed i'sFree, an interaction technique that enables eyes-free gesture typing on a distant display from a touch-enabled remote control.
HotStrokes: Word-Gesture Shortcuts on a Trackpad (CHI 2019)
Wenzhe Cui, Jingjie Zheng, Blaine Lewis, Daniel Vogel, Xiaojun Bi
We proposed HotStrokes to enable entering shortcut commands by holding a modifier key, then gesture typing a command name on a laptop trackpad as if on an imaginary virtual keyboard.
FingerArc & FingerChord (UIST 2018)
Jingjie Zheng, Blaine Lewis, Jeff Avery, Daniel Vogel
We conducted interviews with expert computer users, identified key issues that hinder the wide use of keyboard shortcuts, and proposed FingerArc and FingerChord to support novice to expert transitions.
M3 Gesture Menu (CHI 2018)
Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, Shumin Zhai
We addressed the adoption challenges of marking menus in practice by designing, implementing, and evaluating M3 Gesture Menu (M3), a re-imagination of marking menus targeted at mobile interfaces.
Finger-Aware Shortcuts (CHI 2016)
Jingjie Zheng, Daniel Vogel
We leveraged computer vision and machine learning techniques to identify the hand, finger, and posture on a laptop keyboard when a key is being pressed to enable more shortcut commands.
Paper (ACM DL) | Video | Talk
Knee Image Registration (SIViP)
Jingjie Zheng, Zhenyan Ji, Kuangdi Yu, Qin An, Zhiming Guo, Zuyi Wu
We proposed a feature-based solution to align the human knee images in the modalities of Computed Tomography (CT) and Magnetic Resonance (MR) to construct 3D models for disease diagnosis and stress analysis.
Conference Papers
[C8] Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan. 2020. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP '20). Association for Computational Linguistics, 5495–5510. DOI:https://arxiv.org/abs/2010.04295
[C7] Axel Antoine, Mathieu Nancel, Ella Ge, Jingjie Zheng, Navid Zolghadr, and Géry Casiez. 2020. Modeling and Reducing Spatial Jitter caused by Asynchronous Input and Output Rates. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST '20). Association for Computing Machinery, New York, NY, USA, 869–881. DOI:https://doi.org/10.1145/3379337.3415833
[C6] Suwen Zhu, Yoonsang Kim, Jingjie Zheng, Jennifer Yi Luo, Ryan Qin, Liuping Wang, Xiangmin Fan, Feng Tian, and Xiaojun Bi. 2020. Using Bayes’ Theorem for Command Input: Principle, Models, and Applications. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–15. DOI:https://doi.org/10.1145/3313831.3376771
[C5] Suwen Zhu, Jingjie Zheng, Shumin Zhai, and Xiaojun Bi. 2019. I’sFree: Eyes-Free Gesture Typing via a Touch-Enabled Remote Control. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 448, 1–12. DOI:https://doi.org/10.1145/3290605.3300678
[C4] Wenzhe Cui, Jingjie Zheng, Blaine Lewis, Daniel Vogel, and Xiaojun Bi. 2019. HotStrokes: Word-Gesture Shortcuts on a Trackpad. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 165, 1–13. DOI:https://doi.org/10.1145/3290605.3300395
[C3] Jingjie Zheng, Blaine Lewis, Jeff Avery, and Daniel Vogel. 2018. FingerArc and FingerChord: Supporting Novice to Expert Transitions with Guided Finger-Aware Shortcuts. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). Association for Computing Machinery, New York, NY, USA, 347–363. DOI:https://doi.org/10.1145/3242587.3242589
[C2] Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, and Shumin Zhai. 2018. M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, NY, USA, Paper 249, 1–14. DOI:https://doi.org/10.1145/3173574.3173823
[C1] Jingjie Zheng and Daniel Vogel. 2016. Finger-Aware Shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 4274–4285. DOI:https://doi.org/10.1145/2858036.2858355
Journal Articles
[J1] Jingjie Zheng, Zhenyan Ji, Kuangdi Yu, Qin An, Zhiming Guo, and Zuyi Wu. 2015. A feature-based solution for 3D registration of CT and MRI images of human knee. SIViP 9, 1815–1824 (2015). DOI:https://doi.org/10.1007/s11760-014-0660-5
Theses
[T1] Jingjie Zheng. 2015. Enabling Expressive Keyboard Interaction with Finger, Hand, and Hand Posture Identification. UWSpace. http://hdl.handle.net/10012/12371