Research is about achieving long-term goals, often through incremental progress. As the year comes to an end, it’s a good time to step back and reflect on the work that researchers at Microsoft and their collaborators have done to advance the state of the art in computing, particularly by increasing the capabilities and reach of AI and delivering technology experiences that are more inclusive, secure, and accessible. This covers only a sliver of all the amazing work Microsoft Research has accomplished this year, and we encourage you to discover more of the hundreds of projects undertaken in 2019 by exploring our blog further.
Improving the reach and accessibility of AI and machine learning
Machine learning has made a tremendous impact on people’s everyday lives, especially in the latter half of this decade, while also raising significant policy and societal issues for research to address. This year, Microsoft researchers and their collaborators worked to improve the capabilities of machine learning systems and also explored new models that can take the discipline further. They used unique approaches that can make these systems more accessible and inclusive.
In deep learning, Jianfeng Gao’s team released MT-DNN, a model for learning universal language embeddings that combines the strengths of multi-task learning and the language model pre-training of BERT, helping systems quickly develop the semantic understanding necessary for natural language processing. And Xu Tan and his collaborators at Microsoft Research Asia developed MASS, a pre-training method that outperforms existing models at sequence-to-sequence language generation.
In the coming years, breakthroughs in machine learning will emerge by exploring new models beyond the current foundation of using Markov decision processes, particularly as reinforcement learning—a data-hungry approach generally suited to simulation scenarios—becomes more applicable to real-world scenarios. In this podcast, John Langford and Rafah Hosn discuss these new directions in reinforcement learning and their applications to everyday computing, including the real-world RL now deployed in Personalizer, an Azure Cognitive Service. Langford and Alekh Agarwal also hosted a webinar on the foundations of real-world reinforcement learning.
Many machine learning applications benefit from training with very large datasets; however, many potential uses simply do not have enough data for typical approaches to be effective. Enter machine teaching, where domain experts can build bespoke AI models with little data—and no machine learning expertise. In this podcast, Riham Mansour discusses (among other things) LUIS, one of the first Microsoft products to deploy machine teaching concepts in real-world scenarios.
Researchers from Microsoft labs in Redmond, Montreal, New England, Cambridge (UK), India, and Asia came together for NeurIPS 2019. This year, over 300 Microsoft researchers attended the conference and participated in various events.
Another project aimed at further democratizing AI is the Decentralized & Collaborative AI on Blockchain framework with Justin Harris, which enables users to train and maintain models and datasets on the Ethereum network. At NeurIPS 2019, Debadeepta Dey and collaborators presented Project Petridish, an efficient forward neural architecture search algorithm that helps identify suitable neural architectures for a given machine learning task. And Adith Swaminathan and Emre Kiciman’s February blog post explores researchers’ work to improve causal inference modeling, which helps AI better understand “what if” scenarios in a wide variety of contexts.
Enabling responsible, inclusive, human-centered innovation
2019 kicked off with the inaugural ACM FAT* Conference in Atlanta, which focused on fairness, accountability, and transparency in socio-technical systems. Microsoft Research presented four papers at the conference. They covered gender bias in occupation classification, the role of data-driven decision making in reinforcing or amplifying injustices, strategic manipulation of algorithmic decision systems, and the fair allocation of items in scenarios without money, respectively. This work came from the FATE research group at Microsoft, which studies the complex social implications of AI, machine learning, data science, large-scale experimentation, and automation.
At May’s CHI Conference on Human Factors in Computing Systems, Saleema Amershi and her collaborators presented a set of guidelines for human-AI interaction design that brings together more than 20 years of research, recommendations, and best practices around effective interaction with AI-infused systems. Bringing this work together will help designers manage user expectations, moderate the level of autonomy, resolve ambiguity, and provide users with awareness of how systems learn from their behavior.
To ensure that machine learning systems effectively do the jobs we deploy them to do, we must develop a deeper understanding of how they succeed and fail. This paper, from researchers at Microsoft Ram Shankar Siva Kumar and Jeffrey Snover and their collaborators at Harvard, articulates the various ways machine learning systems can fail—either through intentional adversarial attacks or unintentional failures in which the output is formally correct, but unwanted.
Helping to train autonomous systems that can be trusted in real-world applications, the open-source simulator AirSim provides realistic and detailed testing environments. This year, it played host to the NeurIPS competition Game of Drones. In the drone race challenge, participants competed against a Microsoft Research opponent on the same track, working with a level of strategy and maneuvering not generally offered by contests of its kind. Microsoft researchers and collaborators who organized the competition plan to keep it open and add new racing environments. Visit the GitHub repository for more information.
In January, Jenn Wortman Vaughan and Hanna Wallach hosted a webinar on Fairness in Machine Learning, demonstrating how to make detecting and mitigating biases a first-order priority in the development and deployment of machine learning systems.
Creating human-computer interaction that works for all
This year, Microsoft researchers continued their work to make computing more natural, comfortable, and accessible for everyone. At the ACM CHI Conference on Human Factors in Computing Systems, researchers presented a number of papers and demos exploring how to support accessibility for users with cognitive or sensory disabilities. These include studies on whether web browsers’ “reading mode” is truly helpful for people with dyslexia and tools to help VR be more accessible for people with low vision (including tunnel vision, brightness sensitivity, and low visual acuity).
Also presented at CHI was Microsoft Soundscape, a project that uses 3D audio cues to enhance situational awareness and assist with navigation. (You can try the app yourself here.) In this op-ed in the Toronto Sun, Microsoft researcher Bill Buxton elaborates on the importance of work like this, noting that 1 billion people worldwide have some form of disability, making it imperative that we create technologies that support personal autonomy.
Speaking of sound, Nikunj Raghuvanshi’s podcast explores the physics of audio and discusses Project Triton, an acoustic system that models how sound waves behave so that the audio in 3D game environments can be as rich and immersive as the graphics. Project Triton is available for any game via the Unity and Unreal game engine plugins, as part of Project Acoustics, powered by Azure.
At the ACM Symposium on User Interface Software and Technology, Microsoft researchers presented a number of projects that make virtual environments more realistic, tactile, and navigable. Dreamwalker is a VR project that can augment a real-world walking experience with virtual reality—a virtual environment that detects the user’s surroundings in real time and generates a virtual world that accounts for their path and any obstacles—so that you can walk to work in Seattle, but through a virtual Manhattan. Mise-Unseen is a project that uses gaze detection to modify or replace elements of a virtual world while the user’s attention is directed elsewhere. And CapstanCrunch is a VR controller that leverages centuries-old technology, once used to control ropes on sailing ships, to provide effective and inexpensive haptic feedback.
Architectural designer Jenny Sabin installing ADA at the Microsoft campus.
Meanwhile in the physical world, Microsoft researchers partnered in May with students at the Brooklyn Public Library’s Fashion Academy to weave technology into their designs using Project Alava, which aims to develop microcontroller-based systems that are simple to build and code for people with a limited computer science background. At their end-of-program fashion show, students showed garments that incorporated LEDs, motion sensors, and motors. You can read about other areas where Microsoft researchers are working at the intersection of art and science here, including Ada, a first-of-its-kind architectural pavilion that incorporates AI, on display at Microsoft Research Redmond.
Breakthroughs in security, storage, systems, and applications
2019 saw continued progress in the development and adoption of homomorphic encryption, which enables computation on encrypted data, helping to preserve privacy. Microsoft SEAL has become one of the world’s most popular homomorphic encryption libraries, with broad adoption in both academia and industry. In February, Microsoft took the next step in democratizing homomorphic encryption by releasing SEAL for .NET. (The Microsoft SEAL library is open source and available on GitHub.)
In August 2019, Microsoft researchers joined their industry and academic peers for the Homomorphic Encryption Standards Meeting. The group will reconvene at Microsoft Research in Redmond for next year’s meeting this February. Take our webinar to learn more about homomorphic encryption, and listen to this October podcast with Craig Costello for an overview of the year’s developments in cryptography generally, including efforts to prepare for a post-quantum future.
In April, Project Everest took another step forward in its work to build a verified, secure HTTPS ecosystem with the release of EverCrypt, the first fully verified cryptographic provider to meet the security needs of the TLS protocol. Project Everest is a collaboration between Microsoft, Inria, and Carnegie Mellon University.
By 2023, it’s expected that more than 100 zettabytes of data will be stored in the cloud. To meet that need, Project Silica is developing the first-ever storage technology designed from the media up for use in the cloud. This year, the team collaborated with Warner Bros. on a proof of concept, storing the 1978 film Superman on a nearly indestructible piece of glass roughly the size of a drink coaster. This work is part of the Optics for the Cloud Research Alliance, which you can learn more about here or on the Microsoft Research Podcast. Meanwhile, researchers at Microsoft and the University of Washington achieved a “Hello, World!” moment in April for a different way to meet our growing storage needs: They demonstrated the first fully automated system to store and retrieve data in manufactured DNA. (For more on the intersection between computing and biology, listen to this podcast featuring Andrew Phillips, who leads the Biological Computation Group at Microsoft Research Cambridge.)
Cambridge researchers Andy Gordon and Simon Peyton Jones demonstrated the practical impact of fundamental research by exploring how ideas from programming language research could improve one of the world’s most common business applications: the spreadsheet. In this January blog post, they detail how their collaboration with the Microsoft Excel team led to product improvements such as cells that can contain first-class records linked to external data sources and formulas that can compute array values that “spill” into adjacent cells.
At the ACM International Conference on Web Search and Data Mining, Microsoft researchers presented new work in extreme classification, a research area that promises to dramatically improve the speed and quality of algorithms that can answer multiple-choice questions involving uncertainty, where there could be multiple correct answers. Among other things, this work can lead to more relevant recommendations and search results. In this blog post from February, Manik Varma of Microsoft Research India provides a deep dive into extreme classification.
Thanks to gains in computer vision, particularly object detection and classification, video analysis has become far more accurate; however, fast and affordable real-time video analysis is lagging. In December, Microsoft researchers Ganesh Ananthanarayanan and Yuanchao Shu hosted a webinar on Project Rocket, an extensible software stack that leverages the edge and cloud to meet the needs of video analytic applications.
In April, the Microsoft Research Podcast turned its attention to databases—particularly the need for imperative programming that allows for good software engineering practices like modularity, readability, and reusability. In this episode, Karthik Ramachandra discusses Froid, an extensible and language-agnostic framework for imperative functions in databases, which is available as “Scalar UDF Inlining” in Microsoft SQL Server 2019.
Open-source tools and data for the research community
Throughout the year, researchers from Microsoft made a number of projects open source for the benefit of the academic community, including the following:
-
- SandDance, a data visualization tool included in Azure Data Studio, Visual Studio Code, and Power BI
- TensorWatch, an AI debugging and visualization tool
- PhoneticMatching, a component of Maluuba’s natural language understanding platform
- SpaceFusion, a learning paradigm that brings together a palette of different deep learning models for conversational AI
- Icecaps, a toolkit for conversation modeling
- Icebreaker, a deep generative model that minimizes the amount and cost of data required to train a machine learning model
Building on last year’s announcement of Microsoft Research Open Data—an Azure-based repository for sharing datasets—the company developed a set of data use agreements, released them on GitHub, and adopted them for a number of public datasets. This work aims to make research data more readily available in the cloud and to encourage the reproducibility of research.
Supporting and honoring the research community
This year, Microsoft Research introduced the Ada Lovelace Fellowship to support diverse talent from underrepresented groups pursuing doctorates in computing-related fields. You can read about the fellows and their research here. Ten doctoral students were also awarded two-year fellowships as part of the PhD Fellowship program, supporting research in photonics, systems and networking, and AI. Additionally, Microsoft Research awarded Microsoft Research Faculty Fellowships to five early-career faculty members pursuing high-impact breakthrough research. You can read about their work here.
A number of researchers at Microsoft received awards and honors throughout 2019—check out the full list of recipients here.
Finally, we are saying goodbye to Harry Shum, who is leaving the company in February after 23 years, and hello to Microsoft CTO and EVP Kevin Scott, who has assumed Shum’s responsibilities as head of the Microsoft Artificial Intelligence and Research Group. Listen to Scott on the Microsoft Research Podcast here.
We hope you had a good year, and we look forward to a 2020 full of collaboration and exciting breakthroughs. Happy holidays.
To stay up to date on all things research at Microsoft, follow our blog and subscribe to our newsletter and the Microsoft Research Podcast. You can also follow us on Facebook, Twitter, YouTube, and Instagram.
