The Azure Kinect camera captures depth information with an infrared light and that data helps make the AI model more accurate. We used an app called Speaker Recorder to manage two video signals from the Azure Kinect camera, the RGB signal and the depth signal. Once the recording was complete, the AI model was applied through a command line tool. To get the full details on how this all came together, check out the Microsoft AI Lab.
The AI model we used is based on the work recently published by the University of Washington. In their research, the university developed a deep neural network that takes two images, one with a background and another one with a person in it. The output of the neural network is a smooth transparency mask.
This neural network was trained with images where the masking work was done manually. The UW researchers used a dataset provided by Adobe with many images where a designer manually created the transparency mask.
With this approach, the neural network can learn how to smooth areas like hair or lose clothing. However, there are some limitations. If the person is wearing something with a similar color to the background, the system renders it as holes in the image which defeats the illusion.
So, what the UW researchers did is to combine this method with another. A second neural network tries to guess the contour just by looking at the image. In the case of our virtual stage we know that we have a person on screen, so the neural network tries to identify the silhouette of that person. Adding this second neural network eliminates the color transparency issue but the small details like hair or the fingers can be an issue.
So, here’s the interesting part. The UW researchers created an architecture called Context Switching. Depending on the conditions, the system can pick the best solution, getting the best of the two.
In our case, because we are using Azure Kinect, we can go a step farther and replace the second neural network with the silhouette provided by the Kinect, which is much more accurate since it’s coming from the depth information captured.
The model is improved even more with another AI technique called adversarial network. We connect the output of our neural network with another neural network that identifies if an image is fake or real. This makes small variations to the original neural network to fool it. The result is a neural network that can create even more natural images.
“As we’ve learned more and more about what we need and the different limits of all the components that make up a supercomputer, we were really able to say, ‘If we could design our dream system, what would it look like?’” said OpenAI CEO Sam Altman. “And then Microsoft was able to build it.”
OpenAI’s goal is not just to pursue research breakthroughs but also to engineer and develop powerful AI technologies that other people can use, Altman said. The supercomputer developed in partnership with Microsoft was designed to accelerate that cycle.
“We are seeing that larger-scale systems are an important component in training more powerful models,” Altman said.
For customers who want to push their AI ambitions but who don’t require a dedicated supercomputer, Azure AI provides access to powerful compute with the same set of AI accelerators and networks that also power the supercomputer. Microsoft is also making available the tools to train large AI models on these clusters in a distributed and optimized way.
At its Build conference, Microsoft announced that it would soon begin open sourcing its Microsoft Turing models, as well as recipes for training them in Azure Machine Learning. This will give developers access to the same family of powerful language models that the company has used to improve language understanding across its products.
It also unveiled a new version of DeepSpeed, an open source deep learning library for PyTorch that reduces the amount of computing power needed for large distributed model training. The update is significantly more efficient than the version released just three months ago and now allows people to train models more than 15 times larger and 10 times faster than they could without DeepSpeed on the same infrastructure.
Along with the DeepSpeed announcement, Microsoft announced it has added support for distributed training to the ONNX Runtime. The ONNX Runtime is an open source library designed to enable models to be portable across hardware and operating systems. To date, the ONNX Runtime has focused on high-performance inferencing; today’s update adds support for model training, as well as adding the optimizations from the DeepSpeed library, which enable performance improvements of up to 17 times over the current ONNX Runtime.
“We want to be able to build these very advanced AI technologies that ultimately can be easily used by people to help them get their work done and accomplish their goals more quickly,” said Microsoft principal program manager Phil Waymouth. “These large models are going to be an enormous accelerant.”
In “self-supervised” learning, AI models can learn from large amounts of unlabeled data. For example, models can learn deep nuances of language by absorbing large volumes of text and predicting missing words and sentences. Art by Craighton Berman.
Learning the nuances of language
Designing AI models that might one day understand the world more like people do starts with language, a critical component to understanding human intent, making sense of the vast amount of written knowledge in the world and communicating more effortlessly.
Neural network models that can process language, which are roughly inspired by our understanding of the human brain, aren’t new. But these deep learning models are now far more sophisticated than earlier versions and are rapidly escalating in size.
A year ago, the largest models had 1 billion parameters, each loosely equivalent to a synaptic connection in the brain. The Microsoft Turing model for natural language generation now stands as the world’s largest publicly available language AI model with 17 billion parameters.
This new class of models learns differently than supervised learning models that rely on meticulously labeled human-generated data to teach an AI system to recognize a cat or determine whether the answer to a question makes sense.
In what’s known as “self-supervised” learning, these AI models can learn about language by examining billions of pages of publicly available documents on the internet — Wikipedia entries, self-published books, instruction manuals, history lessons, human resources guidelines. In something like a giant game of Mad Libs, words or sentences are removed, and the model has to predict the missing pieces based on the words around it.
As the model does this billions of times, it gets very good at perceiving how words relate to each other. This results in a rich understanding of grammar, concepts, contextual relationships and other building blocks of language. It also allows the same model to transfer lessons learned across many different language tasks, from document understanding to answering questions to creating conversational bots.
“This has enabled things that were seemingly impossible with smaller models,” said Luis Vargas, a Microsoft partner technical advisor who is spearheading the company’s AI at Scale initiative.
The improvements are somewhat like jumping from an elementary reading level to a more sophisticated and nuanced understanding of language. But it’s possible to improve accuracy even further by fine tuning these large AI models on a more specific language task or exposing them to material that’s specific to a particular industry or company.
“Because every organization is going to have its own vocabulary, people can now easily fine tune that model to give it a graduate degree in understanding business, healthcare or legal domains,” he said.
The 18th annual Imagine Cup saw thousands of students across the world submitting innovations to impact their communities, both locally and globally. The competition advanced through hackathons, Online Semifinals, and virtual Regional Final events bringing together finalists via Microsoft Teams.Six teamswere selected to move forward to the World Championship and present their projects to compete for the 2020 trophy on the biggest stage yet—the Microsoft Build digital event.
Congratulations to Team Hollo from the University of Hong Kongwho won the grand prize for their mental health companion web app leveraging Azure analytics and AI services to advance youth therapy practices. The team won USD100,000, a mentoring session with Microsoft CEO Satya Nadella, and USD50,000 in Azure grants.
Imagine Cup aims to empower students to use their imagination and passion for technology to develop innovative and inclusive solutions that tackle key societal issues. With an increasing focus on bringing the world together digitally, we’re continuously encouraged by the projects young developers create to make a difference. This year’s finalist solutions covered issues in healthcare, agriculture, misinformation detection, and more. Because of global health concerns due to COVID-19, we decided to move this year’s competition to a virtual format, and World Finalists pitched their projects during Microsoft Build’s inaugural digital event.
The 2020 World Championship was judged by innovation experts spanning a variety of technology, diversity, and social change-centered experiences that align to the core of Imagine Cup’s mission. Across her extensive career as a tech innovator and leader Dwana Franklin-Davis, CEO of Reboot Representation, has worked to empower underrepresented groups in technology. The CyberCode Twins, America and Penelope Lopez, are young innovators who’ve participated in tech competitions across the world and hope to introduce other students to opportunities in the field. Finally, Microsoft president, Brad Smith, leads work on a wide range of issues involving the intersection of technology and society, including ethics and AI, human rights, and environmental sustainability.
Kicking off the World Championship with a short pitch of their projects in the lightning round, judges selected Team Hollo, along with Team Syrinx from Japan and Team Tremor Vision from the United States, to advance to the second and final round of the competition. Each of the top three teams then gave an in-depth presentation their solutions, which were assessed for their diversity, originality, and innovative design. Watch the show to experience the moment when Team Hollo is crowned champion!
Meet the top 3 World Championship teams:
2020 Imagine Cup World Champion – Team Hollo, Hong Kong SAR
After meeting in school, the team brought a collaborative desire for social change focusing on young people and inequality to their project,combined with a passion for tech. Team Hollo noticed that mental health aid was not effectively reaching the younger generations in their communities and envision a future of tech-based, accessible, and comprehensive mental health management tools. Team member Cameron van Breda said, “The experience of mental health is felt by everyone… being able to win Imagine Cup and support the awareness of mental health and be able to work on it with technology is something that’s really needed in the world.”
As a smart AI-powered preventative platform, the app aims to improve individual mental health by integrating Machine Learning with suggestive diagnosis, therapy, and continual monitoring, in order to facilitate self-help and professional therapeutic services. With an interactive mobile app for users and a comprehensive case management dashboard for therapists & NGOs, Hollopresents a collaborative and scalable mental health platform which aims to cut the heavy cost of mental health care for youth and make support more accessible. Team member Piyush Jha noted, “The accessibility part of this application makes it really welcoming for anyone and everyone, anywhere in the world, to make sure they’re doing well.”
When asked about future plans,Hollo responded, “We’re going to be able to implement a lot of current research… and really be able to create a collaborative effort in the mental health space. We feel like that’s really important, getting this global perspective.”
Runner-up – Syrinx, Japan
Syrinx, a neck wearable EL (electrolarynx) device, restores the ability to speak for people who have lost their voices. Around the globe, over 300 thousand people lose the ability to speak each year. However, traditional electrolarynx devices to restore speaking capabilities can only produce a monotonic robot-like voice and require the use of one hand to talk. This can cause social and communication issues for the user, which the team hopes to solve.
Leveraging Azure Notebooks technology, Syrinx’s device vibrates the user’s throat to create a sound that matches the movement of their mouth, then uses neural networks to learn the characteristics of the lost voice in order torestore it. This allows Syrinx to produce a more natural sounding human voice through their device, including both male and female voice patterns. “Next step is to increase users for the beta test to try out this device. I’d like to improve the function of our device for a more human-like voice,” commented team member Masaki Takeuchi. “Syrinx will be a device for not only laryngectomy or tracheostomy patients, but also people all over the world.”
Runner-up – Tremor Vision, United States
Tremor Vision is a web-based tool that enables physicians to detect early-onset Parkinson’s and quantitatively track patient progress throughout a prescribed treatment plan. Although Parkinson’s is the second most common age-related neurodegenerative disorder, there is currently no standardized way of evaluating the spiral test, one of the most widely conducted examinations to detect early-onset of thedisease and track its progression. Team member Robert Minneker commented, “The why and how is really important when bringing technology to life. Getting it to the people that need it and being able to communicate why it matters is huge.”
By using a touchscreen device connected to the internet, users of Tremor Vision’s tool can send clinical results to their physician. The platform empowers patients to save the time and money required by routine clinical visits and increases a physician’s reach in screening for early signs of Parkinson’s. The team uses Microsoft Azure Cognitive Services, MATLAB, and Microsoft Visual Studio.“We put 100% of our effort into making something really cool”, shared team member Janae Chan. “We started in a place where we weren’t really sure if we should or if this idea is worthwhile to present. I think any idea is worthwhile as long as you believe in it and you’re giving back to the people that you’re trying to help.”
————–
Registration for the 2021 competition is now open. Join over two million student competitors worldwide in making an impact in what you’re most passionate about and sign up for Imagine Cup today!
One of the defining aspects of COVID-19 is its disproportionate impact on underserved communities and the harsh spotlight it shines on existing social equity issues around the world. From access to quality education, jobs or affordable healthcare, COVID-19 is magnifying virtually every inequality in our communities.
Never has there been a more important time to capture the moment to create the solutions the world needs to make a positive and lasting contribution to the social inequity issues of our generation. Solutions will come from all corners and technology innovators will need to play their part.
Building on Microsoft’s long-standing efforts to ensure technology fulfills its promise to address the world’s biggest challenges, Microsoft joined efforts with Giving Tech Labs to unleash the power of public interest technology. This week, at Build 2020, we are offering developers a preview of X4Impact, the innovation hub spawned by this collaboration, and the opportunity to demo this powerful tool. Built on Azure, X4Impact is an AI-powered market intelligence platform for social innovation where people can define social challenges, contribute ideas, access solutions and identify funding.
YouTube Video
Clearly the challenges are complex and will require strong vision and collaboration across governments, nonprofits, donors and the private sector. The power of AI, data science and high-performance cloud computing have created an unprecedented ability to produce insights and solutions for critical issues.
Take, for example, our work with the COVID-19 High Performance Computing Consortium led by the White House. Bringing together the Federal government, industry and academics, Microsoft is providing researchers in computer science, biology, medicine and public health access to the world’s most powerful computing resources. This collaboration is helping speed the pace of scientific discovery of treatments and a potential vaccine for COVID-19.
While this is a powerful scenario, it’s only one example. The world needs thousands of these solutions to meet the wide-ranging issues that we’re facing today.
Part of the answer lies in unlocking the power of technology for the public interest – a field dedicated to deploying advanced technology, data science, AI and sustainability models to address urgent issues in society. It is about building solutions that work because they reflect the needs of the communities they serve. To achieve this, technology for public interest encompasses important principles such as:
Bringing nonprofits, government, the private sector and donors together to drive change through a focus on empathy and inclusion in the design of solutions
Using ethical AI to transform data into knowledge with a relentless focus on measurable impact for the communities needing help
Recognizing that, because technology alone does not solve problems, building long-lasting, sustainable processes and capacity is essential
What does technology for public interest mean for those working on the front lines? Let’s take clean drinking water as an example. Fighting cholera is one of the world’s most pressing needs. Five million cholera cases are recorded across the globe each year and $3 billion is spent annually in treatments and lost productivity that could be avoidable through early detection. Having lost family members of her own to cholera, Dr. Katherine Clayton, an engineer by training, founded a startup called OmniVis, which has now developed a cloud-based platform that uses a smartphone and mobile, affordable hardware to test water in the field and produce cholera analysis and insights. The relative affordability and speed at which results are returned will allow NGOs to alert nearby communities before an outbreak spreads. This will help save lives.
At Microsoft, we are committed to being a catalyst to help thousands of organizations like OmniVis pursue their technology for public interest ideas. That’s why, in February, we launched a new Global Social Entrepreneurship program to offer qualified startups access to technology, education, customers and grants. Our global initiative is designed to help social entrepreneurs build and scale their organizations to do good globally. The program is available in 140 countries and will actively seek to support underrepresented founders with diverse perspectives and backgrounds.
In this environment of collective problem-solving, we need an easy way for developers to identify the greatest unmet needs, whether through cholera detection or COVID-19 treatments, where technology can play a critical role in helping address these challenges. Similarly, we need to map these social challenges to available funding sources and collaborators to fully understand the opportunities for solution creation.
X4Impact will help social entrepreneurs, nonprofits, citizen developers, funders and foundations identify where they can deploy their time and talent to collectively build a better world. Leveraging the power of AI, X4Impact aggregates content from hundreds of thousands of IRS 990 and 990-PF filings, private investing filings with the SEC and active grants from the federal government, foundations and private companies, in addition to content from over 5,000 trusted sources. The result is over 30 million units of knowledge indexed under the 17 United Nations Sustainable Development Goals and 231 impact indicators. With access to this market intelligence, we can collectively build much-needed solutions at a new level of scale and impact.
While the platform will launch this July, we call on tech trailblazers to join the public interest movement now by registering at x4i.org to receive an invitation to demo the platform. This work builds on our current offers for all nonprofits and we recommend reviewing our COVID-19 Resource Guide for Nonprofits to learn about additional support. At Microsoft, we are committed to learning how to better drive social innovation each day while evolving our social business model to help move nonprofit missions forward and drive social good.
Panos Panay, chief product officer, corporate vice president Windows + Devices, joins Scott Hanselman, Microsoft partner program manager, in a discussion at Build 2020.
Scott Guthrie, executive vice president of Microsoft’s Cloud + AI Group, talks with Julia White, corporate vice president, Microsoft Azure, and Gerri Martin-Flickinger, executive vice president and chief technology officer for Starbucks.
For the past several years, developers from around the world have gathered here in Seattle, the original “Cloud City,” to meet, exchange ideas, hone or learn skills, build community. This year, we won’t be meeting in person, but the spirit of the event will continue as we move online – and the response we’ve received from our community of developers looking to learn, connect and code at Microsoft Build 2020 has been overwhelming, and quite frankly, humbling. This will likely be our biggest event ever … not just for Build, but for all the events that Microsoft holds. Wow.
The title of our show is a fitting mantra to this unprecedented era we find ourselves in: Build.
Socrates (not the Greek philosopher; the gas station attendant) once said: “The secret of change is to focus all of your energy, not on fighting the old, but on building the new.” We believe the developer community is the next wave to join the ranks of those who will build and re-build organizations, industries, communities.
This week, we celebrate the critical role of developers and their tireless efforts to rally during this time of crisis. We’ve unveiled a range of new tools and services to meet their needs to provide immediate impact and value, empowering innovations that help organizations and individuals achieve more.
As work environments evolve, you’ll see how we’re creating solutions to help companies build, rebuild and thrive, including new tools that enable developers to design and deliver artificial intelligence (AI) applications in an ethical and responsible way, as well as help them build connected productivity experiences.
You’ll see an emphasis on impact and value, delivering solutions within Azure, M365 and Windows – from tools to help developers be more collaborative and productive at work, to services that give customers the flexibility to deploy AI capabilities in any environment – and with no prior coding experience.
And finally, you’ll see a nod to technical excellence, and how we’re looking to help developers achieve more in the future through AI and other technology advancements.
Key news highlights:
We’re introducing Azure Synapse Link, bringing operational database services and analytics together in real-time. Launched initially in Azure Cosmos DB, but coming soon to all operational systems, Azure Synapse Link helps customers lower costs and reduce time to gain valuable insights without managing data movement.
Platform enhancements to Microsoft Teams include a streamlined experience for developers to build and publish Teams apps from Visual Studio and Visual Studio Code; the ability for IT admins to evaluate and deploy line-of-business and ISV applications for their users in Teams; and new ways for people to discover and engage with apps in Teams.
We are announcing updates to Fluid Framework, including making it open source to developers, and introducing the first way for end users to experience Fluid with the upcoming availability of Fluid components and Fluid workspaces in Office.com and Outlook for the Web.
We’re delivering new Responsible ML tools in Azure Machine Learning and our OSS toolkits to help customers deploy AI models more responsibly by improving model interpretability, reducing unfairness while ensuring data privacy and confidentiality.
To help unify app development across 1 billion Windows 10 devices we’re introducing Project Reunion: our vision for evolving the Windows developer platform to make it easier to integrate across Win32 and UWP APIs and build great apps that work across all the Windows 10 versions and devices people use.
We’re further investing in bringing comprehensive low-code Robotic Process Automation (RPA) technology into Power Automate with the acquisition of Softomotive, a leading provider of low-code, easy-to-use RPA development environments. Softomotive’s technology will complement UI flows to streamline how our customers get work done.
We’re announcing one of the world’s most powerful AIsupercomputers built in Azure. Developed in collaboration with and exclusively for OpenAI, this supercomputer is purpose-built to train massive distributed AI models, giving it all the benefits of a dedicated appliance paired with the benefits of Azure’s robust modern cloud infrastructure.
We’re highlighting pitches from six finalist teams in this year’s Imagine Cup, and unveiling the 2020 champion as part of Microsoft’s commitment to helping students develop big, bold ideas by providing tools, programs and technology to learn the skills they’ll need to create. Top ideas include solutions that help improve treatment of youth living with mental illness; tools that help battle misinformation in the media; and technology that better enable physicians to detect early onset Parkinson’s disease and track patient progress throughout treatment plans.
And lastly, we’re introducing Microsoft Cloud for Healthcare, our first industry-specific cloud offering, which brings together capabilities for customers and partners to enrich patient engagement, connect caregiving teams, and improve collaboration, decision-making and operational efficiencies. Microsoft Cloud for Healthcare will support capabilities such as the new Bookings app in Teams, now generally available to customers across industries to help schedule, manage and conduct business-to-consumer virtual appointments. Teams supports HIPAA compliance and is HITRUST-certified.
Thanks for joining us this year as we try something – and build something – new together.
Be sure to check out all the highlights on our Microsoft Build site – including key segments from Microsoft executives – and other session content available virtually.
Anyone who runs a business knows that one of the hardest things to do is accuse a customer of malfeasance. That’s why, before members of Scandinavian Airlines’ fraud detection unit accuse a customer of attempting to scam the carrier’s loyalty points program, the detectives need confidence that their case is solid.
“It would hurt us even more if we accidentally managed to say that something is fraud, but it isn’t,” said Daniel Engberg, head of data analytics and artificial intelligence for SAS, which is headquartered in Stockholm, Sweden.
The airline is currently flying a reduced schedule with limited in-flight services to help slow the spread of COVID-19, the disease caused by the novel coronavirus. Before the restrictions, SAS handled more than 800 departures per day and 30 million passengers per year. Maintaining the integrity of the EuroBonus loyalty program is paramount as the airline waits for regular operations to resume, noted Engberg.
EuroBonus scammers, he explained, try to gain as many points as quickly as possible to either book reward travel for themselves or to sell. When fraud occurs, legitimate customers lose an opportunity to claim seats reserved for the loyalty program and SAS loses out on important business revenue.
Today, a large portion of leads on EuroBonus fraud come from an AI system that Engberg and his team built with Microsoft Azure Machine Learning, a service for building, training and deploying machine learning models that are easy to understand, protect and control.
The SAS AI system processes streams of real-time flight, transaction, award claims and other data through a machine learning model with thousands of parameters to find patterns of suspicious behavior.
To understand the model predictions, and thus chase leads and build their cases, the fraud detection unit relies on an Azure Machine Learning capability called interpretability, powered by the InterpretML toolkit. This capability explains what parameters were most important in any given case. For example, it could point to parameters that suggest a scam of pooling points from ghost accounts to book flights.
Model interpretability helps take the mystery out of machine learning, which in turn can build confidence and trust in model predictions, noted Engberg.
“If we build the trust in these models, people start using them and then we can actually start reaping the benefits that the machine learning promised us,” he said. “It’s not about explainability for explainability’s sake. It’s being able to provide both our customers and our own employees with insights into what these models are doing and how they are taking positions for us.”
Graphic courtesy of Microsoft.
Understand, protect and control your machine learning solution
Over the past several years, machine learning has moved out of research labs and into the mainstream, and has transformed from a niche discipline for data scientists with Ph.D.s to one where all developers are expected to be able to participate, noted Eric Boyd, corporate vice president of Microsoft Azure AI in Redmond, Washington.
Microsoft built Azure Machine Learning to enable developers across the spectrum of data science expertise to build and deploy AI systems. Today, noted Boyd, all developers are increasingly asked to build AI systems that are easy to explain and that comply with non-discrimination and privacy regulations.
“It is very challenging to have a good sense of, ‘Hey, have I really assessed whether my model is behaving fairly?’ or ‘Do I really understand why this particular model is predicting the way it is?’” he said.
To navigate these hurdles, Microsoft today announced innovations in responsible machine learning that can help developers understand, protect and control their models throughout the machine learning lifecycle. These capabilities can be accessed through Azure Machine Learning and are also available in open source on GitHub.
The ability to understand model behavior includes the interpretability capabilities powered by the InterpretML toolkit that SAS uses to detect fraud in the EuroBonus loyalty program.
In addition, Microsoft said the Fairlearn toolkit, which includes capabilities to assess and improve the fairness of AI systems, will be integrated with Azure Machine Learning in June.
Microsoft also announced that WhiteNoise, a toolkit for differential privacy, is now available to developers to experiment with in open source on GitHub and can also be accessed through Azure Machine Learning. The differential privacy capabilities were developed in collaboration with researchers at the Harvard Institute for Quantitative Social Science and School of Engineering.
Differential privacy techniques make it possible to derive insights from private data while providing statistical assurances that private information such as names or dates of birth can be protected.
For example, differential privacy could enable a group of hospitals to collaborate on building a better predictive model on the efficacy of cancer treatments while at the same time helping to adhere to legal requirements to protect the privacy of hospital information and helping to ensure that no individual patient data leaks out from the model.
Azure Machine Learning also has built-in controls that enable developers to track and automate their entire process of building, training and deploying a model. This capability, known to many as machine learning and operations, or MLOps, provides an audit trail to help organizations meet regulatory and compliance requirements.
“MLOps is really thinking around the operational, repeatable side of machine learning,” said Boyd. “How do I keep track of all the different experiments that I have run, the parameters that were set with them, the datasets that were used in creating them. And then I can use that to recreate those same things.”
Sarah Bird, Microsoft’s responsible AI lead for Azure AI based in New York City, helps create tools that make responsible machine learning accessible to all developers. Photo courtesy of Sarah Bird.
Contextual bandits and responsibility
In the mid-2010s, Sarah Bird and her colleagues at Microsoft’s research lab in New York were working on a machine learning technology called contextual bandits that learn through exploration experiments how to perform specific tasks better and better over time.
For example, if a visitor to a news website clicks on a story about cats, the contextual bandit learns to present the visitor more stories about cats. To keep learning, the bandit performs experiments such as showing the visitor stories about the Jacksonville Jaguars, a sports team, and the hit musical “Cats.” What story the visitor clicks is another learning data point that leads to greater personalization.
“When it works, it is amazing, you get personalization lifts that you’ve never seen before,” said Bird, who now leads responsible AI efforts for Azure AI. “We started talking to customers and working with our sales team to see who wants to pilot this novel research tech.”
The sales leads gave Bird pause. As potential customers floated ideas about using contextual bandits to optimize the job interview process and insurance claim adjudications, she realized that many people lacked understanding on how contextual bandits work.
“I started saying, ‘Is it even ethical to do experimentation in those scenarios?’” Bird recalled.
The question led to discussions with colleagues in the Fairness, Accountability, Transparency and Ethics in AI research group, or FATE, and a research collaboration on the history of experimental ethics and the implications for reinforcement learning, the type of machine learning behind contextual bandits.
“The technology is good enough that we are using it for real use cases, and if we are using it for real use cases that affect people’s lives, then we better make sure that it is fair and we better make sure that it is safe,” said Bird, who now focuses full time on the creation of tools that make responsible machine learning accessible to all developers.
Huskies, wolves and scammers
Within a few years, ethical AI research had exploded around the world. Model fairness and interpretability were hot topics at major industry gatherings and responsible machine learning tools were being described in the academic literature.
In 2016, for example, Marco Tulio Ribeiro, now a senior researcher at Microsoft’s research lab in Redmond, presented a technique in an academic conference paper to explain the prediction of any classifier, such as computer vision models trained to classify between objects in photos.
To demonstrate the technique, he deliberately trained a classifier to predict “wolf” if a photo had a snowy background and “husky” if there was no snow. He then ran the model on photos of wolves mostly in snowy backgrounds and huskies mostly without snow and showed the results to machine learning experts with two questions: Do you trust the model? How is it making predictions?
Microsoft senior researcher Marco Tulio Ribeiro found that many machine learning experts trusted this model that predicts whether an image is of a wolf or husky. Then he gave them the model explanation, which shows the predictions are based on whether there is snow in the background. “Even experts are likely to be fooled by a bad model,” he said. Graphic courtesy of Microsoft. Photos via Getty.
Many of the machine learning experts said they trusted the model and presented theories on why it was predicting wolves or huskies such as wolves have pointier teeth, noted Ribeiro. Less than half mentioned the background as a potential factor and almost no one zeroed in on the snow.
“Then I showed them the explanations, and after seeing the explanations, of course everyone basically got it and said, ‘Oh, it is just looking at the background,’” he said. “This is a proof-of-concept; even experts are likely to be fooled by a bad model.”
A refined version of Ribeiro’s explanation technique is one of several interpretability capabilities available to all developers using interpretability on Azure Machine Learning, the toolkit that SAS’s fraud detection unit uses to build cases against scammers in the EuroBonus loyalty program.
Other AI solutions that SAS is creating with Azure Machine Learning include one for ticket sales forecasting and a system that optimizes fresh food stocking for in-flight purchases. The fresh food solution reduced food waste by more than 60% before fresh food sales were halted as part of global efforts to slow the spread of COVID-19.
Engberg and his data analytics and artificial intelligence team continue to build, train and test machine learning models, including further experimentation with the Azure Machine Learning capabilities for interpretability and fairness.
“The more we go into things affecting our customers or us as individuals, I think these concepts of fairness, explainable AI, responsible AI, will be even more important,” said Engberg.
Assessing and mitigating unfairness
Bird’s colleagues in FATE pioneered many of the capabilities in the Fairlearn toolkit. The capabilities allow developers to examine model performance across groups of people such as those based on gender, skin tone, age and other characteristics.
“It could be you have a great idea of what fairness means in an application and because these models are so complex, you might not even notice that it doesn’t work as well for one group of people as another group,” explained Bird. “Fairlearn is allowing you to find those issues.”
Eric Boyd, Microsoft corporate vice president of Azure AI in Redmond, Wash., said innovations in responsible machine learning can help developers build AI systems that are easy to explain and comply with non-discrimination and privacy regulations. Photo courtesy of Microsoft.
EY, a global leader in assurance, tax, transaction and advisory services, piloted fairness capabilities in the Fairlearn toolkit on a machine learning model the firm built for automated lending decisions.
The model was trained on mortgage adjudication data from banks that includes transaction and payment history and credit bureau information. This type of data is generally used to enable assessment of the client’s capability and willingness to pay back a loan. But it also raises concerns about regulatory, legal issues and potential unfairness against applicants of specific demographics.
EY used Fairlearn to evaluate the fairness of model outputs with regards to biological sex. The toolkit, which surfaces results on a visual and interactive dashboard, revealed a 15.3 percentage point difference between positive loan decisions for males versus females.
The Fairlearn toolkit allowed the modelling team at EY to quickly develop and train multiple remediated models and visualize the common trade-off between fairness and model accuracy. The team ultimately landed on a final model that optimized and preserved overall accuracy but reduced the difference between males and females to 0.43 percentage points.
The ability for any developer to assess and mitigate unfairness in their models is becoming essential across the financial industry, noted Boyd.
“Increasingly we’re seeing regulators looking closely at these models,” he said. “Being able to document and demonstrate that they followed the leading practices and have worked very hard to improve the fairness of the datasets are essential to being able to continue to operate.”
Responsible machine learning
Bird believes machine learning is changing the world for the better, but she said all developers need the tools and resources to build models in ways that put responsibility front and center.
Consider, for example, a research collaboration within the medical community to compile COVID-19 patient datasets to build a machine learning model that predicts who is at high risk of serious complications from the novel coronavirus.
Before such a model is deployed, she said, the developers need to make sure they understand how it makes decisions in order to explain the process to doctors and patients. The developers will also want to asses fairness, ensuring the model captures the known elevated risks to males, for example.
“I don’t want a model that never predicts that men are high risk, that would be terrible,” said Bird. “Then, obviously, I want to make sure that the model is not revealing the data of the people it was trained on, so you need to use differential privacy for that.”
Top image: An SAS AI-powered fraud detection tool processes streams of real-time flight information along with transaction, award claims and other data through a machine learning model to find patterns of suspicious behavior. An Azure Machine Learning capability called interpretability explains what model parameters were most important in any given case of suspected fraud. Photo courtesy of SAS.
Related:
John Roach writes about Microsoft research and innovation. Follow him on Twitter.
Today, we are announcing that we have signed a definitive agreement to acquire Metaswitch Networks, a leading provider of virtualized network software and voice, data and communications solutions for operators.
The convergence of cloud and communication networks presents a unique opportunity for Microsoft to serve operators globally via continued investment in Azure, adding additional depth to our hyperscale cloud infrastructure with the specialized software required to run virtualized communication functions, applications and networks.
This announcement builds on our recent acquisition of Affirmed Networks, which closed on April 23, 2020. Metaswitch’s complementary portfolio of ultra-high-performance, cloud-native communications software will expand our range of offerings available for the telecommunications industry. Microsoft intends to leverage the talent and technology of these two organizations, extending the Azure platform to both deploy and grow these capabilities at scale in a way that is secure, efficient and creates a sustainable ecosystem.
As the industry moves to 5G, operators will have opportunities to advance the virtualization of their core networks and move forward on a path to an increasingly cloud-native future. Microsoft will continue to meet customers where they are, working together with the industry as operators and network equipment providers evolve their own operations.
We will continue to support hybrid and multi-cloud models to create a more diverse telecom ecosystem and spur faster innovation, an expanded set of unique offerings and greater opportunities for differentiation. We will continue to partner with existing suppliers, emerging innovators and network equipment partners to share roadmaps and explore expanded opportunities to work together, including in the areas of radio access networks (RAN), next-generation core, virtualized services, orchestration and operations support system/business support system (OSS/BSS) modernization. A future that is interoperable has never been more important to ensure the success of customers and partners.
By enabling advancements in enhanced mobile broadband, ultra-reliable low latency communications and massive machine-type communication to enable IoT at scale, 5G offers significant potential for enterprises and governments and in turn creates new opportunities for operators. 5G will ultimately give operators a path to accelerate service innovation and deliver new transformative experiences that are faster, more resilient and more secure, spurred on by software advances to drive transformation at scale.
We have a long history of working with operators as they increasingly embrace software-based solutions and continue to support the advancement of cloud-based networking while helping create new partnership opportunities for existing network equipment providers. Our intention over time is to create modern alternatives to network infrastructure, enabling operators to deliver existing and value-added services – with greater cost efficiency and lower capital investment than they’ve faced in the past.
Our CEO Satya Nadella references a term from the education field that I think is particularly fitting and important during this unprecedented time in history. Satya talks of the importance of being a “learn-it-all” instead of a “know-it-all.” Learn-it-alls are curious, resourceful and willing to fail, understanding that insights from failure lead to future success.
Learn-it-alls see adversity as a challenge to be overcome, and they work toward the future with focus and determination.
Right now, we’re all working toward the future in different ways. And the future itself is evolving rapidly as we work and learn together to fight and defeat COVID-19 across the globe. Work is changing, learning is changing, life is changing. Every person on the planet will need new skills to be successful tomorrow, one year from now, and one decade from now. This is particularly apropos for students (and the educators teaching them), with the World Economic Forum predicting that two-thirds of students today will work in jobs that do not yet exist. Likewise, LinkedIn continues to report cloud and artificial intelligence as top emerging jobs.
Today’s students are the innovators and inventors of the future who can use technology as a bedrock to help find solutions to the types of problems we’re facing today — and those we can’t predict. Educators are key enablers of this ability, and that’s why I’m excited to announce a new set of opportunities and resources for educators to teach Microsoft technical skills aimed at supporting students to continue learning during this pandemic and beyond.
Introducing Microsoft Learn Student Ambassadors
Microsoft Learn Student Ambassadors help their peers learn about things they care about most, from social issues to new technologies. Ambassadors get a first look at new Microsoft technologies, gain leadership skills, and receive mentoring from professionals in the industry, and their peers benefit from their knowledge, which can now be shared via the Microsoft Learn platform. All our incoming 2020 interns are invited to join the Student Ambassadors and it’s open to any higher ed student who wants to apply.
We are aiming to help skill millions of students in the coming years — helping tomorrow’s leaders gain knowledge in areas spanning topics like responsible AI, Internet of Things (IoT), and building cloud-native apps, among so much more.
New hub on Microsoft Learn for educators and students
Students are natural continuing learners — it’s in their DNA. And to make it easier for them to both acquire and transfer knowledge, Microsoft Learn now has a new home just for educators and students, including our Microsoft Learn Student Ambassadors.
We’ve partnered with universities to create new learning paths based on their popular courses in data science, cloud development, and AI engineering, all tailored for the students that want to build in-demand job skills and educators that want to teach them:
We’ve also added a new series of learning paths to inspire and challenge students to build with social impact and responsibility in mind. These take a solution-driven, project-based approach to learning:
We continue to offer foundational developer paths designed especially for students that faculty can easily teach in the classroom. These include:
Educators play a pivotal role in empowering students for future success. At Microsoft, we’re committed to enabling and supporting them in their mission. Microsoft Learn for Educators curates online learning paths and supporting instructor-led training materials into the classroom. Eligible educators and faculty members at universities, community colleges, polytechnics and secondary schools can access Microsoft ready-to-teach curriculum and teaching materials aligned to industry-recognized Microsoft certifications. These certifications augment a students’ existing degree path and validate the skills needed to be successful across a variety of technical careers. Provided Microsoft curriculum and instructor-led training materials will cover:
Last fall, we launched a 44-part video series called Python for Beginners, consisting of short lessons aimed to help students learn Python and then build AI apps on Azure. People kept asking for more, so we’ve expanded on it with 50 additional newvideos that dive deeper into the popular Python libraries like NumPy, Pandas, and Scikit-learn. If you’re looking to try Python for the first time or brush-up your skills, begin here!
Students at Microsoft Build
Microsoft’s annual developer conference, Build, is set to bring the developer community — including student developers and our 2020 class of interns — together virtually May 19-21 to learn, connect and code together. In the spirt of connecting students and professional developers, the Imagine Cup World Championship will be held during Build where teams will compete for the $100,000 grand prize and a mentoring session with Satya Nadella.
The Imagine Cup is perhaps one of the most visible ways we encourage students to address real problems through teamwork and technology. Much like a sports bracket which requires repetitive wins to advance to get the World Championship, teams must win their regional competitions — an impressive feat by itself. This year, tens of thousands of competitors from more than 170 countries participated, culminating in 16 students representing six teams that made it to the championship.
Beyond the excitement of Imagine Cup, the Student Zone at Build will have content tailored to and appropriate for students. Speakers include a variety of top influencers in the digital learning spaces, with content available for each skill level (13-21 years-old) attending our sessions virtually. And, special guest NASA Education Specialist Matthew Wallace will demo a machine learning tool that introduces students to process for analyzing images of Earth taken from the International Space Station, like our astronauts do.
Azure for Students
We believe strongly in providing access to the most current technology, and that’s why we’re providing free Azure accounts, plus a $100 credit, for qualifying students. With their accounts, students can develop in Visual Studio to create custom apps, explore AI through Cognitive Services and smart APIs, and build and train machine learning models faster with the latest open source technologies. Free developer tools are included, as are free learning paths and labs.
I hope you’ll take advantage of all the free content that interests you and join us as learn-it-alls.
As hospitals and other healthcare delivery organizations accelerate their adoption of virtual care and mobile devices in response to the COVID-19 outbreak, it’s critical that providers can access cloud and on-premises apps quickly and securely. Imprivata is a healthcare-focused digital identity company that addresses this need. For today’s “Voice of the ISV” blog, I invited Kristina Cairns and Mark Erwich of Imprivata to provide insight into how Imprivata’s solutions are helping healthcare organizations deliver care beyond the four walls of the hospitals.
Supporting healthcare delivery organizations during COVID-19
By Kristina Cairns, Director of Product Marketing, Imprivata and Mark Erwich, VP Marketing, Imprivata
In response to COVID-19, hospitals and clinics have turned to remote tools to care for a surge of patients, while protecting the health of staff. These tools let clinicians connect remotely with patients, care teams, and other organizations, but they can be difficult to securely access from shared workstations or mobile devices, such as tablets. Imprivata digital identity solutions simplify access while maintaining security, so clinicians can deliver quality care safely and conveniently—no matter where they are located.
At the same time, healthcare staffing demands are skyrocketing, and these needs must be met in real time. This can mean quickly adding, or provisioning, new or re-allocated staff and ensuring they have proper access to applications, immediately. Once the crisis is over, these same staff will need to be de-provisioned to ensure security and compliance requirements are met.
Imprivata is a digital identity company that focuses on healthcare. We employ doctors and nurses who have a real-world understanding of the unique needs of hospital environments. Our solutions are designed to work with healthcare workflows and regulations, so hospitals can get up and running with new tools and upgrades, fast. In these challenging times, we’ve partnered with Microsoft to provide an integrated identity and access management platform that meets the needs of healthcare organizations. Our joint solutions make it easy to connect to healthcare’s existing identity and application data and automate at scale. Healthcare providers can use our platforms to address unique demands, such as:
Saving precious time in hospitals: Accessing necessary apps quickly while healthcare providers move between clinical workstations and new networked devices at the point of patient care.
Protecting healthcare staff and patients: Identifying providers potentially exposed to COVID-19.
Scaling up remote work and virtual care: Providing remote access to a diverse set of tools spanning on-premises and cloud infrastructure as providers and patients move outside of traditional healthcare environments.
Simplifying role-based access identity management: Securely manage access for temporary workers and existing staff who change roles or departments.
Saving precious time in hospitals
Healthcare workers are busy in the best of times. They juggle administrative tasks with a full day of patient care. As the pandemic has driven up the number of patients admitted to hospitals, time has become even more precious. Imprivata OneSign is a single sign-on (SSO) solution that enables care providers to spend less time with technology and more time with patients.
During a shift, healthcare workers use several cloud and on-premises applications including business and enterprise applications, electronic health records, medical imaging, patient management, and other systems. Each of these apps in this hybrid environment often requires a unique username and password. Imprivata OneSign eliminates the need for clinicians to memorize and manually enter their credentials. They can sign in once to access all their on-premises and cloud apps, including Microsoft Teams, Office 365, and 3,000+ Microsoft Azure Active Directory (Azure AD) Marketplace applications. No Click Access™ lets them sign in with a badge or fingerprint making it faster to access applications and workflows.
Protecting healthcare staff and patients
As healthcare delivery organizations treat patients under evaluation for COVID-19, they must also safeguard the health of clinicians. Yale New Haven Health is using Imprivata OneSign reporting capabilities to identify exactly where and when specific users accessed specific workstations in different patient care zones in the clinical environment. By combining these data with workstation mapping and electronic health record data, Yale can more accurately identify all providers potentially exposed to COVID-19 and take necessary actions.
Scaling up remote work and virtual care
To limit the spread of COVID-19, administrative roles at clinics and hospitals have migrated to remote work when possible. Care providers have rapidly scaled up virtual care services to provide non-emergency healthcare consultations. These providers need to access systems on personal laptops, mobile devices, and temporary devices in temporary care sites. It’s important that devices and individuals are authenticated to protect sensitive data and apps.
Imprivata Confirm ID for Remote Access improves security by enabling multi-factor authentication for remote network access, cloud applications, Windows servers and desktops, and other critical systems and workflows. Imprivata Confirm ID for EPCS (electronic prescribing of controlled substances) supports Drug Enforcement Agency (DEA)-compliant two-factor authentication methods so providers can quickly prescribe drugs using EPCS workflows. To support healthcare organizations during this crisis we are offering Imprivata Confirm ID licenses for free.
Simplifying role-based access identity management
As the number of patients increases, hospitals are rapidly re-assigning workers within the organization, while on-boarding clinicians from lower utilized hospitals. Healthcare organizations need easy and secure ways to manage user roles as they scale up and provision temporary workers.
Imprivata Identity Governance is an end-to-end solution with granular, role-based access controls and automated provisioning and de-provisioning. Streamlined auditing processes and analytics enable faster threat evaluation and remediation. These capabilities allow IT to respond to the needs of the organization without sacrificing security. Imprivata Identity Governance ensures that, on day one, the right users have the right access to the right on-premises and cloud applications, and the audit trail to prove it.
Making healthcare technology available to everyone
The following resources can help hospitals and clinics move quickly to support patient care beyond the four walls of the hospitals:
Learn more
Solutions like the Imprivata Identity and Access Management platform, Microsoft Azure AD, and Microsoft Teams are helping keep healthcare workers productive and safe as they confront the current crises. As healthcare evolves, Microsoft and Imprivata will continue to innovate together to further enhance scenarios for in-person and remote access.
