Deep graph learning has been a trending research area in recent years. In this article, I am going to recap a recent research talk on Microsoft Research Summit 2021, delivered by my colleague researcher “Arun Iyer” from Microsoft Research India, on the challenges we faced building a multi-tenant graph learning solution for recommendation problems.
Event: Microsoft Research Summit 2021
Video: Research talk: Challenges in multi-tenant graph representation learning for recommendation problems
Table of Content
What is “DEEGO” graph learning?
DEEGO is short for “Deep Entity Embeddings from Graphs in Office”. The DEEGO embeddings are built from the rich interactions among entities(email, meetings, contacts, messages, documents, etc.) in M365. For example:
- User 1 -> Email E1 -> User 2
- User 1 -> Document D1 -> User 2
- User 1 -> Teams messages M1 -> User 2
Learning from these rich interactions generated by day-to-day operations in M365 tenants, opens a whole new set of opportunity to create better recommendation system to our customers on in multiple products. However, it also comes with a set of new challenges.
How DEEGO graph learning improves recommendation performance?
There are two potential mode to build/consume DEEGO:
- Mode 1: Unsupervised/self-supervised embedding learning shared with downstream tasks
- Mode 2: Supervised/semi-supervised embedding learning “modules” shared with downstream tasks
It is expected that the mode #1 would be the major usage pattern, with a few scenarios to choose mode #2 for maximum flexibility. The main reason for scenarios to go for mode #1 is to avoid the infra overheads to extract, train, and publish the embeddings.
Our experiments shows mode #1 settings gives a significant performance boost in multiple recommendation tasks over current production baselines.
The challenges of building DEEGO graph learning embeddings
There are challenges we faced when building the DEEGO embeddings, to name a few:
- Embeddings for fast moving entities vs. slow moving entities
- Robustness against label-graph correlations
- Privacy control of training
- Consistency of embeddings for downstream tasks
- Tenant overlap on downstream tasks
- Scalability
- …
I will cover the top five of the challenges listed above in this article. For large scale training, privacy control, and our solution to tenant isolation, stay tune for my next couple of articles.
Embeddings for fast moving entities vs. slow moving entities
Entities are not created equal. In M365, things are moving in a different pace. For example, People, Team, and Channel objects are typically moving at a slow pace which only requires monthly refreshes. Whereas Email, Meeting and Chat objects are moving in a near-real-time fashion, which requires close or equivalent refresh frequency to offer best value to the scenario.
The diversified entities also posts challenges in our infrastructure to process each entities on different schedules.
Robustness against label-graph correlations
Some area of the label-graph approach in DEEGO is still under study. For node classification problem, we are still trying to understand the performance impact of using label-graph in GNNs. To get a better view of this problem, we use homophily metrics on node, edge and class to evaluate the performance impact. But we do notice the concatenation of TSVD of nodes features and graph gives significant improvement over SoTA models.
For Recommendation problem, we are yet to define the metrics or find a model to mitigate issues with Label-Graph correlation. We leave it as an open question to address in the future studies.
Privacy control of training
In Microsoft, user privacy is our top priority. To train DEEGO embeddings, we need to be crystal clear on the scope, category, and precision of data used in training. All data movement needs to be trace-able, approved and managed (in zero-trust environment). On top of the programmatic controls, we also implemented routine training and clearances processes to keep everyone in check.
An other privacy challenge is to provide end-to-end isolation through out the extraction, training, and inferencing process. I will talk about our solution to compliant, secure and isolate training of user/tenant data in a separate article. Stay tune.
Consistency of embeddings for downstream tasks
We currently training DEEGO embeddings in certain cadence. The feedback loop between embedding training and online A/B testing could take some time. For time-sensitive scenarios, making sure the version of embeddings does not regress over time is challenging.
Tenant overlap on downstream tasks
There are hundreds of millions of user span across hundreds of thousands of tenants in M365. It is not uncommon to see users interact with users in a different tenant/organization. How can we provide a way to “join” these entities and interactions in graph construction without compromising privacy and security, and yet, preserve as much information as possible in the DEEGO for training.
Subscribe to learn about our solution to DEEGO graph learning
In my next couple of articles, I will demonstrate the solution my team build that powers DEEGO embeddings training in production for a few hundreds millions of users. Follow me underneath on Medium and Subscribe to my blog to get notified when it comes out!