Have you ever wanted to know how large language models work when you connect them to the data in your organization? At Microsoft, we recently demonstrated Microsoft 365 Copilot, which transforms how we work by leveraging large language models that interact with your organizational data. Copilot works alongside you. For example, in Word, Copilot can easily write an entirely new document, like a business proposal using content from your existing files. Or in Outlook, based on the content you select.
Copilot can compose your email replies for you. In PowerPoint, you can transform your written content into a visually beautiful presentation with the click of a button. In Teams, Copilot can generate meeting summaries with discussed follow-up actions. Or while using Business Chat in Microsoft Teams, it can help you catch up on something you may have missed, bringing together information from multiple sources to bring you up to speed. If you're wondering how large language models know what they know in these scenarios,
Let's break down the mechanics of what makes this possible, and how the process respects your privacy, and keeps your data safe with Microsoft 365 Copilot. First, let's look at where large language models, or LLMs, get their knowledge. LLMs are trained on massive amounts of public data, including books, articles, and websites to learn language, context and meaning. You can interact with large language models using natural language with what's called a prompt. A prompt is typically a statement or question.
When you ask a question in the prompt, the LLM generates a response based on its public data training and understanding of context, which can come in part from how you phrase your prompt. For example, you might give it more details to generate a response. As you continue to ask questions and get responses, the large language model is temporarily getting more context. Your full conversation gets sent with each subsequent prompt, so the LLM can generate relevant responses as you chat with it. It's processing natural language and referring to its knowledge like we would in conversation.
A key difference is that it only remembers the conversation while it's in that conversation. The chat history is wiped clean with each new conversation. And it won't use the knowledge from your conversations and interactions to train the model. That said, you can also write your prompt to include additional information, which the large language model will refer to as it generates its response.
Microsoft 365 Copilot uses its own private instances of the large language models. Next, Microsoft 365 Copilot has a powerful orchestration engine that I'll explain in a moment. Copilot capabilities are surfaced in and work with Microsoft 365 apps. Microsoft Search is used for information retrieval to feed prompts, like I did in the example before where information I provided in my prompt was used to help generate an answer.
Then the Microsoft Graph, which has long been foundational to Microsoft 365, includes additional information about the relationships and activities over your organization's data. The Copilot system respects per user access permissions to any content and Graph information it retrieves.
This is important because Microsoft 365 Copilot will only generate responses based on information That you have explicit permission to access Additionally, Microsoft 365 has its own default prompt with responsible rules for interaction. This includes things like where to search to find the right information for example calling the Microsoft Graph to gain more context, like your recent documents or messages also the style and tone of the response. Like being informative for style and positive for tone.
As well as different approaches Copilot can take to gather information for the prompt for example, it can choose to iterate on a few searches until there's enough information to generate a good response Copilot knows how it should cite its sources. And of course, respects responsible AI practices. Such as ensuring that harmful content is not included in generated responses Importantly, this default prompt gets appended to your prompt When you interact with Copilot. The Microsoft 365 Copilot orchestrator combines the default prompt with your prompt with the additional information its gathered and will form one long prompt To present to the LLM in order to generate a response.
Now let's go back to the example you saw earlier in Microsoft Teams where a user asked, "Did anything happen yesterday with Fabrikam?" Copilot didn't just send that question or prompt directly to the large language model. Instead, Copilot knew that it needed more knowledge and context, so using clues from the user's question, like Fabrikam, it inferred that it needed to search for content sources private to the organization.
The Copilot orchestrator searched the Microsoft Graph for activities, ensuring it respected the user's permissions and access to information, in this case, the user Kat. It found the email thread from Mona that Kat received, activities in the Project Checklist and March planning presentation, which are files that Kat had access to, as well as the sharing action where the final contract was sent to Fabrikam for review, again, where Kat would have been part of the share activity.
And Copilot cited each source of information so Kat could easily validate the response. These are all individual steps that Kat could have done manually, like searching her inbox for emails from Mona looking at recent project file activities in SharePoint or reading the sharing notifications sent to Fabrikam for the contract. Copilot removed the tediousness of performing these steps manually and formulated a natural easy-to-follow and concise response in a single step. So that's how Business Chat with Copilot works.