Microsoft is significantly enhancing the capabilities of its Copilot Studio platform by introducing a new feature currently available as a research preview. This innovative tool, named “Computer use,” represents a notable step forward in AI agent functionality. It leverages the power of large language models (LLMs) to enable copilots built within the Studio environment to directly interact with virtually any application or website, moving beyond predefined APIs or connectors. The core function of the “Computer use” tool is to empower AI agents to perform tasks by mimicking human interactions on a computer interface. This means the copilot can observe user actions, such as mouse clicks, keyboard inputs, and navigation across different software, and then replicate these actions autonomously. This capability allows for automation of workflows that span multiple applications, even legacy systems or websites that lack modern integration options. The system essentially learns by watching and then performs the task when instructed. To facilitate this interaction, the feature integrates with Microsoft's Power Automate Desktop. Users demonstrate a task once, and the LLM within Copilot Studio analyzes these actions – like opening an application, navigating menus, filling forms, or copying data between windows. Once learned, the copilot can execute this sequence of actions independently. This opens up possibilities for automating complex, multi-step processes that previously required manual intervention or intricate custom development. For instance, a copilot could potentially extract information from an email, input it into a legacy desktop application, and then update a record on a specific website, all without direct human oversight for each step. This development significantly broadens the scope of what can be achieved with custom copilots built using Copilot Studio. Businesses can potentially automate a wider range of tasks, improving efficiency and reducing manual workload. Examples could include:Automated data entry across different platforms.Streamlining customer service requests involving multiple systems.Generating reports by gathering data from various applications.While currently in a research preview phase, the introduction of the “Computer use” tool signals Microsoft's direction towards creating more versatile and capable AI agents. It bridges the gap between conversational AI and practical desktop automation, allowing copilots to not just converse or retrieve information, but to actively *do* things across the user's digital environment. This advancement holds considerable potential for transforming workflows and enhancing productivity by enabling seamless interaction between AI and the diverse landscape of applications users interact with daily.
