Do It For Me vs. Do It With Me: Why Users Crave Control in the Age of AI?
A study comparing fully automated and semi-automated software copilots and providing insightful results to guide the design of Human-AI Interaction (HAI).
Imagine having a super-smart assistant built right into your favorite software, ready to tackle complex tasks with a simple prompt. That’s the promise of software copilots, powered by the latest advancements in Large Language Models (LLMs). But as these AI helpers become more capable of full automation, a crucial question arises: Do users prefer a copilot that does everything for them, or one that guides them through the process? In our CHI 2025 paper, “Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software”
Meet the Contestants: AutoCopilot vs. GuidedCopilot
We developed two in-application copilots embodying different automation philosophies:
-
AutoCopilot (“Do It For Me”): This copilot aimed for full automation of software tasks based on user prompts, much like many state-of-the-art copilots. Think of it as saying “Hey, create a webpage with a login and product section,” and it attempts to build it for you. It also provided concise follow-up responses based on software documentation and web data. However, they are designed to maximize efficiency and task completion, often with limited or no human interaction.
-
GuidedCopilot (“Do It With Me”): This copilot took a semi-automated approach, focusing on automating only repetitive or trivial steps while offering step-by-step visual guidance to help users navigate the software interface. It integrated visuals directly into its responses, allowed users to initiate automation, and enabled them to review and edit results provided by LLMs before proceeding. For example, for a task like sorting data, it might automate the sorting action but guide you on how to select the data range and customize the sort options.
The User Verdict: Control and Learning Matter
By testing with participants (N=20) across data analysis in Google Sheets and visual design in Figma, we got quite revealing results:
-
GuidedCopilot consistently outperformed AutoCopilot in user control, software utility, and learnability, especially for exploratory and creative tasks. Users felt they had more say in how tasks were performed and understood the software better when using GuidedCopilot.
-
While AutoCopilot sometimes saved time for simpler visual tasks, its full automation often led to frustration when the AI didn’t get it quite right, resulting in time-consuming rounds of trial-and-error and debugging. As one participant noted, “It felt like LLM 1 [AutoCopilot] was doing it for me, which is a bit of a gamble…[as] I was little unsure of the result, whereas LLM2 [GuidedCopilot] gave option of doing it with me, or here are the steps to do it myself (P04)12 .”
-
Task completion rates and accuracy were significantly higher with GuidedCopilot in both applications (Google Sheets and Figma).
How to Design Better AI Guidance in Software Copilots?
Building on these insights, we explored ways to further enhance the semi-automated approach with two key features for GuidedCopilot:
-
GuidedCopilotVisual: Embedded context-specific preview clips that provides short video to guide users finish the current step.
-
GuidedCopilotADP: Adaptive step-by-step instructions that adjust based on user progress.
A follow-up usability study with expert and novice Photoshop users (N=10) showed that these features were highly appreciated for their ability to provide targeted and efficient guidance.
Key Factors for Designing Effective Copilots:
-
Familiarity with the application: For new software, a guided, semi-automatic approach is more beneficial for onboarding. As users become more experienced, adaptive semi-automation can balance time savings with user control26 .
-
Task type: Full automation might be suitable for simple, repetitive tasks. However, for complex, creative, or decision-making tasks, semi-automation with visual guidance is preferred to maintain user control and understanding.
-
User intent (learning vs. performing): Users who want to learn the software while working prefer a semi-automatic approach. Those focused purely on efficiency, like some expert users, might lean towards higher levels of automation.
The Future is Collaborative
Our research provides compelling evidence that while the allure of full automation is strong, users often value control and the opportunity to learn when interacting with software copilots. The “Do It With Me” paradigm, which blends semi-automation with tailored guidance, appears to strike a better balance, leading to increased user satisfaction, productivity, and software learnability.
As AI continues to be integrated into our software, understanding these user preferences is crucial. The future of effective software assistance likely lies in creating copilots that function as true collaborators, empowering users rather than replacing them, and adapting to their individual needs and goals.