Atlassian had a story it wanted to tell about the future of work. Rovo Agents, the company described, were “configurable AI teammates” that could be called on by any team member to collaborate and move work forward, embedded directly into Jira and Confluence, surfacing organisational knowledge, automating workflow steps and joining teams as active participants rather than passive tools. By early 2026, Rovo had surpassed five million monthly active users; in the preceding six months, agents had helped automate 2.4 million workflow processes across its customer base.

Then, on 11 March 2026, Atlassian told a different story. CEO Mike Cannon-Brookes announced the company would cut approximately 10% of its global workforce (roughly 1600 jobs) to self-fund further investment in AI and enterprise sales. The stock rose. The union representing Australian workers demanded urgent consultation. The same week, Atlassian’s shares had lost more than half their value since January, swept up in what traders were calling the “SaaSpocalypse“, a sustained selloff in enterprise software stocks driven by investor fears that AI agents could make conventional SaaS tools obsolete.

The juxtaposition was uncomfortable and instructive. Here was a company simultaneously positioning AI agents as teammates and restructuring its human workforce around them without, by most accounts, a deliberate architecture for how the two would work together. That gap between the promise of human-AI collaboration and the reality of how organisations can manage the transition is precisely what leaders need to close. This is the AI leadership challenge.

A glimpse into the future of AI agents

The scale of what is coming is not in dispute. Gartner predicts that by 2029, at least 50% of knowledge workers will develop new skills to work with, govern or create AI agents on demand for complex tasks. In the near term, the transformation is already underway: Gartner’s best-case scenario projection predicts that agentic AI could drive approximately 30% of enterprise application software revenue by 2035, surpassing $450 billion, up from 2% in 2025. The pace of change leaves little room for organisations to remain passive. As Anushree Verma, Senior Director Analyst at Gartner, observed: “AI agents are evolving rapidly, progressing from basic assistants embedded in enterprise applications today to task-specific agents by 2026 and ultimately multi-agent ecosystems by 2029.”

The implications for how organisations design work are beginning to surface in research. Gensler’s 2026 Global Workplace Survey, drawing on more than 16,400 office workers across 16 countries, found that AI adoption and workplace strategy could no longer be treated as separate concerns. The survey found that 30% of employees now qualified as AI power users, workers who regularly used AI tools in both their professional and personal lives, and that these workers reported spending less time working alone, more time learning, and having stronger team relationships than their peers.

Learn more: How AI is changing work and boosting economic productivity

Gensler posed a practical question that organisations need to address: should AI agents be portable team members who move around the office and join meetings, or should they be project room residents who keep track of files, conduct research, and monitor progress toward team goals? As Gensler framed it: “Is your AI teammate portable? Are they free to move around the office, attend meetings, and participate in conversations? Or do they live in a project room and keep track of project-related notes, files, and research?” The answer is not merely a design preference. It reflects a fundamental strategic choice about whether AI augments human workflows or whether humans, in practice, begin to serve AI agents.

For many organisations, increasing AI adoption signals productivity gains. For Professor Mary-Anne Williams, Michael J Crouch Chair for Innovation and Professor in the School of Management and Governance at UNSW Business School, it signals something more fundamental: a shift in how work itself is structured. If AI agents are becoming workplace colleagues, leaders face design choices now that will shape culture, accountability and innovation for years to come.

“Human-AI collaboration succeeds with careful design, not by accident,” says Prof. Williams, who also serves as Founder and Lead of the UNSW Business AI Lab and Deputy Director of the UNSW AI Institute. “It needs a deliberate collaboration architecture and orchestration strategies with clear roles, escalation paths, decision rights, and disciplined handoffs between humans and agents.”

In practice, this means moving past the assumption that an AI agent will figure things out. “In day-to-day practice, deliberate design means we stop pretending the agent is ‘magic’, and instead treat it like a teammate with a tightly defined role, clear goals, inputs and outputs, and explicit action quality evaluation gates,” Prof. Williams says.

The first design decision: what kind of colleague?

Over the next 12 to 24 months, Prof. Williams says leaders will need to make a small number of foundational decisions to avoid locking in a counterproductive way of working. The most important design decision is to clarify what kind of colleague AI agents are allowed to be.

Most organisations are currently blending three different modes of AI use without distinguishing between them: drafting, where the agent produces a first version; advising, where the agent recommends an option; and executing, where the agent takes direct actions. Treating these three modes as interchangeable is where the governance risk lives. “The first design choice is to separate these modes and be explicit about which work can move into ‘execution’, and which must remain ‘advice or draft’,” Prof. Williams explains. “That one choice prevents a lot of unsafe or low-quality automation-by-default.”

As AI agents move from summarising and drafting into workflow automation, accountability becomes the central design challenge. “If an agent is a ‘colleague’, people will naturally start treating it like an authority,” Prof. Williams says. “So, leaders need to define who is accountable for outcomes, and what the required human sign-offs are for different risk levels.” Clear accountability prevents responsibility from shifting quietly to “the system.”

Learn more: Why AI systems fail without human-machine collaboration

McKinsey’s 2025 State of AI survey confirmed the scale of this governance gap, finding that 88% of organisations now use AI in at least one business function, yet 51% reported at least one negative AI incident in the past year, including inaccuracy, compliance failures and privacy breaches. Notably, McKinsey found that high performers were actually more likely to experience such incidents precisely because they deployed AI in more complex, higher-stakes domains. However, they distinguished themselves by managing risk through human-in-the-loop rules, rigorous output validation, centralised governance and senior leadership involvement in AI oversight. Separately, McKinsey’s research on the emerging agentic organisation concluded that human accountability and oversight would remain essential, but their nature would change. Rather than conducting line-by-line reviews, compliance officers and leaders would define policies, monitor outliers and adjust the level of human involvement.

Designing deliberate handoffs

Prof. Williams emphasises that effective human-AI collaboration depends on clarity at exactly those points where humans and AI agents interact. “Human-AI collaboration works when the humans and AI have aligned goals, joint attention and deliberately designed hand-offs: agent does X, human does Y, agent does Z,” she says.

In practical terms, this means specifying what inputs the agent can access, what constitutes a good output, what triggers escalation to a human, and what the definition of done is for the human reviewer. In systems like Atlassian’s Rovo, where AI-generated insights flow directly into project management and decision workflows, without structured checkpoints, outputs can move forward simply because they are fast and convenient.

Prof. Williams also recommends treating AI agents as part of a formal digital workforce infrastructure. “Treat agents like a digital workforce: each agent needs an identity, least-privilege access, and auditable logs for quality, security, incident response, and resilience,” she says. Without this operational layer, organisations risk the emergence of what she calls 'shadow agents': unapproved tools, unclear data flows, and ambiguous ownership.

As McKinsey’s agentic organisation research concluded, governance in this environment “cannot remain a periodic, paper-heavy exercise. As agents operate continuously, governance must become real-time, data-driven, and embedded with humans holding final accountability.”

Psychological safety as innovation infrastructure

Beyond structure and governance, Prof. Williams highlights a cultural factor that becomes even more important in AI-enabled environments: psychological safety. “If you want innovation, you need psychological safety: people must be able to challenge an agent’s output, admit uncertainty, and ask ‘what are we missing?’ without penalty,” she says.

AI agents monitor, record and optimise interactions. That can create performance benefits, but it can also create hesitation. “The uncomfortable truth is: if employees feel they’re being judged for ‘slowing things down’, or that every interaction is monitored in a punitive way, they will stop challenging the agent, even when they should,” Prof. Williams says. “Psychological safety is not a ‘nice-to-have’. It’s the control mechanism that keeps human judgement alive.”

Without it, organisations may appear efficient while losing critical thinking capacity. “Without psychological safety, you don’t get experimentation; you create risk. You get silent compliance and no competitive advantage,” she says.

Learn more: AI agents: Why the future belongs to specialists, not generalists

Research published in AI & Society, drawing on a 2025 systematic review of 35 peer-reviewed studies across cognitive psychology, human factors engineering and human-computer interaction, found that automation bias (the tendency to over-rely on automated recommendations) has emerged as a critical challenge in human-AI collaboration, particularly in high-stakes domains such as healthcare, law and public administration. The same body of research documents the opposing dynamic: algorithm aversion, where negative experiences with AI erode user trust and lead people to reject AI recommendations, with experts particularly prone to this given their reliance on established heuristics and the added pressure of high-stakes settings.

“There are two well-documented failure modes: automation bias (over-relying on the system) and algorithm aversion (rejecting it after seeing it err). Both are predictable human responses. Both can be managed with training, norms and innovative workflow design,” Prof. Williams explains. In high-risk contexts, such as money movements, safety, compliance, hiring decisions, and public claims, employees should be empowered and expected to pause automation and escalate.

The warning signs of AI deference

Prof. Williams has identified a set of early indicators that an organisation is developing an unhealthy dependence on AI outputs. The first is when “because the agent said so” becomes an acceptable justification, in which decisions are defended by appealing to the system’s authority rather than evidence and reasoning. A related signal is when meetings grow faster but thinner, with fewer questions raised and fewer dissenting views offered. The productive friction that generates better decisions quietly disappears.

A third warning sign is a drop in verification behaviour: fewer source checks, fewer second opinions, fewer independent tests. Prof. Williams says organisations should pay particular attention when override rates (the frequency with which humans override agent outputs) fall to near zero in complex work. That is rarely a sign the system has become more accurate, and Prof. Williams says this is more often a sign that people have stopped looking carefully.

Over time, outputs themselves begin to converge. When every team member’s writing starts to look the same, or when strategy discussions settle too quickly on one reasonable path, Prof. Williams says the diversity of thinking that drives competitive advantage is eroding. At the far end of the spectrum sits skill atrophy: employees struggle to explain the underlying logic of a recommendation without reference to the tool, and new hires lack the ability to operate without it. Experienced staff stop mentoring because they assume the agent handles it.

The final and perhaps most subtle signal, according to Prof. Williams, is inappropriate trust driven by anthropomorphism. This is when AI agents feel conversational and confident; people can develop a level of trust that runs ahead of the system’s actual competence. Interventions should focus on resetting accountability standards, rebuilding critical-thinking capabilities and reinforcing that AI drafts or recommends, but Prof. Williams says humans should remain responsible for outcomes.

Subscribe to BusinessThink for the latest research, analysis and insights from UNSW Business School

A strategic opportunity

To address some of the challenges of implementing AI effectively, Prof. Williams’ Australian Research Council Discovery Project, Agentic AI for the Strategic Management of Innovation, will run from 2026 to 2029 in collaboration with Professor David Teece (the world’s most cited strategy scholar) of the Haas School of Business at the University of California, Berkeley. The project aims to develop a human-AI dynamic capability framework, along with practical methods and tools, to help organisations drive innovation by strategically managing generative AI agents.

The broader opportunity is not simply productivity improvement. It is designing an organisation’s strategic and competitive advantage through safe human-AI innovation. For organisations, the critical question is no longer whether AI agents will enter the workplace. The question is whether leaders will design how those agents will interact to complete tasks and deliver work, before arbitrary, unwanted patterns become embedded by default.