Most writing about AI interface design assumes a desktop browser, a large screen and a reliable connection. A phone breaks all three assumptions, and it adds four things a browser does not have: a camera pointed at the world, a microphone in a pocket, a battery that people watch, and a permissions system that asks the user to consent before any of it works. Those differences change the design problem enough that patterns which work well on a laptop fail in the hand. This is what actually changes, and what to do about it.
Why the phone is a different problem
The temptation is to treat a mobile AI product as a smaller version of the web one. That assumption produces a chat box with a keyboard covering two thirds of the screen, and a product that is worse than the website it was ported from.
Sensors change what assistance means
On a laptop, an assistant can act on what you type and what is already in your account. On a phone it can act on what you are looking at, where you are standing, what you just said out loud and what happened five minutes ago in another app. That turns assistance from retrieval into observation, and observation is a far more valuable product.
The design consequence is that the primary input is often not text. A user pointing a camera at a serial number, a meter, a rash, a plant, a receipt or a damaged part is giving the system better information than they could type, faster than they could type it, and in a context where typing is genuinely awkward.
Latency is felt differently in the hand
People tolerate a spinning cursor on a desktop while they look at another window. On a phone, they are holding the device, looking at it, waiting. The same two seconds feels considerably longer, and the same wait repeated twenty times a day becomes the defining texture of the product. Everything known about how response time is actually perceived applies here with the volume turned up.
The screen cannot hold the desktop pattern
A web AI product can put a conversation on the left, a document in the middle and sources on the right. A phone has one column. Every decision about what to show becomes a decision about what to hide, and the reasoning that must accompany a model output competes directly with the output itself for space.
On device, cloud, or both
This is an engineering decision with large design consequences, which is why designers need to be in the conversation rather than receiving its result.
What local processing buys you
Speed for short interactions, because there is no round trip. Operation with no signal. And a privacy story that is qualitatively different: data that never leaves the device does not need a consent conversation about where it is stored, which shortens onboarding and opens categories that otherwise cannot ship.
The cost is capability. Smaller local models are less able, so the product must either accept lower quality for local tasks or route the hard ones elsewhere.
The hybrid pattern most products need
In practice the sensible architecture is a split: quick, private, high frequency operations run locally, and heavy reasoning goes to a server. Classification, transcription, redaction, simple extraction and ranking are often good local candidates. Long form generation, complex reasoning and anything needing large context usually is not.
The design work is making that split invisible when it should be and visible when it matters. A user does not need to know which model ran. They do need to know when their data left the device, because that is a promise rather than an implementation detail.
Designing for the moment the signal is gone
Every AI mobile product needs a defined behaviour for no connectivity, and “spinner forever” is not one. Decide per feature: does it degrade to a local model, queue for later, or refuse clearly? Then design the state that says so, because the states nobody demos are the ones users meet.
Queuing deserves particular care. If a request will run later, the user needs to know it is queued, be able to cancel it, and be told when it completes without being interrupted at a bad moment.
Chat is the wrong default on a phone
The chat box is the most copied pattern in AI product design and usually the weakest choice on a small screen.
Why teams reach for it anyway
It is easy to build, it demonstrates capability, and it puts the burden of figuring out what the product can do onto the user. That last property is what makes it feel flexible in a demo and hopeless in daily use: a blank input with a blinking cursor tells a new user nothing about what is possible, and on a phone it immediately costs half the screen to a keyboard.
The full argument against defaulting to conversation is in the patterns that beat a chat box. What follows is the mobile specific version.
Attach intelligence to objects, not to a text field
The strongest mobile pattern is putting capability on the thing the user is already looking at. A photo has an “identify this” action. A document has “summarise”. A transaction has “categorise”. A job has “generate the report”. The user does not have to describe the object, because they are already holding it, and the system does not have to guess what they meant.
Offer structured input before free text
A short set of buttons produces better results than an empty field, because it constrains the request to things the system does well. Three chips with concrete actions beat a prompt box for almost every mobile use, and they double as documentation of what the feature can do.
Reserve conversation for genuine refinement
Where chat earns its place is the second turn: the user got a result and wants it changed. That is a real conversation with a real referent, and it is worth the keyboard. Opening with it is what fails.
The camera as the primary input
If your product has any visual component, the camera is likely the highest value input on the device, and it carries its own design vocabulary.
Capture is a designed moment
A raw camera view with a shutter button produces bad photographs and therefore bad results. Guidance in the viewfinder does most of the work: a frame showing what to include, a prompt to move closer, a warning about glare or motion, and an automatic capture when the frame is good enough. Every one of those reduces failed attempts, and a failed attempt in an AI product reads as the product being stupid rather than the photo being poor.
Confirm before committing
Show what was captured and what was understood before acting on it. A scanned invoice should display the extracted total next to the image, not replace the image with a total. This gives the user a cheap correction point and builds the sense that the system is showing its work.
Design for the retake
Assume roughly a third of first captures will be unusable in real conditions, and make the retake path one tap with the guidance improved based on what went wrong. “Too blurry, hold steady” is worth far more than a generic error, and it is a microcopy decision as much as a technical one.
Voice, and why it is harder than it looks
Voice is genuinely valuable on mobile, particularly hands free contexts like driving, cooking, clinical work or trades. It is also where products most often overreach.
The contexts where voice wins
Hands occupied, eyes occupied, gloves on, or input longer than anyone wants to thumb type. A technician describing a fault while looking at it produces better notes than the same person typing at the end of the day, and that is a real productivity gain rather than a novelty.
The problems to design around
Voice input is unreliable in noise, socially awkward in public, and unforgiving of accents and domain vocabulary. It also produces long, unstructured text that then needs turning into something useful.
The pattern that works is voice as capture, not voice as command: let the user speak freely, transcribe visibly, then structure the result and show it for correction. Never let a spoken instruction trigger an irreversible action without a visible confirmation step, because transcription errors on a noisy site are not rare.
Always provide the typed path
Voice must be an accelerator, never the only route. Some users cannot speak, some will not in public, and some environments make it useless. That is a baseline accessibility requirement rather than a preference.
Designing the wait
Model latency is the defining interaction problem of AI products, and mobile makes it acute.
Acknowledge instantly, always
Something must respond within about a tenth of a second of the tap, even if the real work takes four seconds. A button that visibly reacts, then shows progress, feels dramatically faster than one that sits inert and then succeeds.
Stream whatever can be streamed
Watching text appear is far more tolerable than watching a spinner, because it converts an indeterminate wait into visible progress. Where output cannot stream, narrate the stages truthfully: reading the document, extracting the figures, checking the totals. Truthfully is the operative word, because invented progress messages are noticed.
Let people leave
On a phone, a four second wait competes with a notification, a message and someone talking. Long running work should continue in the background and notify on completion, and the user should be able to lock the phone without losing the result. Anything that demands the screen stay open for thirty seconds will be abandoned.
Designing for the model being wrong
Every AI product ships with an error rate. The question is whether the interface treats that as an embarrassment to hide or a condition to design for.
Make correction cheaper than starting again
If the user must delete the output and re describe the request, they will stop using the feature after the second failure. If they can tap the wrong field and fix it, or swap one chip, or say “no, the other one”, they stay. Correction cost is the single strongest predictor of whether an AI feature survives its first month.
Show the basis, not a confidence score
Percentages mean little to most users and invite false precision. Showing the source is better: the region of the photo that was read, the sentence a summary came from, the transaction that triggered a categorisation. That is the practical form of the trust patterns that hold up in shipped work.
Fail to a usable state, never a dead end
When the system cannot do the task, the screen it lands on should still let the user proceed manually. An AI feature that blocks the underlying workflow when it fails has made the product worse than it was before the feature existed.
Permissions, and the trust ladder
Mobile is the only platform where the user is asked, explicitly, to grant access to the camera, microphone, location, photos and notifications. Each prompt is a moment where the product can lose them.
Never ask cold
A permission prompt on first launch, before the user has seen anything of value, gets declined at a high rate, and on most platforms a denied permission is expensive to recover. Ask at the moment of use, immediately after the user has expressed intent, and explain in your own interface what the permission enables before the system dialog appears.
Ask for the minimum, and say what leaves the device
Request the narrowest scope that works, and be specific about processing. “Photos are analysed on your device and are not uploaded” is a sentence that changes conversion rates, and it must be true. Products that overreach on permissions get uninstalled, and privacy labels on the stores make the overreach public.
Design the denied state
Someone will say no. The product needs a version of itself that works without the camera or without location, plus a clear route back if they change their mind. Treating denial as a broken state rather than a supported one is a common and avoidable failure.
Battery, data and the costs users can see
Phones surface resource consumption in a way desktops do not, and users act on it.
Continuous inference, constant location polling and frequent large uploads all show up in battery settings with your app’s name attached, and that screen is where uninstall decisions get made. The design responses are practical: batch work rather than running it continuously, do heavy processing while charging where the task allows, respect low power mode by degrading gracefully, and be careful with large uploads on metered connections. Users on limited data plans notice a product that quietly uploads every photo, and the same instincts that drive disciplined image handling on the web apply here.
Onboarding without a tour
AI products have a discovery problem: the capability is invisible, so users do not know what to try. The instinct is a carousel explaining the features, which almost nobody reads.
The better pattern is to make the first action succeed. Pre load an example the user can act on, or point the camera at something and demonstrate the result immediately, so the value is experienced rather than described within the first thirty seconds. After that, teach one capability at a time at the moment it becomes relevant, which is the same discipline that makes the first ten minutes of any product decide retention.
Set expectations honestly in that first run, including what the product cannot do. A user who knows the boundary is far less disappointed than one who discovers it by hitting it.
Do not skip the boring instrumentation either. Knowing which capture attempts failed, and why, is the single most useful dataset an AI mobile product can collect about itself, and it costs almost nothing to record at the point of failure.
Measuring whether the intelligence actually helps
Usage of an AI feature is not evidence that it works. People try things once.
The numbers worth watching are repeat use of the feature by the same person, correction rate and whether it is falling, task completion time compared with the manual path, and abandonment during the wait. If corrections are frequent and never improving, the model or the prompt is wrong. If the feature is used once and never again, it was a novelty. If task time is no better than doing it by hand, the feature is decoration with a running cost attached.
Instrument this before launch rather than after, because a launch week with no data cannot be recovered.
What to build first
If you are scoping an AI mobile product, resist building the general assistant. Pick the single highest frequency task where the phone’s sensors give you an advantage a browser cannot have, build that one path to a high standard including its failures, and ship it.
Narrow products with a clear job are easier to make reliable, easier to explain, and far easier to evaluate honestly. Breadth can follow once one thing works, which is the same argument that governs any MVP.
Sequence the work so the riskiest assumption is tested first. In most mobile AI products that is not the model, it is whether real users in real conditions can supply an input good enough to act on. Build the capture path and the correction path before the intelligence is finished, put it in front of five people in the actual environment, and you will learn more in an afternoon than a month of refining prompts against clean test data.
Before any of it, be willing to conclude the model is not needed. Plenty of briefs describe an outcome achievable with a form, a rule and a lookup, at a fraction of the cost and with none of the error rate. Saying so is the most useful thing a design partner can do at that stage.
Proactive assistance and the notification problem
The most valuable thing an AI product on a phone can do is act before being asked. It is also the fastest way to get uninstalled.
The asymmetry that governs everything
An unprompted notification that is right feels like magic. An unprompted notification that is wrong feels like spam, and the cost is not symmetric: one bad interruption undoes several good ones, because it teaches the user that your alerts are not worth opening. Design proactivity with that ratio in mind and the bar for sending becomes much higher than teams expect.
Rules for anything unprompted
Send only when the user can act on it now. Include enough in the notification itself that they can decide without opening the app, since a notification that says “we found something” is a demand for attention rather than a delivery of value. Respect time of day and time zone, and never send anything sensitive to a lock screen where someone else can read it, which matters enormously in health and finance contexts.
Give per category control rather than a single on and off switch, because the user who turns off everything to stop one annoying alert is lost for good.
Start reactive, earn the right to be proactive
Ship the feature as something the user invokes, watch the correction rate, and only move to unprompted delivery once accuracy is high enough that being wrong is rare. Proactivity is a capability you earn with a track record, not a launch feature.
Combining inputs, which is where mobile pulls ahead
The genuinely differentiated products use more than one signal at once, because that combination is what a browser cannot replicate.
A field technician points a camera at a serial number while saying what is wrong with it, and the location confirms which site they are on. That single interaction carries an identifier, a description and a context, and none of it was typed. A shopper photographs a shelf label and the location supplies the store. A patient photographs a medication box and the time of day supplies the dose context.
The design work is deciding which signal is authoritative when they disagree, and showing the user what was assumed. If location says one site and the serial number says another, the interface must surface the conflict rather than silently picking one. Silent resolution of contradictory inputs is how AI products lose trust in a way that is very hard to recover, and the honest pattern is the same one that governs showing the basis for any machine decision.
Keep each input independently usable too. The combined path is the fast path, not the only path, because in practice one of the three signals will be unavailable more often than anyone plans for.
The running cost, and why design controls it
Unlike most design decisions, interface choices in an AI product change the monthly bill.
Where the money goes
Every model call costs something, and mobile products generate more calls than web ones because sessions are shorter and more frequent. A design that fires a request on every keystroke, or re processes an image each time a screen is revisited, multiplies cost without improving the experience.
Design levers that reduce it
Debounce input so a request fires when the user pauses rather than continuously. Cache results against the object rather than the session, so revisiting a scanned invoice does not re run extraction. Route the easy cases to a cheap local path and reserve the expensive model for the genuinely hard ones. Batch background work. And let the user trigger the expensive operation explicitly when the task is expensive, because an explicit tap is also a signal that they want the result.
Why this belongs in the design conversation
If the unit economics do not work, the feature gets removed or rate limited later, and a rate limited AI feature is a worse experience than none at all. Designers who understand the cost shape make different and better decisions early, in the same way that understanding what a chart is for changes how it gets drawn.
Testing an AI product, which is not like testing an app
Conventional QA asks whether a button works. An AI product needs a different question: whether the output is good enough, often enough, on the inputs your users actually have.
Build an evaluation set before you build the feature
Collect fifty to a hundred real examples from the environment the product will run in. Real photos from real warehouses, real recordings from real sites, real documents with coffee stains and skewed angles. Run every change against that set and record how many outputs are acceptable. Without this, every prompt change is a guess and every regression is discovered by a customer.
Test in the conditions, not at the desk
An AI camera feature validated on a clean desk in good light tells you almost nothing. Take the build outside, into a basement, into rain, into a vehicle. Hand it to someone wearing gloves. The failure rate you find there is the real one, and it is usually several times the one measured indoors.
Watch people correct it
Sit with five users and observe every correction. Where they correct tells you what the model is weak at. How they correct tells you whether your correction affordance is good enough. Both are findings you cannot get from analytics, and a week of that is affordable on any budget.
Store review, disclosure and the rules you inherit
Mobile adds a gatekeeper that the web does not have, and AI features attract scrutiny.
App stores expect apps generating content to have moderation and reporting mechanisms, expect accurate privacy disclosures covering what data is collected and where it goes, and expect age ratings that reflect what the model can produce. A product that sends photos to a server while its privacy label says otherwise is not just a trust problem, it is a rejection or removal risk.
There are practical design consequences. You need a reporting affordance on generated content. You need the privacy explanation written in language a person understands, not only in the store label. In regulated categories you may need explicit disclosure that a person is interacting with an automated system, and in health or finance contexts you may need clear statements about what the output is not, such as advice or diagnosis.
Build those in from the start. Retrofitting disclosure into a finished product usually means reopening onboarding, which is the most expensive screen in the app to change.
The failures we see repeatedly
A chat box as the entire interface, shipped because it was fast to build. Permissions requested on launch and declined. No offline behaviour, so the product is dead on a train. Correction that costs more than redoing the task manually. Confidence percentages presented as if they mean something. A feature that works beautifully in a demo with a clean photo in good light and fails on a real photo in a warehouse. And, most commonly, an AI capability bolted onto a product whose underlying workflow was already confusing, where the intelligence amplifies the confusion rather than removing it.
Each of these is a design decision rather than a modelling problem, which is the encouraging part: they are all fixable before a line of production code is written.
If you are scoping a mobile product with intelligence in it and want a straight read on which parts genuinely need a model, which need a sensor, and which need neither, describe the task in a paragraph: hello@beconfidency.agency.
When the hard part is making the intelligence usable rather than making it work, that is exactly what our AI design service is for.