Reference · Hey Mint, the open-source voice assistant for macOS

Reference

Installing

./set-key.sh          # paste your Gemini key
./install.sh          # builds ~/Applications/Mint.app
open ~/Applications/Mint.app

Permissions

Microphone first. Accessibility from the menu bar icon (clicking, typing, menus, scrolling). Screen Recording, Automation (per app), Calendars, Reminders and Files and Folders are asked for the first time a tool needs them.

Keys

In .env, readable only by you: GEMINI_API_KEY, TYPESAFE_API_KEY (Jev), optionally OPENAI_API_KEY / OPENROUTER_API_KEY (agents) and EXA_API_KEY / TAVILY_API_KEY (web search).

The installed app runs from ~/Library/Application Support/Mint; re-run ./install.sh after changing code.

Reference

What runs Mint

Model	What it does
Gemini Live	Listens, talks, reasons, plans, and calls the tools
Jev	Fast, careful choices from real lists: which button, which skill, which memories, which app you meant, whether you were talking to Mint
Gemini Flash	Background work: summaries, learning facts, writing skills, reading screenshots
GPT-6 Luna, Codex	The agents

Reference

Where everything lives

Where	What
`.env`	Keys and model settings
`settings.json`	Appearance, microphone, voice, shortcuts, stop words
`custom.json`	Accounts, workspaces, aliases, routines, instructions
`agents.json`	Agents
`skills/`	Skills by category; `_archive/` for removed ones
`memory/bank.json`, `memory/MEMORY.md`	Memories
`history.jsonl`, `summary.md`	Conversation log and running summary
`~/Documents/Mint/`	Files Mint makes: PDFs, notes, agent work
`…/Mint/backups/`	Earlier versions of files Mint overwrote
`~/Library/Logs/Mint/mint.log`	The log: every tool call and result

Reference

Fixing common problems

Problem	Fix
“Mint lacks Accessibility permission”	Menu bar ▸ Grant Accessibility…, switch Mint on
“has not allowed Mint to control that app”	System Settings ▸ Privacy & Security ▸ Automation ▸ Mint
Browser JavaScript from Apple Events is off	Reading and clicking still work. For raw JavaScript: Chrome ▸ View ▸ Developer
“macOS did not let Mint read it”	System Settings ▸ Privacy & Security ▸ Files and Folders ▸ Mint
It hears the room	Train your voice, or turn the mic off and type (`⌘J`)
It mishears a name	Tell it once (see Words it gets wrong)
It hangs	`kill -USR1 <pid>` writes every thread's stack to the log

Reference

Testing it yourself

.venv/bin/python -m mint --demo                  # the animation tour
.venv/bin/python -m mint --say "do a trick"      # talk to the running Mint by text
.venv/bin/python bench/harness_bench.py          # files, guards, browser helpers, web
.venv/bin/python guide/make_clips.py             # re-record this guide's clips (keep the screen quiet)
.venv/bin/python guide/encode_clips.py && .venv/bin/python guide/build.py

Reference

Every tool

What Mint can call, with the first line of each description. You never need these names: just say what you want.

All tools

Tool	Arguments	What it does
`open_app`	name	Instantly launch or switch to a Mac application by name.
`open_url`	url, browser	Instantly open a web address in a new browser tab (reuses a tab already showing it).
`open_folder`	name	Instantly open a folder in Finder.
`scroll`	direction, amount	Instantly scroll the window under the pointer.
`press_key`	key, modifiers, times	Instantly press a key, optionally with modifiers.
`set_volume`	level	Instantly set the Mac's output volume.
`media_key`	action	Instantly send a media key: play/pause, next or previous track.
`frontmost_app`		Which application is in front right now.
`list_windows`		List the visible windows as app name and title.
`quit_app`	name	Ask an application to quit.
`type_text`	text, press_return, field	Instantly insert text at the cursor in whatever field is focused, in any app.
`get_selected_text`		Read the text the user has selected in the front app.
`get_status`		Current local time and date, battery level, and volume.
`set_timer`	minutes, label	Start a timer.
`notify`	title, message	Show a macOS notification.
`system_action`	action	Lock the screen, sleep the display, switch dark mode, or mute.
`calendar_events`	days	The user's calendar events from today onwards.
`create_reminder`	title, in_minutes, when	Create a reminder in the Reminders app, optionally due at a time.
`create_note`	title, body	Create a note in the Notes app.
`compose_email`	to, subject, body	Open a pre-filled email draft in the user's mail app for them to review and send.
`list_emails`	count	List the newest messages in the macOS MAIL APP's inbox.
`read_email`	number	Read one message from the macOS Mail app's inbox in full (see list_emails).
`open_chrome`	account, url	Open Google Chrome as one particular account (Chrome profile), optionally at a URL.
`open_slack`	workspace, channel	Open Slack, optionally switch to a workspace, and optionally jump to a channel, DM or person by name.
`list_accounts`		List the Chrome profiles and Slack workspaces available.
`read_window`	max_chars	Read ALL the text in the front window - a web page, a Gmail inbox or open email, a document, an app - exactly, including parts scrolled out of view.
`list_open`		What is open right now: the app in front, and every Chrome window with its profile and tabs (* = the selected tab).
`switch_to`	what	Bring an already-open Chrome tab or window to the front, by words from its title, e.g.
`export_doc_pdf`	open_after	Export the open Google Doc itself as a PDF, exactly as Docs renders it, and open it.
`plan_task`	goal, steps	Start a multi-step task: state the goal and the ordered steps BEFORE doing any of them.
`step_done`	step, result, failed	Mark a step of the current task finished (or failed), with a one-line result.
`create_pdf`	title, content, open_after	Write a nicely formatted PDF and open it.
`run_routine`	name	Run one of the user's saved routines by name.
`desktop`	goal	Drive the screen to do something that the instant tools cannot: click a specific button or link, type into a particular field, choose a menu item, pick a search result.
`set_preference`	setting, value	Change how you (Mint) behave or look, when the user asks: 'don't speak, just chat' -> spoken_replies off; 'talk to me again' / 'speak' -> spoken_replies on; 'turn off the mic' -> microphone off (then they can only type); 'make it purple' -> theme; 'move to the bottom left' -> position; also the orb's face, the on-screen effects, word-by-word captions, and whether you listen while you work.
`chat_action`	action	Manage the chat window's conversation when the user asks: 'clear the chat' -> clear (window only; memory is kept); 'summarise / compact our conversation' -> summarize (a Flash model summarises it, a summary card appears, and you are restarted with just that summary as context - like compacting); 'start a new session / start fresh / new conversation' -> new_session (saves this conversation to memory, then restarts you with a clean context).
`show_chat`	open	Open or close the chat window (conversation history, typing box, settings).
`stop_listening`		Go back to sleep and wait for the wake word.
`look`	display	Capture the screen and look at it.
`ui_act`	action, target, text, press_return, app	THE way to click, type into or pick anything in an app's window.
`ui_elements`		List the controls in the front window right now (numbered, with role, label and where they are; says if a dialog is open).
`click_text`	text, double	Click a piece of text you can name on screen - a sidebar item, tab, button or link label - in apps the desktop tool cannot see into (Slack, some Electron apps).
`click_at`	x, y, button, double	Click a point you can see in the latest look screenshot.
`find_skill`	task, app	Find the saved skill (a learned how-to) for a task.
`skill_result`	name, worked, note	Report how following a skill went, so good skills rise and bad ones get fixed.
`create_skill`	title, when, steps, notes, category, apps	Save a new skill: how to do a kind of task, as steps that worked.
`update_skill`	name, steps, note, when	Change a saved skill: replace its steps, add a note (e.g.
`list_skills`	category	List saved skills by category, ranked by how well they work.
`delete_skill`	name	Remove a saved skill (a copy is archived).
`remember`	fact, group, fixed	Save a fact about the user for future conversations: people, accounts by name, work, schedule, preferences, places, or a word you misheard.
`recall`	question	Look up what you remember that is relevant to a question.
`update_memory`	what, new_fact, fixed, group	Change a saved fact (the user says it changed or was wrong), or make it fixed / not fixed.
`forget`	what	Forget a saved fact the user no longer wants kept.
`list_memories`	group	List what is remembered, optionally one group.
`show_skills_and_memory`	tab	Open the Skills & Memory window on screen, where the user can see and edit every skill and every remembered fact.
`show_on_screen`	text, style, note, scroll, app	Show the user where something is, in whatever app is in front (a PDF, a web page, a chat, a document, code): it finds the words on screen - scrolling the window to them if needed - and draws a box, underline, highlight, circle or arrow around them, with an optional short note; your orb flies over beside it.
`mark_area`	x, y, w, h, style, note	Mark a region with no text (an image, a chart, an icon) using 0-1000 coordinates of your latest look screenshot: x, y of its top-left corner, and its width and height.
`clear_marks`		Remove the marks you drew on screen.
`move_orb`	direction, amount	Move your orb on screen when the user asks: 'go a little down', 'move up', 'move left a lot', 'you are covering that' (direction away), 'go to the middle'.
`screen_share_visibility`	visible	Show or hide Mint in screen sharing, recordings and screenshots.
`express`	emotion, requested	Play an expression on your orb body.
`set_voice`	voice, style	Change your speaking voice and/or style when the user asks ('use a deeper voice', 'sound more cheerful', 'talk slower', 'use Puck').
`list_agents`		List your sub-agents: name, what each is for, its model, and what each is doing now.
`delegate_task`	agent, task, why, thinking, context, folder, attach_window	Hand a substantial task to a sub-agent, which works in the background while you keep talking.
`delegate_tasks`	tasks	Start several sub-agent tasks at once (in parallel), e.g.
`agent_status`	agent	What your sub-agents are doing, or have finished, right now.
`message_agent`	agent, message, thinking	Send new or changed instructions to a running sub-agent - when the user changes what they want mid-task ('tell Luna to use pytest', 'make it shorter').
`answer_agent`	agent, answer	Pass the user's answer to a sub-agent that asked a question.
`stop_agent`	agent	Stop a running sub-agent, or 'all'.
`create_agent`	name, role, instructions, thinking, web, color	Create (or update) a named sub-agent when the user asks for one: its name, what it is for, how it should work, and optionally which model.
`wait_until_done`	app, until_text, timeout_minutes	Wait until an app has FINISHED what it is doing before you use the result: ChatGPT (or any AI chat) still writing its answer, a page loading, something generating.
`preview_site`	path	Show a web page or site that was built on this Mac (e.g.
`screenshot`	what, target, size, copy, save, name, format	Take a screenshot and save it as a file (and copy it to the clipboard).
`clipboard`	action, text, path, index	Manage the clipboard.
`read_file`	path, start_line, max_chars	Read a file on this Mac: text, code, Markdown, CSV, JSON, PDF, Word/RTF/Pages-exported docs.
`write_file`	path, content, mode, find	Create or change a text file (notes, code, Markdown, CSV, HTML...).
`find_files`	query, folder, kind, content, limit	Find files and folders by name or content with Spotlight, newest first.
`file_action`	action, path, to	Do something with a file or folder: open (in its default app), reveal (in Finder), info, move / copy / rename (to to), make_folder, trash (moves it to the Trash - only when the user asked to delete it; it can be restored from there).
`web_search`	query, max_results	Search the web and get titles, links and snippets - for facts, news, prices, docs, anything current.
`read_url`	url, max_chars	Fetch a web page or text URL and get its readable text, without opening it on screen.
`browser`	action, target, text, browser	Full control of the web browser (Chrome, Safari, Brave, Arc, Edge) - faster and surer than clicking pixels.
`scroll_to`	target, where, direction, amount	Scroll so something is visible, in any app: target is text or a control to bring into view (scrolls a long list or page until it appears).
`menu`	path, action, app	Use the app's menu bar - every command an app has is there, with its keyboard shortcut.
`wait_for_text`	text, seconds, gone	Short wait (seconds, at most 2 minutes) for text to appear in the front window - or to go away, with gone=true (a 'Loading…' or 'Exporting…' label, a dialog closing).
`pointer`	action	The user's mouse pointer: where (what is under it - the control, its app, nearby text) or click / double_click / right_click right where it is.
`quit_mint`		Quit Mint itself completely - the assistant (you), everything you started (sub-agents, Codex, preview servers) and the screen-control engine - when the user says 'quit Mint', 'turn yourself off', 'shut down Mint', 'close Mint'.
`run_applescript`	script	Run an AppleScript for apps with a scripting dictionary: Finder, Music, Notes, Reminders, Calendar, Mail drafts, Safari/Chrome tabs, System Events UI scripting.
`fix_hearing`	heard, meant, forget	Remember a word or name you misheard, so it is heard right from now on.

PreviousSettings