Skip to main content

FYI Automation Tool - Getting Started

Welcome to the FYI Automation Tool documentation! This AI-powered Android automation platform provides natural language commands, Vision Model UI detection, flow-based testing, and Plane.so integration for project management.

🚀 Quick Start Overview​

Get up and running with the FYI Automation Tool in just a few steps. This platform allows you to automate Android device testing through natural language commands and computer vision.

📋 Prerequisites​

System Requirements​

  • Operating System: macOS (10.14+), Windows (10+), or Linux (Ubuntu 20.04+)
  • RAM: 8GB minimum, 16GB recommended (for AI models)
  • Storage: 5GB free space
  • USB: Available USB ports for device connections
  • Object Storage: S3-compatible bucket (AWS S3 or MinIO) for recordings and audio files

What you'll need​

Core Dependencies​

  • Node.js version 20.0 or higher:
    • When installing Node.js, you are recommended to check all checkboxes related to dependencies
  • npm version 9.0 or higher (usually comes with Node.js)
  • PostgreSQL database

Android Development Tools​

  • ADB (Android Debug Bridge) - part of Android SDK Platform Tools
  • scrcpy 2.0 or higher for screen mirroring
  • Appium CLI 3.x installed globally (npm install -g appium) for device control

Object Storage for Recordings & Audio​

  • AWS S3 or MinIO (or any S3-compatible service)
  • Create two buckets:
    • recordings (for screen recordings)
    • audio (for audio prompts and playback)
  • Required environment variables in server/.env: 
    S3_ACCESS_KEY=YOUR_KEY
    S3_SECRET_KEY=YOUR_SECRET
    S3_ENDPOINT=https://s3.your-cloud.com   # or http://localhost:9000 for MinIO
    S3_RECORDINGS_BUCKET_NAME=recordings
    S3_AUDIO_BUCKET_NAME=audio
    S3_REGION=us-east-1
    S3_USE_PATH_STYLE=true                  # set true for MinIO

Optional AI Services​

  • Python 3.11+ for MoonDream Vision API
  • LM Studio for AI features
  • Ollama for screen verification (required if using Verify Screen steps):
    • Install Ollama from https://ollama.ai
    • Pull a vision-capable model: Recommended: ollama pull qwen3-vl:30b-a3b-instruct-q8_0 (or any qwen3-vl model)
    • Configure in server/.env: 
      OLLAMA_BASE_URL=http://localhost:11434
      OLLAMA_MODEL_ID=qwen3-vl:30b-a3b-instruct-q8_0  # recommended, or your chosen vision model
    • See Supported Steps - Ollama Configuration for details

⚡ Quickstart: Use Hosted MoonDream​

Skip local MoonDream setup by using the hosted endpoint:

# In server/.env
MOONDREAM_URL=https://moondream.fyi.ai:9443/

Installation Commands​

macOS (using Homebrew)​

# Install core dependencies
brew install node postgresql adb scrcpy
npm install -g appium

# Optional: Install Python for vision services
brew install python@3.11

Ubuntu/Debian​

# Update package list
sudo apt update

# Install core dependencies
sudo apt install nodejs npm postgresql android-tools-adb scrcpy
sudo npm install -g appium

# Optional: Install Python
sudo apt install python3.11 python3-pip

Windows​

# Install Node.js from: https://nodejs.org/
# Install Appium CLI: npm install -g appium
# Install ADB: https://developer.android.com/studio/command-line/adb
# Install scrcpy from: https://github.com/Genymobile/scrcpy

🏗️ Project Setup​

1. Clone and Install Dependencies​

# Clone the repository
git clone <your-repo-url>
cd fyi-automation-tool

# Install frontend dependencies
npm install

# Install backend dependencies
cd server && npm install && cd ..

2. Setup Database​

# Start PostgreSQL service
sudo systemctl start postgresql

# Enable PostgreSQL to start on boot
sudo systemctl enable postgresql

# Verify database is running
sudo systemctl status postgresql

3. Configure Environment Variables​

# Copy environment templates
cp sample.env .env
cp server/sample.env server/.env
cp moondream/sample.env moondream/.env

# Configure your environment variables (see README.md for details)
# Important: If using Verify Screen steps, configure Ollama in server/.env:
# OLLAMA_BASE_URL=http://localhost:11434
# OLLAMA_MODEL_ID=qwen3-vl:30b-a3b-instruct-q8_0  # recommended, or any qwen3-vl model

4. Initialize Database​

cd server
npx prisma generate
npx prisma db push
cd ..

📱 Android Device Setup​

Enable Developer Options​

  1. Go to Settings > About phone
  2. Tap Build number 7 times until you see "You are now a developer"

Enable USB Debugging​

  1. Go to Settings > Developer options
  2. Enable USB debugging
  3. Connect your Android device via USB
  4. Authorize the connection when prompted

🚀 Start the Application​

Development Mode​

# Start both backend and frontend
npm run dev:full

# Or start separately:
npm run dev                    # Frontend on :8080
cd server && npm run dev      # Backend on :3001

Production Mode​

# Start full stack with Docker
docker-compose up -d

🌐 Access Points​

🧪 Basic Usage​

Once everything is running, you can start automating your Android device with natural language commands like:

  • "take a screenshot"
  • "tap on the search button"
  • "type hello world"
  • "swipe up slowly"
  • "open Settings app"

📚 What's Next?​

For detailed information about features, API endpoints, and advanced configuration, refer to the main README.md in the project root.