Skip to main content

AI Automation & Vision

Active automation endpoints for natural language execution, vision service status, and single-step execution.

Service Information​

GET /api/automation/status​

Returns the current automation system status and any relevant warnings.

Response:

{
  "success": true,
  "status": {
    "automationReady": true,
    "activeSessions": 1
  },
  "message": "Automation service is healthy"
}

GET /api/automation/validate​

Validates that required automation services are available.

Response:

{
  "success": true,
  "validation": {
    "automation": "ready",
    "vision": "ready",
    "nlp": "ready"
  },
  "message": "Validation completed"
}

GET /api/automation/nlp/info​

Gets information about the NLP service.

Response:

{
  "status": "ready",
  "model": "gpt-3.5-turbo",
  "endpoint": "http://localhost:1234",
  "capabilities": [
    "command_parsing",
    "text_generation",
    "intent_recognition"
  ]
}

GET /api/automation/vision/info​

Gets information about the vision service.

Response:

{
  "status": "ready",
  "model": "moondream",
  "endpoint": "http://localhost:20200",
  "capabilities": [
    "object_detection",
    "text_recognition",
    "ui_element_detection"
  ]
}

GET /api/automation/devices​

Lists devices enabled for automation.

Response:

[
  {
    "id": "device-123",
    "serialNumber": "emulator-5554",
    "model": "Pixel 6",
    "capabilities": {
      "nlp": true,
      "vision": true,
      "automation": true
    },
    "lastSeen": "2024-01-01T00:00:00.000Z"
  }
]

Command Execution​

POST /api/automation/execute​

Executes a natural language command on a specific device.

Request Body:

{
  "deviceId": "device-123",
  "command": "open Chrome and search for FYI",
  "options": {
    "waitForCompletion": true,
    "timeout": 30000
  }
}

Response:

{
  "success": true,
  "result": {
    "action": "open",
    "target": "Chrome"
  },
  "executionTime": 1250
}

POST /api/automation/execute-with-device​

Executes a command with explicit device selection in the payload.

Request Body:

{
  "command": "toggle wifi",
  "options": {
    "deviceId": "emulator-5554",
    "timeout": 15000
  }
}

Response:

{
  "success": true,
  "result": {
    "action": "toggle_wifi",
    "deviceId": "emulator-5554"
  },
  "message": "Command executed"
}

POST /api/automation/execute-parsed-with-device​

Executes a pre-parsed command on a device.

Request Body:

{
  "parsedCommand": {
    "action": "tap",
    "targetElement": "login button",
    "parameters": {
      "confidence": 0.85
    }
  },
  "options": {
    "deviceId": "device-123"
  }
}

Response:

{
  "success": true,
  "result": {
    "action": "tap",
    "targetElement": "login button"
  },
  "message": "Parsed command executed"
}

Flow Step Execution​

POST /api/automation/execute-step​

Executes a single structured flow step with optional coordinate cache usage.

Request Body:

{
  "deviceId": "device-123",
  "step": {
    "action": "tap",
    "targetElement": "submit button"
  },
  "flowId": "flow-abc",
  "useCache": true
}

Response:

{
  "success": true,
  "usedCache": true,
  "executionTime": 820
}