For makers · firmware 2.6

ESP32-CAM face tracking with servos, explained.

How OKO-1 runs face detection on a $6 ESP32-CAM, sends a few bytes to a second ESP32 and turns them into calm, accurate pan/tilt moves with two MG90S servos. Real numbers from the open-source firmware.

Read the codeBuild guide
Architecture

Two boards, one wire pair

OKO-1 splits the job across two ESP32 chips. The ESP32-CAM (AI-Thinker, OV2640) sits on the tilting head next to your webcam and only looks for faces. The main ESP32 sits in the housing, drives both servos and talks to your computer over USB for settings. They share one UART at 921 600 baud. Because the tracking camera rides on the head, the loop is closed: every correction changes what the camera sees next.

Detection≈7.2 fps on-chip
Frame320 × 240 (PSRAM)
LinkUART 921 600
Latency comp.640 ms history
Gain kp0.70
Dead zone2°
Max speed90°/s
Settle≈500 ms, no overshoot
ESP32-CAM tracking board with four Dupont wires
ESP32-CAM tracking board with four Dupont wires
Main ESP32: camera UART on IO16 / IO17, servos on IO18 / IO19
Main ESP32: camera UART on IO16 / IO17, servos on IO18 / IO19
ESP32-CAM

Face detection at 7 fps on a $6 board

The camera sketch uses Espressif’s esp-dl face detector from the Arduino core esp32 2.0.17. The headers it needs were removed in core 3.x, so the version matters. The detector costs about 17 µs per input pixel, which makes a full 320 × 240 frame slow. The firmware therefore has two paths:

  • Search: the whole frame is scanned only when no face is locked, or right after a miss.
  • Fast path: once a face is found, only a square window around it is cut out, averaged down to a small input (each output pixel is the mean of 4 samples, which also removes noise) and searched again. That is what keeps detection near 7.2 per second.
  • The sensor runs free with two frame buffers and the newest frame wins, so the loop never waits for the camera.
  • With PSRAM it works at QVGA (320 × 240); without PSRAM it still tracks at QQVGA (160 × 120). Wi-Fi is never started.
UART

A text protocol you can read

Every processed frame produces exactly one line. No pictures leave the camera during tracking, only the face box and timing:

camera → main, 921600 baud
F <cx> <cy> <w> <score*100> <W> <H> <ms>    face found
N <W> <H> <ms>                              no face in this frame

cx cy w are the face centre and width in pixels, W H the frame size and ms how old the frame was when the line was sent. That last number is what makes the control law work.

Main ESP32

The control law: correct from where the camera was

A naive tracker takes the error in the latest picture and moves by that amount. With a slow detector this overshoots: by the time the report arrives the head has already moved. OKO-1 records the servo positions every 10 ms for the last 640 ms, looks up where the head pointed when the frame was taken, and sets the target relative to that position. Nothing is integrated, so the loop stays calm at any frame rate.

firmware 2.6, plain law
// main ESP32, on every face report from the camera
dpp   = fov / max(W, H)                 // degrees per pixel, fov = 56° (OV2640 stock lens)
ex    = (cx - W/2) * dpp - aimX         // horizontal error, degrees
ey    = (H/2 - cy) * dpp - aimY         // vertical error, degrees
back  = (frameAge + lag) / 10 ms        // lag = 80 ms by default
pan0  = panHistory[back]                // where the head pointed when the frame was taken
if |ex| > dead:  panTarget = pan0 + ex * kp     // kp = 0.70, dead = 2°
(same for tilt)
  • Self-check: if corrections keep making the error bigger, the firmware flips that axis by itself and saves it. A servo mounted the other way round fixes itself in a few seconds.
  • Aim offset: your webcam sits a few centimetres from the tracking camera. aimx / aimy shift the target so the face is centred in the webcam, not in the ESP32-CAM.
  • Home: after 6 s without a face the head glides back to the middle.
  • A predictive alpha-beta filter exists as an experiment (set pred), but it is off after every start: in live tests the plain law felt better.
Servos

Smooth motion with a critically damped follower

The servos never jump to the target. They follow it like a critically damped spring: the move starts gently, slows down gently and never overshoots. About 90 % of a move is done after smooth ms (500 by default), capped at 90°/s. A new target in the middle of a move just bends the path. The PWM value is only rewritten when it changes, which removes the classic hobby-servo jitter. MG90S scale: about 10.3 µs per degree.

Gotchas

Powering two servos from a laptop port

  • Two MG90S pulling current dip the USB 5 V rail. The firmware disables the ESP32 brownout reset, otherwise the board would reboot on every fast move.
  • At power-up the servos start one after another, 350 ms apart, to lower the current peak.
  • A 470-1000 µF capacitor across 5 V and GND helps a weak port.
  • The camera runs from the 3V3 pin of the main board; its own 5 V pin stays empty.
Serial monitor, 921600

Tuning commands

Defaults vs the reference build. Numbers on this page are firmware defaults. The OKO-1 in the videos runs tuned values: kp 0.85, dead zone 1.5°, 180°/s, lag 120 ms, smooth 500 ms, tilt down to −65°. Tune yours with the commands below and save.
CommandWhat it does
statusLive state: LOCK / SEARCH / NOLINK, angles, fps
track 0 / 1Tracking off / on
centerBoth servos to the middle (for fitting the horns)
pan <deg> / tilt <deg>Manual move
flip pan | tiltReverse an axis
set kp | dead | speed | smooth | aimx | aimy …Tuning; `show` lists everything
cam <rot | score | in | roi | perf …>ESP32-CAM settings, passed through
bridgeUSB ↔ camera link for flashing the ESP32-CAM
logLast 256 camera reports with servo angles
saveStore settings in flash
FAQ

Maker questions

Why does the ESP32-CAM face detection not compile on Arduino core 3.x?

The esp-dl face detector headers used by the classic CameraWebServer example were removed from the Arduino-ESP32 core in 3.x. OKO-1 pins core 2.0.17, where they are still included.

Can one ESP32-CAM drive the servos directly?

It can, but the camera board has few free pins and shares its power with the sensor. OKO-1 keeps the camera board only for detection and lets a second, cheap ESP32 do PWM, smoothing and USB settings. It is more robust and still costs about $4.50 extra.

How do you stop servo tracking from overshooting?

Correct from where the head pointed when the frame was taken, not from where it is now. OKO-1 stores the servo positions every 10 ms and uses the frame age from the camera to look back, then moves by 70 % of the error with a 2° dead zone and critically damped easing.

Does it work without PSRAM?

Yes. Without PSRAM the camera works at 160 × 120 and still tracks a face at desk distance.

Source

Fork it, port it, improve it

MIT licence. Both sketches, the flashing scripts and a Python monitor with live telemetry are in the repository.