Commit ยท
91cfdeb
1
Parent(s): 5f422a0
docs: translate PROJECT_PLAN.md to English
Browse files- PROJECT_PLAN.md +315 -1058
PROJECT_PLAN.md
CHANGED
|
@@ -1,29 +1,29 @@
|
|
| 1 |
-
# Reachy Mini Home Assistant Voice Assistant -
|
| 2 |
|
| 3 |
-
##
|
| 4 |
|
| 5 |
-
|
| 6 |
|
| 7 |
-
##
|
| 8 |
-
1. [linux-voice-assistant](reference/linux-voice-assistant)
|
| 9 |
-
2. [Reachy Mini SDK](reference/reachy_mini)
|
| 10 |
-
3. [reachy_mini_conversation_app](reference/reachy_mini_conversation_app) - Reachy Mini
|
| 11 |
-
4. [reachy-mini-desktop-app](reference/reachy-mini-desktop-app) - Reachy Mini
|
| 12 |
-
5. [sendspin](reference/sendspin-cli/) -
|
| 13 |
|
| 14 |
-
##
|
| 15 |
|
| 16 |
-
1. **
|
| 17 |
-
2. **
|
| 18 |
-
3. **Home Assistant
|
| 19 |
-
4. **
|
| 20 |
-
5. **
|
| 21 |
-
6. **
|
| 22 |
-
7. **
|
| 23 |
-
8. **
|
| 24 |
-
9. **
|
| 25 |
|
| 26 |
-
##
|
| 27 |
|
| 28 |
```
|
| 29 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
@@ -40,16 +40,16 @@
|
|
| 40 |
โ โ
|
| 41 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 42 |
โ โ Camera + Face Tracking (YOLO) โ โ
|
| 43 |
-
โ โ - 15Hz
|
| 44 |
-
โ โ - look_at_image()
|
| 45 |
-
โ โ -
|
| 46 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 47 |
โ โ
|
| 48 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 49 |
โ โ Motion Controller (Head + Antennas) - 5Hz โ โ
|
| 50 |
โ โ - Face tracking offsets (secondary pose) โ โ
|
| 51 |
-
โ โ - Speech sway (
|
| 52 |
-
โ โ - Breathing animation (
|
| 53 |
โ โ - on_wakeup โ on_listening โ on_speaking โ on_idle โ โ
|
| 54 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 55 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
@@ -65,1035 +65,348 @@
|
|
| 65 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 66 |
```
|
| 67 |
|
| 68 |
-
##
|
| 69 |
-
|
| 70 |
-
###
|
| 71 |
-
- [x] ESPHome
|
| 72 |
-
- [x] mDNS
|
| 73 |
-
- [x]
|
| 74 |
-
- [x]
|
| 75 |
-
- [x]
|
| 76 |
-
- [x] TTS
|
| 77 |
-
- [x]
|
| 78 |
-
|
| 79 |
-
### Reachy Mini
|
| 80 |
-
- [x]
|
| 81 |
-
- [x]
|
| 82 |
-
- [x]
|
| 83 |
-
- [x]
|
| 84 |
-
- [x]
|
| 85 |
-
- [x] YOLO
|
| 86 |
-
- [x] 5Hz
|
| 87 |
-
|
| 88 |
-
###
|
| 89 |
-
- [x]
|
| 90 |
-
- [x]
|
| 91 |
-
- [x]
|
| 92 |
-
- [x]
|
| 93 |
-
|
| 94 |
-
##
|
| 95 |
|
| 96 |
```
|
| 97 |
reachy_mini_ha_voice/
|
| 98 |
โโโ reachy_mini_ha_voice/
|
| 99 |
-
โ โโโ __init__.py #
|
| 100 |
-
โ โโโ __main__.py
|
| 101 |
-
โ โโโ main.py # ReachyMiniApp
|
| 102 |
-
โ โโโ voice_assistant.py #
|
| 103 |
-
โ โโโ satellite.py # ESPHome
|
| 104 |
-
โ โโโ audio_player.py #
|
| 105 |
-
โ โโโ camera_server.py # MJPEG
|
| 106 |
-
โ โโโ head_tracker.py # YOLO
|
| 107 |
-
โ โโโ motion.py #
|
| 108 |
-
โ โโโ movement_manager.py #
|
| 109 |
-
โ โโโ models.py #
|
| 110 |
-
โ โโโ entity.py # ESPHome
|
| 111 |
-
โ โโโ entity_extensions.py #
|
| 112 |
-
โ โโโ reachy_controller.py # Reachy Mini
|
| 113 |
-
โ โโโ api_server.py # API
|
| 114 |
-
โ โโโ zeroconf.py # mDNS
|
| 115 |
-
โ โโโ util.py #
|
| 116 |
-
โโโ wakewords/ #
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
โ โโโ stop.json
|
| 122 |
-
โ โโโ stop.tflite
|
| 123 |
-
โโโ sounds/ # ้ณๆๆไปถ๏ผ่ชๅจไธ่ฝฝ๏ผ
|
| 124 |
-
โ โโโ wake_word_triggered.flac
|
| 125 |
-
โ โโโ timer_finished.flac
|
| 126 |
-
โโโ pyproject.toml # ้กน็ฎ้
็ฝฎ
|
| 127 |
-
โโโ README.md # ่ฏดๆๆๆกฃ
|
| 128 |
-
โโโ PROJECT_PLAN.md # ้กน็ฎ่ฎกๅ
|
| 129 |
```
|
| 130 |
|
| 131 |
-
##
|
| 132 |
-
|
| 133 |
-
```toml
|
| 134 |
-
dependencies = [
|
| 135 |
-
"reachy-mini", # Reachy Mini SDK
|
| 136 |
-
"sounddevice>=0.4.6", # ้ณ้ขๅค็๏ผๅค็จ๏ผ
|
| 137 |
-
"soundfile>=0.12.0", # ้ณ้ขๆไปถ่ฏปๅ
|
| 138 |
-
"numpy>=1.24.0", # ๆฐๅผ่ฎก็ฎ
|
| 139 |
-
"pymicro-wakeword>=2.0.0,<3.0.0", # ๅค้่ฏๆฃๆต
|
| 140 |
-
"pyopen-wakeword>=1.0.0,<2.0.0", # ๅค็จๅค้่ฏ
|
| 141 |
-
"aioesphomeapi>=42.0.0", # ESPHome ๅ่ฎฎ
|
| 142 |
-
"zeroconf>=0.100.0", # mDNS ๅ็ฐ
|
| 143 |
-
"scipy>=1.10.0", # ่ฟๅจๆงๅถ
|
| 144 |
-
"pydantic>=2.0.0", # ๆฐๆฎ้ช่ฏ
|
| 145 |
-
]
|
| 146 |
-
```
|
| 147 |
-
|
| 148 |
-
## ไฝฟ็จๆต็จ
|
| 149 |
-
|
| 150 |
-
1. **ๅฎ่ฃ
ๅบ็จ**
|
| 151 |
-
- ไป Reachy Mini App Store ๅฎ่ฃ
`reachy-mini-ha-voice`
|
| 152 |
-
|
| 153 |
-
2. **ๅฏๅจๅบ็จ**
|
| 154 |
-
- ๅบ็จ่ชๅจๅฏๅจ ESPHome ๆๅกๅจ๏ผ็ซฏๅฃ 6053๏ผ
|
| 155 |
-
- ่ชๅจไธ่ฝฝๆ้ๆจกๅๅ้ณๆ
|
| 156 |
-
|
| 157 |
-
3. **่ฟๆฅ Home Assistant**
|
| 158 |
-
- Home Assistant ่ชๅจๅ็ฐ่ฎพๅค๏ผmDNS๏ผ
|
| 159 |
-
- ๆๆๅจๆทปๅ ๏ผ่ฎพ็ฝฎ โ ่ฎพๅคไธๆๅก โ ๆทปๅ ้ๆ โ ESPHome
|
| 160 |
-
|
| 161 |
-
4. **ไฝฟ็จ่ฏญ้ณๅฉๆ**
|
| 162 |
-
- ่ฏด "Okay Nabu" ๅค้
|
| 163 |
-
- ่ฏดๅบๅฝไปค
|
| 164 |
-
- Reachy Mini ไผๅๅบ่ฟๅจๅ้ฆ
|
| 165 |
-
|
| 166 |
-
## ESPHome ๅฎไฝ่งๅ
|
| 167 |
-
|
| 168 |
-
ๅบไบ Reachy Mini SDK ๆทฑๅ
ฅๅๆ๏ผไปฅไธๅฎไฝๅทฒๆด้ฒ็ป Home Assistant๏ผ
|
| 169 |
-
|
| 170 |
-
### ๅทฒๅฎ็ฐๅฎไฝ
|
| 171 |
-
|
| 172 |
-
| ๅฎไฝ็ฑปๅ | ๅ็งฐ | ่ฏดๆ |
|
| 173 |
-
|---------|------|------|
|
| 174 |
-
| Media Player | `media_player` | ้ณ้ขๆญๆพๆงๅถ |
|
| 175 |
-
| Voice Assistant | `voice_assistant` | ่ฏญ้ณๅฉๆ็ฎก้ |
|
| 176 |
-
|
| 177 |
-
### ๅทฒๅฎ็ฐ็ๆงๅถๅฎไฝ (Controls) - ๅฏ่ฏปๅ
|
| 178 |
-
|
| 179 |
-
#### Phase 1-3: ๅบ็กๆงๅถไธๅงฟๆ
|
| 180 |
-
|
| 181 |
-
| ESPHome ๅฎไฝ็ฑปๅ | ๅ็งฐ | SDK API | ่ๅด/้้กน | ่ฏดๆ |
|
| 182 |
-
|-----------------|------|---------|----------|------|
|
| 183 |
-
| `Number` | `speaker_volume` | `AudioPlayer.set_volume()` | 0-100 | ๆฌๅฃฐๅจ้ณ้ |
|
| 184 |
-
| `Select` | `motor_mode` | `set_motor_control_mode()` | enabled/disabled/gravity_compensation | ็ตๆบๆจกๅผ้ๆฉ |
|
| 185 |
-
| `Switch` | `motors_enabled` | `enable_motors()` / `disable_motors()` | on/off | ็ตๆบๆญ็ฉๅผๅ
ณ |
|
| 186 |
-
| `Button` | `wake_up` | `mini.wake_up()` | - | ๅค้ๆบๅจไบบๅจไฝ |
|
| 187 |
-
| `Button` | `go_to_sleep` | `mini.goto_sleep()` | - | ็ก็ ๆบๅจไบบๅจไฝ |
|
| 188 |
-
| `Number` | `head_x` | `goto_target(head=...)` | ยฑ50mm | ๅคด้จ X ไฝ็ฝฎๆงๅถ |
|
| 189 |
-
| `Number` | `head_y` | `goto_target(head=...)` | ยฑ50mm | ๅคด้จ Y ไฝ็ฝฎๆงๅถ |
|
| 190 |
-
| `Number` | `head_z` | `goto_target(head=...)` | ยฑ50mm | ๅคด้จ Z ไฝ็ฝฎๆงๅถ |
|
| 191 |
-
| `Number` | `head_roll` | `goto_target(head=...)` | -40ยฐ ~ +40ยฐ | ๅคด้จ็ฟปๆป่งๆงๅถ |
|
| 192 |
-
| `Number` | `head_pitch` | `goto_target(head=...)` | -40ยฐ ~ +40ยฐ | ๅคด้จไฟฏไปฐ่งๆงๅถ |
|
| 193 |
-
| `Number` | `head_yaw` | `goto_target(head=...)` | -180ยฐ ~ +180ยฐ | ๅคด้จๅ่ช่งๆงๅถ |
|
| 194 |
-
| `Number` | `body_yaw` | `goto_target(body_yaw=...)` | -160ยฐ ~ +160ยฐ | ่บซไฝๅ่ช่งๆงๅถ |
|
| 195 |
-
| `Number` | `antenna_left` | `goto_target(antennas=...)` | -90ยฐ ~ +90ยฐ | ๅทฆๅคฉ็บฟ่งๅบฆๆงๅถ |
|
| 196 |
-
| `Number` | `antenna_right` | `goto_target(antennas=...)` | -90ยฐ ~ +90ยฐ | ๅณๅคฉ็บฟ่งๅบฆๆงๅถ |
|
| 197 |
-
|
| 198 |
-
#### Phase 4: ๆณจ่งๆงๅถ
|
| 199 |
-
|
| 200 |
-
| ESPHome ๅฎไฝ็ฑปๅ | ๅ็งฐ | SDK API | ่ๅด/้้กน | ่ฏดๆ |
|
| 201 |
-
|-----------------|------|---------|----------|------|
|
| 202 |
-
| `Number` | `look_at_x` | `look_at_world(x, y, z)` | ไธ็ๅๆ | ๆณจ่ง็น X ๅๆ |
|
| 203 |
-
| `Number` | `look_at_y` | `look_at_world(x, y, z)` | ไธ็ๅๆ | ๆณจ่ง็น Y ๅๆ |
|
| 204 |
-
| `Number` | `look_at_z` | `look_at_world(x, y, z)` | ไธ็ๅๆ | ๆณจ่ง็น Z ๅๆ |
|
| 205 |
-
|
| 206 |
-
### ๅทฒๅฎ็ฐ็ไผ ๆๅจๅฎไฝ (Sensors) - ๅช่ฏป
|
| 207 |
-
|
| 208 |
-
#### Phase 1 & 5: ๅบ็ก็ถๆไธ้ณ้ขไผ ๆๅจ
|
| 209 |
-
|
| 210 |
-
| ESPHome ๅฎไฝ๏ฟฝ๏ฟฝ๏ฟฝๅ | ๅ็งฐ | SDK API | ่ฏดๆ |
|
| 211 |
-
|-----------------|------|---------|------|
|
| 212 |
-
| `Text Sensor` | `daemon_state` | `DaemonStatus.state` | Daemon ็ถๆ |
|
| 213 |
-
| `Binary Sensor` | `backend_ready` | `backend_status.ready` | ๅ็ซฏๆฏๅฆๅฐฑ็ปช |
|
| 214 |
-
| `Text Sensor` | `error_message` | `DaemonStatus.error` | ๅฝๅ้่ฏฏไฟกๆฏ |
|
| 215 |
-
| `Sensor` | `doa_angle` | `DoAInfo.angle` | ๅฃฐๆบๆนๅ่งๅบฆ (ยฐ) |
|
| 216 |
-
| `Binary Sensor` | `speech_detected` | `DoAInfo.speech_detected` | ๆฏๅฆๆฃๆตๅฐ่ฏญ้ณ |
|
| 217 |
-
|
| 218 |
-
#### Phase 6: ่ฏๆญไฟกๆฏ
|
| 219 |
-
|
| 220 |
-
| ESPHome ๅฎไฝ็ฑปๅ | ๅ็งฐ | SDK API | ่ฏดๆ |
|
| 221 |
-
|-----------------|------|---------|------|
|
| 222 |
-
| `Sensor` | `control_loop_frequency` | `control_loop_stats` | ๆงๅถๅพช็ฏ้ข็ (Hz) |
|
| 223 |
-
| `Text Sensor` | `sdk_version` | `DaemonStatus.version` | SDK ็ๆฌๅท |
|
| 224 |
-
| `Text Sensor` | `robot_name` | `DaemonStatus.robot_name` | ๆบๅจไบบๅ็งฐ |
|
| 225 |
-
| `Binary Sensor` | `wireless_version` | `DaemonStatus.wireless_version` | ๆฏๅฆไธบๆ ็บฟ็ๆฌ |
|
| 226 |
-
| `Binary Sensor` | `simulation_mode` | `DaemonStatus.simulation_enabled` | ๆฏๅฆๅจไปฟ็ๆจกๅผ |
|
| 227 |
-
| `Text Sensor` | `wlan_ip` | `DaemonStatus.wlan_ip` | ๆ ็บฟ็ฝ็ป IP |
|
| 228 |
-
|
| 229 |
-
#### Phase 7: IMU ไผ ๆๅจ (ไป
ๆ ็บฟ็ๆฌ)
|
| 230 |
-
|
| 231 |
-
| ESPHome ๅฎไฝ็ฑปๅ | ๅ็งฐ | SDK API | ่ฏดๆ |
|
| 232 |
-
|-----------------|------|---------|------|
|
| 233 |
-
| `Sensor` | `imu_accel_x` | `mini.imu["accelerometer"][0]` | X ่ฝดๅ ้ๅบฆ (m/sยฒ) |
|
| 234 |
-
| `Sensor` | `imu_accel_y` | `mini.imu["accelerometer"][1]` | Y ่ฝดๅ ้ๅบฆ (m/sยฒ) |
|
| 235 |
-
| `Sensor` | `imu_accel_z` | `mini.imu["accelerometer"][2]` | Z ่ฝดๅ ้ๅบฆ (m/sยฒ) |
|
| 236 |
-
| `Sensor` | `imu_gyro_x` | `mini.imu["gyroscope"][0]` | X ่ฝด่ง้ๅบฆ (rad/s) |
|
| 237 |
-
| `Sensor` | `imu_gyro_y` | `mini.imu["gyroscope"][1]` | Y ่ฝด่ง้ๅบฆ (rad/s) |
|
| 238 |
-
| `Sensor` | `imu_gyro_z` | `mini.imu["gyroscope"][2]` | Z ่ฝด่ง้ๅบฆ (rad/s) |
|
| 239 |
-
| `Sensor` | `imu_temperature` | `mini.imu["temperature"]` | IMU ๆธฉๅบฆ (ยฐC) |
|
| 240 |
-
|
| 241 |
-
#### Phase 8-12: ๆฉๅฑๅ่ฝ
|
| 242 |
-
|
| 243 |
-
| ESPHome ๅฎไฝ็ฑปๅ | ๅ็งฐ | ่ฏดๆ |
|
| 244 |
-
|-----------------|------|------|
|
| 245 |
-
| `Select` | `emotion` | ่กจๆ
้ๆฉๅจ (Happy/Sad/Angry/Fear/Surprise/Disgust) |
|
| 246 |
-
| `Number` | `microphone_volume` | ้บฆๅ
้ฃ้ณ้ (0-100%) |
|
| 247 |
-
| `Camera` | `camera` | ESPHome Camera ๅฎไฝ๏ผๅฎๆถ้ข่ง๏ผ |
|
| 248 |
-
| `Number` | `led_brightness` | LED ไบฎๅบฆ (0-100%) |
|
| 249 |
-
| `Select` | `led_effect` | LED ๆๆ (off/solid/breathing/rainbow/doa) |
|
| 250 |
-
| `Number` | `led_color_r` | LED ็บข่ฒๅ้ (0-255) |
|
| 251 |
-
| `Number` | `led_color_g` | LED ็ปฟ่ฒๅ้ (0-255) |
|
| 252 |
-
| `Number` | `led_color_b` | LED ่่ฒๅ้ (0-255) |
|
| 253 |
-
| `Switch` | `agc_enabled` | ่ชๅจๅข็ๆงๅถๅผๅ
ณ |
|
| 254 |
-
| `Number` | `agc_max_gain` | AGC ๆๅคงๅข็ (0-30 dB) |
|
| 255 |
-
| `Number` | `noise_suppression` | ๅชๅฃฐๆๅถ็บงๅซ (0-100%) |
|
| 256 |
-
| `Binary Sensor` | `echo_cancellation_converged` | ๅๅฃฐๆถ้คๆถๆ็ถๆ |
|
| 257 |
-
|
| 258 |
-
> **ๆณจๆ**: ๅคด้จไฝ็ฝฎ (x/y/z) ๅ่งๅบฆ (roll/pitch/yaw)ใ่บซไฝๅ่ช่งใๅคฉ็บฟ่งๅบฆ้ฝๆฏ**ๅฏๆงๅถ**็ๅฎไฝ๏ผ
|
| 259 |
-
> ไฝฟ็จ `Number` ็ฑปๅๅฎ็ฐๅๅๆงๅถใ่ฎพ็ฝฎๆฐๅผๆถ่ฐ็จ `goto_target()`๏ผ่ฏปๅๅฝๅๅผๆถ่ฐ็จ `get_current_head_pose()` ็ญใ
|
| 260 |
-
|
| 261 |
-
### ๅฎ็ฐไผๅ
็บง
|
| 262 |
-
|
| 263 |
-
1. **Phase 1 - ๅบ็ก็ถๆไธ้ณ้** (้ซไผๅ
็บง) โ
**ๅทฒๅฎๆ**
|
| 264 |
-
- [x] `daemon_state` - Daemon ็ถๆไผ ๆๅจ
|
| 265 |
-
- [x] `backend_ready` - ๅ็ซฏๅฐฑ็ปช็ถๆ
|
| 266 |
-
- [x] `error_message` - ้่ฏฏไฟกๆฏ
|
| 267 |
-
- [x] `speaker_volume` - ๆฌๅฃฐๅจ้ณ้ๆงๅถ
|
| 268 |
-
|
| 269 |
-
2. **Phase 2 - ็ตๆบๆงๅถ** (้ซไผๅ
็บง) โ
**ๅทฒๅฎๆ**
|
| 270 |
-
- [x] `motors_enabled` - ็ตๆบๅผๅ
ณ
|
| 271 |
-
- [x] `motor_mode` - ็ตๆบๆจกๅผ้ๆฉ (enabled/disabled/gravity_compensation)
|
| 272 |
-
- [x] `wake_up` / `go_to_sleep` - ๅค้/็ก็ ๆ้ฎ
|
| 273 |
-
|
| 274 |
-
3. **Phase 3 - ๅงฟๆๆงๅถ** (ไธญไผๅ
็บง) โ
**ๅทฒๅฎๆ**
|
| 275 |
-
- [x] `head_x/y/z` - ๅคด้จไฝ็ฝฎๆงๅถ
|
| 276 |
-
- [x] `head_roll/pitch/yaw` - ๅคด้จ่งๅบฆๆงๅถ
|
| 277 |
-
- [x] `body_yaw` - ่บซไฝๅ่ช่งๆงๅถ
|
| 278 |
-
- [x] `antenna_left/right` - ๅคฉ็บฟ่งๅบฆๆงๅถ
|
| 279 |
-
|
| 280 |
-
4. **Phase 4 - ๆณจ่งๆงๅถ** (ไธญไผๅ
็บง) โ
**ๅทฒๅฎๆ**
|
| 281 |
-
- [x] `look_at_x/y/z` - ๆณจ่ง็นๅๆ ๆงๅถ
|
| 282 |
-
|
| 283 |
-
5. **Phase 5 - ้ณ้ขไผ ๆๅจ** (ไฝไผๅ
็บง) โ
**ๅทฒๅฎๆ**
|
| 284 |
-
- [x] `doa_angle` - ๅฃฐๆบๆนๅ
|
| 285 |
-
- [x] `speech_detected` - ่ฏญ้ณๆฃๆต
|
| 286 |
-
|
| 287 |
-
6. **Phase 6 - ่ฏๆญไฟกๆฏ** (ไฝไผๅ
็บง) โ
**ๅทฒๅฎๆ**
|
| 288 |
-
- [x] `control_loop_frequency` - ๆงๅถๅพช็ฏ้ข็
|
| 289 |
-
- [x] `sdk_version` - SDK ็ๆฌ
|
| 290 |
-
- [x] `robot_name` - ๆบๅจไบบๅ็งฐ
|
| 291 |
-
- [x] `wireless_version` - ๆ ็บฟ็ๆฌๆ ่ฏ
|
| 292 |
-
- [x] `simulation_mode` - ไปฟ็ๆจกๅผๆ ่ฏ
|
| 293 |
-
- [x] `wlan_ip` - ๆ ็บฟ IP ๅฐๅ
|
| 294 |
-
|
| 295 |
-
7. **Phase 7 - IMU ไผ ๆๅจ** (ๅฏ้๏ผไป
ๆ ็บฟ็ๆฌ) โ
**ๅทฒๅฎๆ**
|
| 296 |
-
- [x] `imu_accel_x/y/z` - ๅ ้ๅบฆ่ฎก
|
| 297 |
-
- [x] `imu_gyro_x/y/z` - ้่บไปช
|
| 298 |
-
- [x] `imu_temperature` - IMU ๆธฉๅบฆ
|
| 299 |
-
|
| 300 |
-
8. **Phase 8 - ่กจๆ
ๆงๅถ** โ
**ๅทฒๅฎๆ**
|
| 301 |
-
- [x] `emotion` - ่กจๆ
้ๆฉๅจ (Happy/Sad/Angry/Fear/Surprise/Disgust)
|
| 302 |
-
|
| 303 |
-
9. **Phase 9 - ้ณ้ขๆงๅถ** โ
**ๅทฒๅฎๆ**
|
| 304 |
-
- [x] `microphone_volume` - ้บฆๅ
้ฃ้ณ้ๆงๅถ (0-100%)
|
| 305 |
-
|
| 306 |
-
10. **Phase 10 - ๆๅๅคด้ๆ** โ
**ๅทฒๅฎๆ**
|
| 307 |
-
- [x] `camera` - ESPHome Camera ๅฎไฝ๏ผๅฎๆถ้ข่ง๏ผ
|
| 308 |
-
|
| 309 |
-
11. **Phase 11 - LED ๆงๅถ** โ **ๅทฒ็ฆ็จ๏ผLED ้่ๅจๆบๅจไบบๅ
้จ๏ผ**
|
| 310 |
-
- [ ] `led_brightness` - LED ไบฎๅบฆ (0-100%) - ๅทฒๆณจ้
|
| 311 |
-
- [ ] `led_effect` - LED ๆๆ (off/solid/breathing/rainbow/doa) - ๅทฒๆณจ้
|
| 312 |
-
- [ ] `led_color_r/g/b` - LED RGB ้ข่ฒ (0-255) - ๅทฒๆณจ้
|
| 313 |
-
|
| 314 |
-
12. **Phase 12 - ้ณ้ขๅค็ๅๆฐ** โ
**ๅทฒๅฎๆ**
|
| 315 |
-
- [x] `agc_enabled` - ่ชๅจๅข็ๆงๅถๅผๅ
ณ
|
| 316 |
-
- [x] `agc_max_gain` - AGC ๆๅคงๅข็ (0-30 dB)
|
| 317 |
-
- [x] `noise_suppression` - ๅชๅฃฐๆๅถ็บงๅซ (0-100%)
|
| 318 |
-
- [x] `echo_cancellation_converged` - ๅๅฃฐๆถ้คๆถๆ็ถๆ๏ผๅช่ฏป๏ผ
|
| 319 |
-
|
| 320 |
-
13. **Phase 13 - Sendspin ้ณ้ขๆญๆพๆฏๆ** โ
**ๅทฒๅฎๆ**
|
| 321 |
-
- [x] `sendspin_enabled` - Sendspin ๅผๅ
ณ (Switch)
|
| 322 |
-
- [x] `sendspin_url` - Sendspin ๆๅกๅจ URL (Text Sensor)
|
| 323 |
-
- [x] `sendspin_connected` - Sendspin ่ฟๆฅ็ถๆ (Binary Sensor)
|
| 324 |
-
- [x] AudioPlayer ้ๆ aiosendspin ๅบ
|
| 325 |
-
- [x] TTS ้ณ้ขๅๆถๅ้ๅฐๆฌๅฐๆฌๅฃฐๅจๅ Sendspin ๆๅกๅจ
|
| 326 |
-
|
| 327 |
-
---
|
| 328 |
-
|
| 329 |
-
## ๐ Phase 1-13 ๅฎไฝๅทฒๅฎๆ๏ผ
|
| 330 |
-
|
| 331 |
-
**ๅทฒๅฎๆๆป่ฎก๏ผ43 ไธชๅฎไฝ**
|
| 332 |
-
- Phase 1: 4 ไธชๅฎไฝ (ๅบ็ก็ถๆไธ้ณ้)
|
| 333 |
-
- Phase 2: 4 ไธชๅฎไฝ (็ตๆบๆงๅถ)
|
| 334 |
-
- Phase 3: 9 ไธชๅฎไฝ (ๅงฟๆๆงๅถ)
|
| 335 |
-
- Phase 4: 3 ไธชๅฎไฝ (ๆณจ่งๆงๅถ)
|
| 336 |
-
- Phase 5: 2 ไธชๅฎไฝ (้ณ้ขไผ ๆๅจ)
|
| 337 |
-
- Phase 6: 6 ไธชๅฎไฝ (่ฏๆญไฟกๆฏ)
|
| 338 |
-
- Phase 7: 7 ไธชๅฎไฝ (IMU ไผ ๆๅจ)
|
| 339 |
-
- Phase 8: 1 ไธชๅฎไฝ (่กจๆ
ๆงๅถ)
|
| 340 |
-
- Phase 9: 1 ไธชๅฎไฝ (้บฆๅ
้ฃ้ณ้)
|
| 341 |
-
- Phase 10: 1 ไธชๅฎไฝ (ๆๅๅคด)
|
| 342 |
-
- Phase 11: 0 ไธชๅฎไฝ (LED ๆงๅถ - ๅทฒ็ฆ็จ)
|
| 343 |
-
- Phase 12: 4 ไธชๅฎไฝ (้ณ้ขๅค็ๅๆฐ)
|
| 344 |
-
- Phase 13: 3 ไธชๅฎไฝ (Sendspin ้ณ้ข่พๅบ)
|
| 345 |
-
|
| 346 |
-
---
|
| 347 |
-
|
| 348 |
-
## ๐ ่ฏญ้ณๅฉๆๅขๅผบๅ่ฝๅฎ็ฐ็ถๆ
|
| 349 |
-
|
| 350 |
-
### Phase 14 - ๆ
ๆๅจไฝๅ้ฆ็ณป็ป (้จๅๅฎ็ฐ) ๐ก
|
| 351 |
-
|
| 352 |
-
**ๅฎ็ฐ็ถๆ**: ๅบ็กๆถๆๅทฒๅฐฑ็ปช,ๆฏๆๆๅจ่งฆๅ,ๅฏน่ฏๆถไฝฟ็จ่ฏญ้ณ้ฉฑๅจ็่ช็ถๅพฎๅจ
|
| 353 |
-
|
| 354 |
-
**ๅทฒๅฎ็ฐๅ่ฝ**:
|
| 355 |
-
- โ
Phase 8 Emotion Selector ๅฎไฝ (`emotion`)
|
| 356 |
-
- โ
ๅบ็กๆ
ๆๅจไฝๆญๆพAPI (`_play_emotion`)
|
| 357 |
-
- โ
ๆ
ๆๆ ๅฐ: Happy/Sad/Angry/Fear/Surprise/Disgust
|
| 358 |
-
- โ
ไธ HuggingFace ๅจไฝๅบ้ๆ (`pollen-robotics/reachy-mini-emotions-library`)
|
| 359 |
-
- โ
ๅฏน่ฏๆถไฝฟ็จ SpeechSway ็ณป็ปๆไพ่ช็ถ็ๅคด้จๅพฎๅจ (ไธ้ปๅกๅฏน่ฏไฝ้ช)
|
| 360 |
-
|
| 361 |
-
**่ฎพ่ฎกๅณ็ญ**:
|
| 362 |
-
- ๐ฏ ๅฏน่ฏๆถไธ่ชๅจๆญๆพๅฎๆดๆ
ๆๅจไฝ,้ฟๅ
้ปๅกๅฏน่ฏไฝ้ช
|
| 363 |
-
- ๐ฏ ไฝฟ็จ่ฏญ้ณ้ฉฑๅจ็ๅคด้จๆๅจ (SpeechSway) ๆไพ่ช็ถ็ๅจไฝๅ้ฆ
|
| 364 |
-
- ๐ฏ ๆ
ๆๅจไฝไฟ็ไธบๆๅจ่งฆๅๅ่ฝ,ๅฏ้่ฟ ESPHome ๅฎไฝๆงๅถ
|
| 365 |
-
|
| 366 |
-
**ๆชๅฎ็ฐๅ่ฝ**:
|
| 367 |
-
- โ ่ชๅจๆ นๆฎ่ฏญ้ณๅฉๆๅๅบ่งฆๅๆ
ๆๅจไฝ (ๅทฒๅณๅฎไธๅฎ็ฐ,้ฟๅ
้ปๅก)
|
| 368 |
-
- โ ๆๅพ่ฏๅซไธๆ
ๆๅน้
|
| 369 |
-
- โ ่่นๅจไฝๅบ้ๆ
|
| 370 |
-
- โ ไธไธๆๆ็ฅ(ๅฆๅคฉๆฐๆฅ่ฏข-ๆดๅคฉๆญๆพ happy,้จๅคฉๆญๆพ sad)
|
| 371 |
-
|
| 372 |
-
**ไปฃ็ ไฝ็ฝฎ**:
|
| 373 |
-
- `entity_registry.py:633-658` - Emotion Selector ๅฎไฝ
|
| 374 |
-
- `satellite.py:544-574` - `_play_emotion()` ๆนๆณ
|
| 375 |
-
- `motion.py:132-156` - ๅฏน่ฏๅผๅงๆถ็ๅจไฝๆงๅถ (ไฝฟ็จ SpeechSway)
|
| 376 |
-
- `movement_manager.py:541-595` - Move ้ๅ็ฎก็ (ๅ
่ฎธ SpeechSway ๅ ๅ )
|
| 377 |
-
|
| 378 |
-
**ๅฎ้
่กไธบ**:
|
| 379 |
-
|
| 380 |
-
| ่ฏญ้ณๅฉๆไบไปถ | ๅฎ้
ๅจไฝ | ๅฎ็ฐ็ถๆ |
|
| 381 |
-
|-------------|---------|---------|
|
| 382 |
-
| ๅค้่ฏๆฃๆต | ่ฝฌๅๅฃฐๆบ + ็นๅคด็กฎ่ฎค | โ
ๅทฒๅฎ็ฐ |
|
| 383 |
-
| ๅฏน่ฏๅผๅง | ่ฏญ้ณ้ฉฑๅจ็ๅคด้จๅพฎๅจ (SpeechSway) | โ
ๅทฒๅฎ็ฐ |
|
| 384 |
-
| ๅฏน่ฏ่ฟ่กไธญ | ๆ็ปญ็่ฏญ้ณ้ฉฑๅจๅพฎๅจ + ๅผๅธๅจ็ป | โ
ๅทฒๅฎ็ฐ |
|
| 385 |
-
| ๅฏน่ฏ็ปๆ | ่ฟๅไธญ็ซไฝ็ฝฎ + ๅผๅธๅจ็ป | โ
ๅทฒๅฎ็ฐ |
|
| 386 |
-
| ๆๅจ่งฆๅๆ
ๆ | ้่ฟ ESPHome `emotion` ๅฎไฝๆญๆพ | โ
ๅทฒๅฎ็ฐ |
|
| 387 |
-
|
| 388 |
-
**ๆๆฏ่ฏดๆ**:
|
| 389 |
-
```python
|
| 390 |
-
# motion.py - ๅฏน่ฏๆถไฝฟ็จ SpeechSway ่้ๅฎๆดๆ
ๆๅจไฝ
|
| 391 |
-
def on_speaking_start(self):
|
| 392 |
-
self._is_speaking = True
|
| 393 |
-
self._movement_manager.set_state(RobotState.SPEAKING)
|
| 394 |
-
# SpeechSway ไผ่ชๅจๆ นๆฎ้ณ้ขๅๅบฆไบง็่ช็ถ็ๅคด้จๅพฎๅจ
|
| 395 |
-
# ไธๆญๆพๅฎๆดๆ
ๆๅจไฝ,้ฟๅ
้ปๅกๅฏน่ฏไฝ้ช
|
| 396 |
-
|
| 397 |
-
# movement_manager.py - ๅจไฝๅๅฑ็ณป็ป
|
| 398 |
-
# 1. Move ้ๅ (ๆ
ๆๅจไฝ) - ่ฎพ็ฝฎๅบ็กๅงฟๆ
|
| 399 |
-
# 2. Action (็นๅคด/ๆๅคด็ญ) - ๅ ๅ ๅจๅบ็กๅงฟๆไธ
|
| 400 |
-
# 3. SpeechSway - ่ฏญ้ณ้ฉฑๅจๅพฎๅจ,ๅฏไธ Move ๅ
ฑๅญ
|
| 401 |
-
# 4. Breathing - ็ฉบ้ฒๆถ็ๅผๅธๅจ็ป
|
| 402 |
-
```
|
| 403 |
-
|
| 404 |
-
**ๅๅง่งๅ** (ๅทฒๅณๅฎไธๅฎ็ฐ,้ฟๅ
้ปๅกๅฏน่ฏ):
|
| 405 |
-
|
| 406 |
-
| ่ฏญ้ณๅฉๆไบไปถ | ๅ่ฎกๅๅจไฝ | ไธๅฎ็ฐๅๅ |
|
| 407 |
-
|-------------|---------|---------|
|
| 408 |
-
| ๆถๅฐ่ฏๅฎๅๅค | ๆญๆพ "happy" ๅจไฝ | ๅฎๆดๅจไฝไผ้ปๅกๅฏน่ฏๆต็
ๆง |
|
| 409 |
-
| ๆถๅฐๅฆๅฎๅๅค | ๆญๆพ "sad" ๅจไฝ | ๅฎๆดๅจไฝไผ้ปๅกๅฏน่ฏๆต็
ๆง |
|
| 410 |
-
| ๆญๆพ้ณไน/ๅจฑไน | ๆญๆพ "dance" ๅจไฝ | ๅฎๆดๅจไฝไผ้ปๅกๅฏน่ฏๆต็
ๆง |
|
| 411 |
-
| ๅฎๆถๅจๅฎๆ | ๆญๆพ "alert" ๅจไฝ | ๅฎๆดๅจไฝไผ้ปๅกๅฏน่ฏๆต็
ๆง |
|
| 412 |
-
| ้่ฏฏ/ๆ ๆณ็่งฃ | ๆญๆพ "confused" ๅจไฝ | ๅฎๆดๅจไฝไผ้ปๅกๅฏน่ฏๆต็
ๆง |
|
| 413 |
-
|
| 414 |
-
**ๆๅจ่งฆๅๆ
ๆๅจไฝ็คบไพ**:
|
| 415 |
-
```yaml
|
| 416 |
-
# Home Assistant ่ชๅจๅ็คบไพ - ๆๅจ่งฆๅๆ
ๆ
|
| 417 |
-
automation:
|
| 418 |
-
- alias: "Reachy ๆฉๅฎ้ฎๅ"
|
| 419 |
-
trigger:
|
| 420 |
-
- platform: time
|
| 421 |
-
at: "07:00:00"
|
| 422 |
-
action:
|
| 423 |
-
- service: select.select_option
|
| 424 |
-
target:
|
| 425 |
-
entity_id: select.reachy_mini_emotion
|
| 426 |
-
data:
|
| 427 |
-
option: "Happy"
|
| 428 |
-
```
|
| 429 |
-
|
| 430 |
-
### Phase 15 - ไบบ่ธ่ฟฝ่ธช๏ผๆฟไปฃ DOA ๅฃฐๆบ่ฟฝ่ธช๏ผโ
**ๅทฒๅฎๆ**
|
| 431 |
-
|
| 432 |
-
**็ฎๆ **: ๅฎ็ฐ่ช็ถ็ไบบ่ธ่ฟฝ่ธช๏ผ่ฎฉๆบๅจไบบๅจๅฏน่ฏๆถๆณจ่ง่ฏด่ฏไบบใ
|
| 433 |
-
|
| 434 |
-
**่ฎพ่ฎกๅณ็ญ**:
|
| 435 |
-
- โ ๅ่ฎกๅไฝฟ็จ DOA (Direction of Arrival) ๅฃฐๆบ่ฟฝ่ธช
|
| 436 |
-
- โ
ๆน็จ YOLO ไบบ่ธๆฃๆต๏ผๆด็จณๅฎใๆดๅ็กฎ
|
| 437 |
-
- ๅๅ ๏ผDOA ๅจๅค้ๆถๅคๆญไธๅคๅ็กฎ๏ผไธ้ข็นๆฅ่ฏขไผๅฏผ่ด daemon ๅดฉๆบ
|
| 438 |
-
|
| 439 |
-
**ๅทฒๅฎ็ฐๅ่ฝ**:
|
| 440 |
-
|
| 441 |
-
| ๅ่ฝ | ่ฏดๆ | ๅฎ็ฐไฝ็ฝฎ | ๅฎ็ฐ็ถๆ |
|
| 442 |
-
|------|------|---------|---------|
|
| 443 |
-
| YOLO ไบบ่ธๆฃๆต | ไฝฟ็จ `AdamCodd/YOLOv11n-face-detection` ๆจกๅ | `head_tracker.py` | โ
ๅทฒๅฎ็ฐ |
|
| 444 |
-
| ่ช้ๅบๅธง็่ฟฝ่ธช | ๅฏน่ฏๆถ15fps๏ผ็ฉบ้ฒๆ ไบบ่ธๆถ3fps | `camera_server.py` | โ
ๅทฒๅฎ็ฐ |
|
| 445 |
-
| look_at_image() | ๆ นๆฎไบบ่ธไฝ็ฝฎ่ฎก็ฎ็ฎๆ ๅงฟๆ | `camera_server.py` | โ
ๅทฒๅฎ็ฐ |
|
| 446 |
-
| ๅนณๆปๅไธญๆงไฝ็ฝฎ | ไบบ่ธไธขๅคฑๅ 1 ็งๅ
ๅนณๆปๅๅฝ | `camera_server.py` | โ
ๅทฒๅฎ็ฐ |
|
| 447 |
-
| face_tracking_offsets | ไฝไธบ secondary pose ๅ ๅ ๅฐ่ฟๅจๆงๅถ | `movement_manager.py` | โ
ๅทฒๅฎ็ฐ |
|
| 448 |
-
| ่ฏญ้ณๆดปๅจๆฃๆต | DOA ๅฎไฝไปๅฏ็จไบๆฃๆต่ฏญ้ณ | `DoAInfo.speech_detected` | โ
ๅทฒๆด้ฒไธบๅฎไฝ |
|
| 449 |
-
| ๆจกๅไธ่ฝฝ้่ฏ | 3ๆฌก้่ฏ๏ผ5็ง้ด้ | `head_tracker.py` | โ
ๅทฒๅฎ็ฐ |
|
| 450 |
-
| ๅฏน่ฏๆจกๅผ่ๅจ | ่ฏญ้ณๅฉๆ็ถๆๅๅๆถ่ชๅจๅๆข่ฟฝ่ธช้ข็ | `satellite.py` | โ
ๅทฒๅฎ็ฐ |
|
| 451 |
-
|
| 452 |
-
**่ตๆบไผๅ (v0.5.1)**:
|
| 453 |
-
- ๅฏน่ฏๆ้ด๏ผlistening/thinking/speaking๏ผ๏ผ้ซ้ข่ฟฝ่ธช 15fps
|
| 454 |
-
- ็ฉบ้ฒไธๆฃๆตๅฐไบบ่ธ๏ผ้ซ้ข่ฟฝ่ธช 15fps
|
| 455 |
-
- ็ฉบ้ฒไธ10็งๆ ไบบ่ธ๏ผไฝๅ่ๆจกๅผ 3fps๏ผไป
ๆฃๆตๆฏๅฆๆไบบๅบ็ฐ๏ผ
|
| 456 |
-
- ๆฃๆตๅฐไบบ่ธๅ็ซๅณๆขๅค้ซ้ข่ฟฝ่ธช
|
| 457 |
-
|
| 458 |
-
**ไปฃ็ ไฝ็ฝฎ**:
|
| 459 |
-
- `head_tracker.py` - YOLO ไบบ่ธๆฃๆตๅจ (`HeadTracker` ็ฑป)
|
| 460 |
-
- `camera_server.py:_capture_frames()` - ่ช้ๅบๅธง็ไบบ่ธ่ฟฝ่ธช
|
| 461 |
-
- `camera_server.py:set_conversation_mode()` - ๅฏน่ฏๆจกๅผๅๆข API
|
| 462 |
-
- `satellite.py:_set_conversation_mode()` - ่ฏญ้ณๅฉๆ็ถๆ่ๅจ
|
| 463 |
-
- `movement_manager.py:set_face_tracking_offsets()` - ไบบ่ธ่ฟฝ่ธชๅ็งป้ API
|
| 464 |
-
|
| 465 |
-
**ๆๆฏ็ป่**:
|
| 466 |
-
```python
|
| 467 |
-
# camera_server.py - ่ช้ๅบๅธง็ไบบ่ธ่ฟฝ่ธช
|
| 468 |
-
class MJPEGCameraServer:
|
| 469 |
-
def __init__(self):
|
| 470 |
-
self._fps_high = 15 # ๅฏน่ฏ/ๆไบบ่ธๆถ
|
| 471 |
-
self._fps_low = 3 # ็ฉบ้ฒๆ ไบบ่ธๆถ
|
| 472 |
-
self._low_power_threshold = 10.0 # 10็งๆ ไบบ่ธๅๆขไฝๅ่
|
| 473 |
-
|
| 474 |
-
def _should_run_face_tracking(self, current_time):
|
| 475 |
-
# ๅฏน่ฏๆจกๅผ๏ผๅง็ป้ซ้ข่ฟฝ่ธช
|
| 476 |
-
if self._in_conversation:
|
| 477 |
-
return True
|
| 478 |
-
# ้ซ้ขๆจกๅผ๏ผๆฏๅธง่ฟฝ่ธช
|
| 479 |
-
if self._current_fps == self._fps_high:
|
| 480 |
-
return True
|
| 481 |
-
# ไฝๅ่ๆจกๅผ๏ผๅจๆๆงๆฃๆต
|
| 482 |
-
return time.since_last_check >= 1/self._fps_low
|
| 483 |
-
|
| 484 |
-
# satellite.py - ่ฏญ้ณๅฉๆ็ถๆ่ๅจ
|
| 485 |
-
def _reachy_on_listening(self):
|
| 486 |
-
self._set_conversation_mode(True) # ๅผๅงๅฏน่ฏ๏ผ้ซ้ข่ฟฝ่ธช
|
| 487 |
-
|
| 488 |
-
def _reachy_on_idle(self):
|
| 489 |
-
self._set_conversation_mode(False) # ็ปๆๅฏน่ฏ๏ผ่ช้ๅบ่ฟฝ่ธช
|
| 490 |
-
```
|
| 491 |
-
|
| 492 |
-
### Phase 16 - ๅก้้ฃๆ ผ่ฟๅจๆจกๅผ (้จๅๅฎ็ฐ) ๐ก
|
| 493 |
-
|
| 494 |
-
**็ฎๆ **: ไฝฟ็จ SDK ็ๆๅผๆๆฏ่ฎฉๆบๅจไบบๅจไฝๆดๆไธชๆงๅ่กจ็ฐๅใ
|
| 495 |
-
|
| 496 |
-
**SDK ๆฏๆ**: `InterpolationTechnique` ๆไธพ
|
| 497 |
-
- `LINEAR` - ็บฟๆง๏ผๆบๆขฐๆ
|
| 498 |
-
- `MIN_JERK` - ๆๅฐๅ ๅ ้ๅบฆ๏ผ่ช็ถๅนณๆป๏ผ้ป่ฎค๏ผ
|
| 499 |
-
- `EASE_IN_OUT` - ็ผๅ
ฅ็ผๅบ๏ผไผ้
|
| 500 |
-
- `CARTOON` - ๅก้้ฃๆ ผ๏ผๅธฆๅๅผนๆๆ๏ผๆดปๆณผๅฏ็ฑ
|
| 501 |
-
|
| 502 |
-
**ๅทฒๅฎ็ฐๅ่ฝ**:
|
| 503 |
-
- โ
20Hz ็ปไธๆงๅถๅพช็ฏ (`movement_manager.py`) - ไป 100Hz ้ไฝไปฅ้ฒๆญข daemon ๅดฉๆบ
|
| 504 |
-
- โ
ๅงฟๆๅๅๆฃๆต - ไป
ๅจๅงฟๆๆพ่ๅๅๆถๅ้ๅฝไปค (้ๅผ 0.001)
|
| 505 |
-
- โ
็ถๆๆฅ่ฏข็ผๅญ - 100ms TTL๏ผๅๅฐ daemon ่ด่ฝฝ
|
| 506 |
-
- โ
ๅนณๆปๆๅผๅจไฝ (ease in-out ๆฒ็บฟ)
|
| 507 |
-
- โ
ๅผๅธๅจ็ป - ็ฉบ้ฒๆถ Z ่ฝดๅพฎๅจ + ๅคฉ็บฟๆๅจ (`BreathingAnimation`)
|
| 508 |
-
- โ
ๅฝไปค้ๅๆจกๅผ - ็บฟ็จๅฎๅ
จ็ๅค้จ API
|
| 509 |
-
- โ
้่ฏฏ่ๆต - ้ฒๆญขๆฅๅฟ็็ธ
|
| 510 |
-
- โ
่ฟๆฅๅฅๅบท็ๆง - ่ชๅจๆฃๆตๅๆขๅค่ฟๆฅไธขๅคฑ
|
| 511 |
-
|
| 512 |
-
**ๆชๅฎ็ฐๅ่ฝ**:
|
| 513 |
-
- โ ๅจๆๆๅผๆๆฏๅๆข (CARTOON/EASE_IN_OUT ็ญ)
|
| 514 |
-
- โ ๅคธๅผ ็ๅก้ๅๅผนๆๆ
|
| 515 |
-
|
| 516 |
-
**ไปฃ็ ไฝ็ฝฎ**:
|
| 517 |
-
- `movement_manager.py:192-243` - BreathingAnimation ็ฑป
|
| 518 |
-
- `movement_manager.py:246-697` - MovementManager ็ฑป
|
| 519 |
-
|
| 520 |
-
**ๅบๆฏๅฎ็ฐ็ถๆ**:
|
| 521 |
-
|
| 522 |
-
| ๅบๆฏ | ๆจ่ๆๅผ | ๆๆ | ๅฎ็ฐ็ถๆ |
|
| 523 |
-
|------|---------|------|---------|
|
| 524 |
-
| ๅค้็นๅคด | `CARTOON` | ๆดปๆณผ็ๅๅผนๆๆ | โ ๆชๅฎ็ฐ |
|
| 525 |
-
| ๆ่ๆฌๅคด | `EASE_IN_OUT` | ไผ้
็่ฟๆธก | โ
ๅทฒๅฎ็ฐ (ๅนณๆปๆๅผ) |
|
| 526 |
-
| ่ฏด่ฏๆถๅพฎๅจ | `MIN_JERK` | ่ช็ถๆต็
| โ
ๅทฒๅฎ็ฐ (SpeechSway) |
|
| 527 |
-
| ้่ฏฏๆๅคด | `CARTOON` | ๅคธๅผ ็ๅฆๅฎ | โ ๆชๅฎ็ฐ |
|
| 528 |
-
| ่ฟๅไธญ็ซ | `MIN_JERK` | ๅนณๆปๅฝไฝ | โ
ๅทฒๅฎ็ฐ |
|
| 529 |
-
| ็ฉบ้ฒๅผๅธ | - | ๅพฎๅฆ็็ๅฝๆ | โ
ๅทฒๅฎ็ฐ (BreathingAnimation) |
|
| 530 |
-
|
| 531 |
-
### Phase 17 - ่ฏด่ฏๆถๅคฉ็บฟๅๆญฅๅจ็ป (้จๅๅฎ็ฐ) ๐ก
|
| 532 |
-
|
| 533 |
-
**็ฎๆ **: TTS ๆญๆพๆถ๏ผๅคฉ็บฟ้้ณ้ข่ๅฅๆๅจ๏ผๆจกๆ"่ฏด่ฏ"ๆๆใ
|
| 534 |
-
|
| 535 |
-
**ๅทฒๅฎ็ฐๅ่ฝ**:
|
| 536 |
-
- โ
่ฏญ้ณ้ฉฑๅจๅคด้จๆๅจ (`SpeechSwayGenerator`)
|
| 537 |
-
- โ
ๅบไบ้ณ้ขๅๅบฆ็ VAD ๆฃๆต
|
| 538 |
-
- โ
ๅค้ข็ๆญฃๅผฆๆณขๅ ๅ (Lissajous ่ฟๅจ)
|
| 539 |
-
- โ
ๅนณๆปๅ
็ป่ฟๆธก
|
| 540 |
-
|
| 541 |
-
**ไปฃ็ ไฝ็ฝฎ**:
|
| 542 |
-
- `movement_manager.py:124-189` - SpeechSwayGenerator ็ฑป
|
| 543 |
-
- `motion.py:212-222` - update_audio_loudness() ๆนๆณ
|
| 544 |
-
|
| 545 |
-
**ๆๆฏ็ป่**:
|
| 546 |
-
```python
|
| 547 |
-
# ่ฏญ้ณๆๅจๅๆฐ
|
| 548 |
-
SWAY_A_PITCH_DEG = 3.0 # ไฟฏไปฐๅน
ๅบฆ (ๅบฆ)
|
| 549 |
-
SWAY_A_YAW_DEG = 2.0 # ๅ่ชๅน
ๅบฆ
|
| 550 |
-
SWAY_A_ROLL_DEG = 2.0 # ็ฟปๆปๅน
ๅบฆ
|
| 551 |
-
SWAY_F_PITCH = 0.8 # ไฟฏไปฐ้ข็ Hz
|
| 552 |
-
SWAY_F_YAW = 0.6 # ๅ่ช้ข็
|
| 553 |
-
SWAY_F_ROLL = 0.5 # ็ฟปๆป้ข็
|
| 554 |
-
|
| 555 |
-
# VAD ้ๅผ
|
| 556 |
-
VAD_DB_ON = -35 # ๅผๅงๆฃๆต้ๅผ
|
| 557 |
-
VAD_DB_OFF = -45 # ๅๆญขๆฃๆต้ๅผ
|
| 558 |
-
```
|
| 559 |
-
|
| 560 |
-
**ๆชๅฎ็ฐๅ่ฝ**:
|
| 561 |
-
- โ ๅคฉ็บฟ้้ณ้ข่ๅฅๆๅจ (ๅฝๅไป
ๅคด้จๆๅจ)
|
| 562 |
-
- โ ้ณ้ข้ข่ฐฑๅๆ้ฉฑๅจๅจ็ป
|
| 563 |
-
|
| 564 |
-
### Phase 18 - ่ง่งๆณจ่งไบคไบ (ๆชๅฎ็ฐ) โ
|
| 565 |
-
|
| 566 |
-
**็ฎๆ **: ๅฉ็จๆๅๅคดๆฃๆตไบบ่ธ๏ผๅฎ็ฐ็ผ็ฅไบคๆตใ
|
| 567 |
-
|
| 568 |
-
**SDK ๆฏๆ**:
|
| 569 |
-
- `look_at_image(u, v)` - ๆณจ่งๅพๅไธญ็็น
|
| 570 |
-
- `look_at_world(x, y, z)` - ๆณจ่งไธ็ๅๆ ็น
|
| 571 |
-
- `media.get_frame()` - ่ทๅๆๅๅคด็ป้ข (โ
ๅทฒๅจ `camera_server.py:146` ๅฎ็ฐ)
|
| 572 |
-
|
| 573 |
-
**ๆชๅฎ็ฐๅ่ฝ**:
|
| 574 |
-
|
| 575 |
-
| ๅ่ฝ | ่ฏดๆ | ๅฎ็ฐ็ถๆ |
|
| 576 |
-
|------|------|---------|
|
| 577 |
-
| ไบบ่ธๆฃๆต | ไฝฟ็จ OpenCV/MediaPipe ๆฃๆตไบบ่ธ | โ ๆชๅฎ็ฐ |
|
| 578 |
-
| ็ผ็ฅ่ฟฝ่ธช | ๅฏน่ฏๆถๆณจ่ง่ฏด่ฏไบบ็่ธ | โ ๆชๅฎ็ฐ |
|
| 579 |
-
| ๅคไบบๅๆข | ๆฃๆตๅฐๅคไบบๆถ๏ผๆณจ่งๅฝๅ่ฏด่ฏไบบ | โ ๆชๅฎ็ฐ |
|
| 580 |
-
| ็ฉบ้ฒๆซ่ง | ็ฉบ้ฒๆถ้ๆบ็ฏ้กพๅๅจ | โ ๆชๅฎ็ฐ |
|
| 581 |
-
|
| 582 |
-
### Phase 19 - ้ๅ่กฅๅฟไบๅจๆจกๅผ (้จๅๅฎ็ฐ) ๐ก
|
| 583 |
-
|
| 584 |
-
**็ฎๆ **: ๅ
่ฎธ็จๆท็ฉ็่งฆๆธๅๅผๅฏผๆบๅจไบบๅคด้จ๏ผๅฎ็ฐ"ๆๅญฆ"ๅผไบคไบใ
|
| 585 |
-
|
| 586 |
-
**SDK ๆฏๆ**: `enable_gravity_compensation()` - ็ตๆบ่ฟๅ
ฅ้ๅ่กฅๅฟๆจกๅผ๏ผๅฏๆๅจ็งปๅจ
|
| 587 |
-
|
| 588 |
-
**ๅทฒๅฎ็ฐๅ่ฝ**:
|
| 589 |
-
- โ
้ๅ่กฅๅฟๆจกๅผๅๆข (`motor_mode` Select ๅฎไฝ๏ผ้้กน "gravity_compensation")
|
| 590 |
-
- โ
`reachy_controller.py:236-237` - ้ๅ่กฅๅฟ API ่ฐ็จ
|
| 591 |
-
|
| 592 |
-
**ๆชๅฎ็ฐๅ่ฝ**:
|
| 593 |
-
- โ ๆๅญฆๆจกๅผ - ๅฝๅถๅจไฝ่ฝจ่ฟน
|
| 594 |
-
- โ ไฟๅญ/ๆญๆพ่ชๅฎไนๅจไฝ
|
| 595 |
-
- โ ่ฏญ้ณๅฝไปค่งฆๅๆๅญฆๆต็จ
|
| 596 |
-
|
| 597 |
-
**ๅบ็จๅบๆฏ**:
|
| 598 |
-
- โ ็จๆท่ฏด "่ฎฉๆๆไฝ ไธไธชๅจไฝ" โ ่ฟๅ
ฅ้ๅ่กฅๅฟๆจกๅผ
|
| 599 |
-
- โ ็จๆทๆๅจ็งปๅจๅคด้จ โ ๅฝๅถๅจไฝ่ฝจ่ฟน
|
| 600 |
-
- โ ็จๆท่ฏด "่ฎฐไฝ่ฟไธช" โ ไฟๅญๅจไฝ
|
| 601 |
-
- โ ็จๆท่ฏด "ๅๅๆ็ๅจไฝ" โ ๆญๆพๅฝๅถ็ๅจไฝ
|
| 602 |
-
|
| 603 |
-
### Phase 20 - ็ฏๅขๆ็ฅๅๅบ (้จๅๅฎ็ฐ) ๐ก
|
| 604 |
-
|
| 605 |
-
**็ฎๆ **: ๅฉ็จ IMU ไผ ๆๅจๆ็ฅ็ฏๅขๅๅๅนถๅๅบๅๅบใ
|
| 606 |
-
|
| 607 |
-
**SDK ๆฏๆ**:
|
| 608 |
-
- โ
`mini.imu["accelerometer"]` - ๅ ้ๅบฆ่ฎก (Phase 7 ๅทฒๅฎ็ฐไธบๅฎไฝ)
|
| 609 |
-
- โ
`mini.imu["gyroscope"]` - ้่บไปช (Phase 7 ๅทฒๅฎ็ฐไธบๅฎไฝ)
|
| 610 |
-
|
| 611 |
-
**ๅทฒๅฎ็ฐๅ่ฝ**:
|
| 612 |
-
|
| 613 |
-
| ๆฃๆตไบไปถ | ๅๅบๅจไฝ | ๅฎ็ฐ็ถๆ |
|
| 614 |
-
|---------|---------|---------|
|
| 615 |
-
| ๆไธๆๅค้ | ่ฟๅ
ฅๆ็ปญๅฏน่ฏๆจกๅผ | โ
ๅทฒๅฎ็ฐ |
|
| 616 |
-
| ๅๆฌกๆไธๆ | ้ๅบๆ็ปญๅฏน่ฏๆจกๅผ | โ
ๅทฒๅฎ็ฐ |
|
| 617 |
-
|
| 618 |
-
**ๆไธๆๅค้ vs ่ฏญ้ณๅค้**:
|
| 619 |
-
|
| 620 |
-
| ๅค้ๆนๅผ | ๅฏน่ฏๆจกๅผ | ่ฏดๆ |
|
| 621 |
-
|---------|---------|------|
|
| 622 |
-
| ่ฏญ้ณๅค้ (Okay Nabu) | ๅๆฌกๅฏน่ฏ | ๆฏๆฌกๅฏน่ฏ้่ฆ้ๆฐ่ฏดๅค้่ฏ |
|
| 623 |
-
| ๆไธๆๅค้ | ๆ็ปญๅฏน่ฏ | TTS ็ปๆๅ่ชๅจ็ปง็ปญ็ๅฌ๏ผๅๆไธๆฌก้ๅบ |
|
| 624 |
-
|
| 625 |
-
**ๆๆฏๅฎ็ฐ**:
|
| 626 |
-
- `tap_detector.py` - IMU ๅ ้ๅบฆ็ชๅๆฃๆต
|
| 627 |
-
- `satellite.py:_tap_conversation_mode` - ๆ็ปญๅฏน่ฏๆจกๅผๆ ๅฟ
|
| 628 |
-
- ้ๅผ: 2.0g (ๅฏ้
็ฝฎ)
|
| 629 |
-
- ๅทๅดๆถ้ด: 1.0s (้ฒๆญข้ๅค่งฆๅ)
|
| 630 |
-
- ไป
้ๆ ็บฟ็ๆฌ (Wireless) ๅฏ็จ
|
| 631 |
-
|
| 632 |
-
```python
|
| 633 |
-
# satellite.py - ๆ็ปญๅฏน่ฏๆจกๅผ
|
| 634 |
-
def wakeup_from_tap(self):
|
| 635 |
-
if self._tap_conversation_mode:
|
| 636 |
-
# ็ฌฌไบๆฌกๆ - ้ๅบๆ็ปญๅฏน่ฏ
|
| 637 |
-
self._tap_conversation_mode = False
|
| 638 |
-
self._reachy_on_idle()
|
| 639 |
-
else:
|
| 640 |
-
# ็ฌฌไธๆฌกๆ - ่ฟๅ
ฅๆ็ปญๅฏน่ฏ
|
| 641 |
-
self._tap_conversation_mode = True
|
| 642 |
-
self.send_messages([VoiceAssistantRequest(start=True)])
|
| 643 |
-
|
| 644 |
-
def _tts_finished(self):
|
| 645 |
-
if self._tap_conversation_mode:
|
| 646 |
-
# ๆ็ปญๅฏน่ฏๆจกๅผ๏ผ่ชๅจ็ปง็ปญ็ๅฌ
|
| 647 |
-
self.send_messages([VoiceAssistantRequest(start=True)])
|
| 648 |
-
```
|
| 649 |
-
|
| 650 |
-
**ๆชๅฎ็ฐๅ่ฝ**:
|
| 651 |
-
|
| 652 |
-
| ๆฃๆตไบไปถ | ๅๅบๅจไฝ | ๅฎ็ฐ็ถๆ |
|
| 653 |
-
|---------|---------|---------|
|
| 654 |
-
| ่ขซๆๆ | ๆญๆพๆ็ฉๅจไฝ + ่ฏญ้ณ "ๅซๆๆ~" | โ ๆชๅฎ็ฐ |
|
| 655 |
-
| ๅพๆ/ๅไธ | ๆญๆพๆฑๅฉๅจไฝ + ่ฏญ้ณ "ๆๅไบ๏ผๅธฎๅธฎๆ" | โ ๆชๅฎ็ฐ |
|
| 656 |
-
| ้ฟๆถ้ด้ๆญข | ่ฟๅ
ฅไผ็ ๅจ็ป | โ ๆชๅฎ็ฐ |
|
| 657 |
-
|
| 658 |
-
### Phase 21 - Home Assistant ๅบๆฏ่ๅจ (ๆชๅฎ็ฐ) โ
|
| 659 |
-
|
| 660 |
-
**็ฎๆ **: ๆ นๆฎ Home Assistant ็ๅบๆฏ/่ชๅจๅ่งฆๅๆบๅจไบบๅจไฝใ
|
| 661 |
-
|
| 662 |
-
**ๅฎ็ฐๆนๆก**: ้่ฟ ESPHome ๆๅก่ฐ็จ
|
| 663 |
-
|
| 664 |
-
**ๆชๅฎ็ฐๅบๆฏ**:
|
| 665 |
-
|
| 666 |
-
| HA ๅบๆฏ | ๆบๅจไบบๅๅบ | ๅฎ็ฐ็ถๆ |
|
| 667 |
-
|--------|-----------|---------|
|
| 668 |
-
| ๆฉๅฎๅบๆฏ | ๆญๆพๅค้ๅจไฝ + "ๆฉไธๅฅฝ!" | โ ๆชๅฎ็ฐ |
|
| 669 |
-
| ๆๅฎๅบๆฏ | ๆญๆพ็ก็ ๅจไฝ + "ๆๅฎ~" | โ ๆชๅฎ็ฐ |
|
| 670 |
-
| ๆไบบๅๅฎถ | ่ฝฌๅ้จๅฃ + ๆฅๆ + "ๆฌข่ฟๅๅฎถ!" | โ ๆชๅฎ็ฐ |
|
| 671 |
-
| ้จ้ๅ่ตท | ่ฝฌๅ้จๅฃ + ่ญฆ่งๅจไฝ | โ ๆชๅฎ็ฐ |
|
| 672 |
-
| ๆญๆพ้ณไน | ้้ณไน่ๅฅๆๅจ | โ ๆชๅฎ็ฐ |
|
| 673 |
-
|
| 674 |
-
---
|
| 675 |
-
|
| 676 |
-
## ๐ ๅ๏ฟฝ๏ฟฝๅฎ็ฐๆป็ป
|
| 677 |
|
| 678 |
-
|
| 679 |
-
|
| 680 |
-
|
| 681 |
-
|
| 682 |
-
- **ๅบ็ก่ฏญ้ณไบคไบ** - ๅค้่ฏๆฃๆตใSTT/TTS ้ๆ
|
| 683 |
-
- **่ฟๅจๅ้ฆ** - ็นๅคดใๆๅคดใๆณจ่ง็ญๅบ็กๅจไฝ
|
| 684 |
-
- **้ณ้ขๅค็** - AGCใๅชๅฃฐๆๅถใๅๅฃฐๆถ้ค
|
| 685 |
-
- **ๆๅๅคดๆต** - MJPEG ๅฎๆถ้ข่ง
|
| 686 |
-
|
| 687 |
-
#### ้จๅๅฎ็ฐๅ่ฝ (Phase 14-21)
|
| 688 |
-
- **Phase 14** - ๆ
ๆๅจไฝ API ๅบ็ก่ฎพๆฝ (ๆๅจ่งฆๅๅฏ็จ)
|
| 689 |
-
- **Phase 19** - ้ๅ่กฅๅฟๆจกๅผๅๆข (ๆๅญฆๆต็จๆชๅฎ็ฐ)
|
| 690 |
-
|
| 691 |
-
### โ ๆชๅฎ็ฐๅ่ฝ
|
| 692 |
-
|
| 693 |
-
#### ้ซไผๅ
็บง
|
| 694 |
-
- ~~**Phase 13** - Sendspin ้ณ้ขๆญๆพๆฏๆ~~ โ
**ๅทฒๅฎๆ**
|
| 695 |
-
- **Phase 14** - ่ชๅจๆ
ๆๅจไฝๅ้ฆ (้ไธ่ฏญ้ณๅฉๆไบไปถๅ
ณ่)
|
| 696 |
-
- **Phase 15** - ๆ็ปญๅฃฐๆบ่ฟฝ่ธช (ไป
ๅค้ๆถ่ฝฌๅ)
|
| 697 |
-
|
| 698 |
-
#### ไธญไผๅ
็บง
|
| 699 |
-
- **Phase 16** - ๅก้้ฃๆ ผ่ฟๅจๆจกๅผ (้ๅจๆๆๅผๅๆข)
|
| 700 |
-
- **Phase 17** - ๅคฉ็บฟๅๆญฅๅจ็ป
|
| 701 |
-
- **Phase 18** - ไบบ่ธ่ฟฝ่ธชไธ็ผ็ฅไบคไบ
|
| 702 |
-
|
| 703 |
-
#### ไฝไผๅ
็บง
|
| 704 |
-
- **Phase 19** - ๆๅญฆๆจกๅผๅฝๅถ/ๆญๆพๅ่ฝ
|
| 705 |
-
- **Phase 20** - IMU ็ฏๅขๆ็ฅๅๅบ
|
| 706 |
-
- **Phase 21** - Home Assistant ๅบๆฏ่ๅจ
|
| 707 |
-
|
| 708 |
-
---
|
| 709 |
-
|
| 710 |
-
## ๅ่ฝไผๅ
็บงๆป็ป (ๆดๆฐ็)
|
| 711 |
-
|
| 712 |
-
### ้ซไผๅ
็บง (ๅทฒๅฎๆ โ
)
|
| 713 |
-
- โ
**Phase 1-12**: ๅบ็ก ESPHome ๅฎไฝ (45+ ไธช)
|
| 714 |
-
- โ
ๆ ธๅฟ่ฏญ้ณๅฉๆๅ่ฝ
|
| 715 |
-
- โ
ๅบ็ก่ฟๅจๅ้ฆ (็นๅคดใๆๅคดใๆณจ่ง)
|
| 716 |
-
|
| 717 |
-
### ้ซไผๅ
็บง (้จๅๅฎ็ฐ ๐ก)
|
| 718 |
-
- ๐ก **Phase 13**: ๆ
ๆๅจไฝๅ้ฆ็ณป็ป
|
| 719 |
-
- โ
Emotion Selector ๅฎไฝไธ API ๅบ็ก่ฎพๆฝ
|
| 720 |
-
- โ ่ชๅจๆ นๆฎ่ฏญ้ณๅฉๆๅๅบ่งฆๅๆ
ๆๅจไฝ
|
| 721 |
-
- โ ๆๅพ่ฏๅซไธๆ
ๆๅน้
|
| 722 |
-
- โ ่่นๅจไฝๅบ้ๆ
|
| 723 |
-
|
| 724 |
-
### ้ซไผๅ
็บง (ๆชๅฎ็ฐ โ)
|
| 725 |
-
- โ **Phase 14**: ๆบ่ฝๅฃฐๆบ่ฟฝ่ธชๅขๅผบ
|
| 726 |
-
- โ
ๅค้ๆถ่ฝฌๅๅฃฐๆบ
|
| 727 |
-
- โ ๆ็ปญๅฃฐๆบ่ฟฝ่ธช
|
| 728 |
-
- โ ๅคไบบๅฏน่ฏๅๆข
|
| 729 |
-
- โ ๅฃฐๆบๅฏ่งๅ
|
| 730 |
-
|
| 731 |
-
### ไธญไผๅ
็บง (้จๅๅฎ็ฐ ๐ก)
|
| 732 |
-
- ๐ก **Phase 15**: ๅก้้ฃๆ ผ่ฟๅจๆจกๅผ
|
| 733 |
-
- โ
20Hz ็ปไธๆงๅถๅพช็ฏๆถๆ (ไผๅไปฅ้ฒๆญข daemon ๅดฉๆบ)
|
| 734 |
-
- โ
ๅงฟๆๅๅๆฃๆต + ็ถๆๆฅ่ฏข็ผๅญ (ๅๅฐ daemon ่ด่ฝฝ)
|
| 735 |
-
- โ
ๅนณๆปๆๅผๅจไฝ + ๅผๅธๅจ็ป
|
| 736 |
-
- โ ๅจๆๆๅผๆๆฏๅๆข (CARTOON ็ญ)
|
| 737 |
-
- ๐ก **Phase 16**: ่ฏด่ฏๆถๅคฉ็บฟๅๆญฅ
|
| 738 |
-
- โ
่ฏญ้ณ้ฉฑๅจๅคด้จๆๅจ (SpeechSwayGenerator)
|
| 739 |
-
- โ ๅคฉ็บฟ้้ณ้ข่ๅฅๆๅจ
|
| 740 |
-
|
| 741 |
-
### ไธญไผๅ
็บง (ๆชๅฎ็ฐ โ)
|
| 742 |
-
- โ **Phase 17**: ่ง่งๆณจ่งไบคไบ - ็ผ็ฅไบคๆต
|
| 743 |
-
|
| 744 |
-
### ไฝไผๅ
็บง (้จๅๅฎ็ฐ ๐ก)
|
| 745 |
-
- ๐ก **Phase 18**: ้ๅ่กฅๅฟไบๅจๆจกๅผ
|
| 746 |
-
- โ
้ๅ่กฅๅฟๆจกๅผๅๆข
|
| 747 |
-
- โ ๆๅญฆๅผไบคไบ (ๅฝๅถ/ๆญๆพๅ่ฝ)
|
| 748 |
-
|
| 749 |
-
### ไฝไผๅ
็บง (ๆชๅฎ็ฐ โ)
|
| 750 |
-
- โ **Phase 19**: ็ฏๅขๆ็ฅๅๅบ - IMU ่งฆๅๅจไฝ
|
| 751 |
-
- โ **Phase 20**: Home Assistant ๅบๆฏ่ๅจ - ๆบ่ฝๅฎถๅฑ
ๆดๅ
|
| 752 |
|
| 753 |
---
|
| 754 |
|
| 755 |
-
##
|
| 756 |
-
|
| 757 |
-
|
| 758 |
-
|
| 759 |
-
|
| 760 |
-
|
| 761 |
-
|
| 762 |
-
|
| 763 |
-
|
| 764 |
-
|
| 765 |
-
|
| 766 |
-
|
| 767 |
-
|
|
| 768 |
-
|
| 769 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 770 |
|
| 771 |
---
|
| 772 |
|
| 773 |
-
##
|
| 774 |
-
|
| 775 |
-
### ้ฎ้ขๆ่ฟฐ
|
| 776 |
-
้ฟๆ่ฟ่ก่ฟ็จไธญ๏ผ`reachy_mini daemon` ไผๅดฉๆบ๏ผๅฏผ่ดๆบๅจไบบๅคฑๅปๅๅบใ
|
| 777 |
-
|
| 778 |
-
### ๆ นๆฌๅๅ
|
| 779 |
-
1. **100Hz ๆงๅถๅพช็ฏ่ฟไบ้ข็น** - ๆฏ 10ms ่ฐ็จไธๆฌก `robot.set_target()`๏ผๅณไฝฟๅงฟๆๆฒกๆๅๅ
|
| 780 |
-
2. **้ข็น็็ถๆๆฅ่ฏข** - ๆฏๆฌก่ฏปๅๅฎไฝ็ถๆ้ฝ่ฐ็จ `get_status()`ใ`get_current_head_pose()` ็ญ
|
| 781 |
-
3. **็ผบๅฐๅๅๆฃๆต** - ๅณไฝฟๅงฟๆๆฒกๆๅๅ๏ผไนไผๆ็ปญๅ้็ธๅ็ๅฝไปค
|
| 782 |
-
4. **Zenoh ๆถๆฏ้ๅๅ ตๅก** - ็ดฏ็งฏ่ตทๆฅๅฏ่ฝๆฏ็ง 150+ ๆกๆถๆฏ๏ผdaemon ๆ ๆณๅๆถๅค็
|
| 783 |
-
|
| 784 |
-
### ไฟฎๅคๆนๆก
|
| 785 |
|
| 786 |
-
###
|
| 787 |
-
```python
|
| 788 |
-
# ไป 100Hz ้ไฝๅฐ 20Hz
|
| 789 |
-
CONTROL_LOOP_FREQUENCY_HZ = 20 # ๅๅฐ 80% ็ๆถๆฏ้
|
| 790 |
-
```
|
| 791 |
-
|
| 792 |
-
#### 2. ๆทปๅ ๅงฟๆๅๅๆฃๆต (movement_manager.py)
|
| 793 |
-
```python
|
| 794 |
-
# ไป
ๅจๅงฟๆๆพ่ๅๅๆถๅ้ๅฝไปค
|
| 795 |
-
if self._last_sent_pose is not None:
|
| 796 |
-
max_diff = max(abs(pose[k] - self._last_sent_pose.get(k, 0.0)) for k in pose.keys())
|
| 797 |
-
if max_diff < 0.001: # ้ๅผ: 0.001 rad ๆ 0.001 m
|
| 798 |
-
return # ่ทณ่ฟๅ้
|
| 799 |
-
```
|
| 800 |
|
| 801 |
-
|
| 802 |
-
```python
|
| 803 |
-
# ็ผๅญ daemon ็ถๆๆฅ่ฏข็ปๆ
|
| 804 |
-
self._cache_ttl = 0.1 # 100ms TTL
|
| 805 |
-
self._last_status_query = 0.0
|
| 806 |
-
|
| 807 |
-
def _get_cached_status(self):
|
| 808 |
-
now = time.time()
|
| 809 |
-
if now - self._last_status_query < self._cache_ttl:
|
| 810 |
-
return self._state_cache.get('status') # ไฝฟ็จ็ผๅญ
|
| 811 |
-
# ... ๆฅ่ฏขๅนถๆดๆฐ็ผๅญ
|
| 812 |
-
```
|
| 813 |
|
| 814 |
-
|
| 815 |
-
``
|
| 816 |
-
|
| 817 |
-
|
| 818 |
-
|
| 819 |
-
|
| 820 |
|
| 821 |
-
|
|
|
|
|
|
|
|
|
|
| 822 |
|
| 823 |
-
|
| 824 |
-
|------|--------|--------|------|
|
| 825 |
-
| ๆงๅถๆถๆฏ้ข็ | ~100 msg/s | ~20 msg/s | โ 80% |
|
| 826 |
-
| ็ถๆๆฅ่ฏข้ข็ | ~50 msg/s | ~5 msg/s | โ 90% |
|
| 827 |
-
| ๆป Zenoh ๆถๆฏ | ~150 msg/s | ~25 msg/s | โ 83% |
|
| 828 |
-
| Daemon CPU ่ด่ฝฝ | ๆ็ปญ้ซ่ด่ฝฝ | ๆญฃๅธธ่ด่ฝฝ | ๆพ่้ไฝ |
|
| 829 |
-
| ้ขๆ็จณๅฎๆง | ๆฐๅฐๆถๅ
ๅดฉๆบ | ๅฏ็จณๅฎ่ฟ่กๆฐๅคฉ | ๅคงๅน
ๆๅ |
|
| 830 |
|
| 831 |
-
|
| 832 |
-
- `DAEMON_CRASH_FIX_PLAN.md` - ่ฏฆ็ปไฟฎๅคๆนๆกๅๆต่ฏ่ฎกๅ
|
| 833 |
-
- `movement_manager.py` - ๆงๅถๅพช็ฏไผๅ
|
| 834 |
-
- `reachy_controller.py` - ็ถๆๆฅ่ฏข็ผๅญ
|
| 835 |
|
| 836 |
-
|
| 837 |
-
|
| 838 |
-
|
| 839 |
-
|
| 840 |
|
| 841 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 842 |
|
| 843 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 844 |
|
| 845 |
-
###
|
| 846 |
-
้ฟๆ่ฟ่ก่ฟ็จไธญ๏ผ`reachy_mini daemon` ไป็ถไผๅดฉๆบ๏ผไนๅ็ไฟฎๅคไธๅคๅฝปๅบใ
|
| 847 |
|
| 848 |
-
|
| 849 |
|
| 850 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 851 |
|
| 852 |
-
|
| 853 |
-
|
| 854 |
-
|
| 855 |
-
- `set_target_body_yaw()` - 1 ๆกๆถๆฏ
|
| 856 |
|
| 857 |
-
|
| 858 |
-
- ่ง `reachy_mini/daemon/backend/robot/backend.py`: `control_loop_frequency = 50.0`
|
| 859 |
-
- ๅฆๆๆถๆฏๅ้้ข็่ถ
่ฟ 50Hz๏ผdaemon ๅฏ่ฝๆ ๆณๅๆถๅค็
|
| 860 |
|
| 861 |
-
|
| 862 |
-
- 20Hz ร 3 ๆถๆฏ = 60 ๆถๆฏ/็ง
|
| 863 |
-
- ๅทฒ็ป่ถ
่ฟ daemon ็ 50Hz ๅค็่ฝๅ
|
| 864 |
|
| 865 |
-
|
| 866 |
-
|
| 867 |
-
|
|
|
|
|
|
|
| 868 |
|
| 869 |
-
|
|
|
|
|
|
|
| 870 |
|
| 871 |
-
###
|
| 872 |
-
```python
|
| 873 |
-
# ไป 20Hz ้ไฝๅฐ 10Hz
|
| 874 |
-
# 10Hz ร 3 ๆถๆฏ = 30 ๆถๆฏ/็ง๏ผๅฎๅ
จไฝไบ daemon ็ 50Hz ๅฎน้
|
| 875 |
-
CONTROL_LOOP_FREQUENCY_HZ = 10
|
| 876 |
-
```
|
| 877 |
|
| 878 |
-
|
| 879 |
-
```python
|
| 880 |
-
# ไป 0.002 ๅขๅคงๅฐ 0.005
|
| 881 |
-
# 0.005 rad โ 0.29 ๅบฆ๏ผไป็ถ่ถณๅคๅนณๆป
|
| 882 |
-
self._pose_change_threshold = 0.005
|
| 883 |
-
```
|
| 884 |
|
| 885 |
-
###
|
| 886 |
-
```python
|
| 887 |
-
# ไป 15fps ้ไฝๅฐ 10fps
|
| 888 |
-
fps: int = 10
|
| 889 |
-
```
|
| 890 |
|
| 891 |
-
|
| 892 |
-
``
|
| 893 |
-
# ไป 50Hz ้ไฝๅฐ 20Hz
|
| 894 |
-
TAP_DETECTION_RATE_HZ = 20
|
| 895 |
-
```
|
| 896 |
|
| 897 |
-
|
| 898 |
-
|
| 899 |
-
|
| 900 |
-
self._cache_ttl = 2.0
|
| 901 |
-
```
|
| 902 |
|
| 903 |
-
###
|
| 904 |
|
| 905 |
-
|
| 906 |
-
|
| 907 |
-
|
| 908 |
-
| ๆๅคง Zenoh ๆถๆฏ | 60 msg/s | 30 msg/s | โ 50% |
|
| 909 |
-
| ๅฎ้
๏ฟฝ๏ฟฝๆฏ (ๆๅๅๆฃๆต) | ~40 msg/s | ~15 msg/s | โ 62% |
|
| 910 |
-
| ไบบ่ธ่ฟฝ่ธช้ข็ | 15 Hz | 10 Hz | โ 33% |
|
| 911 |
-
| IMU ่ฝฎ่ฏข้ข็ | 50 Hz | 20 Hz | โ 60% |
|
| 912 |
-
| ็ถๆ็ผๅญ TTL | 1 ็ง | 2 ็ง | โ 100% |
|
| 913 |
-
| ้ขๆ็จณๅฎๆง | ๆฐๅฐๆถๅดฉๆบ | ๅฏ็จณๅฎ่ฟ่ก | ๅคงๅน
ๆๅ |
|
| 914 |
|
| 915 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 916 |
|
| 917 |
-
|
|
|
|
|
|
|
|
|
|
| 918 |
|
| 919 |
-
###
|
| 920 |
-
- `movement_manager.py` - ๆงๅถๅพช็ฏ้ข็ๅๅงฟๆ้ๅผ
|
| 921 |
-
- `camera_server.py` - ไบบ่ธ่ฟฝ่ธช้ข็
|
| 922 |
-
- `tap_detector.py` - IMU ่ฝฎ่ฏข้ข็
|
| 923 |
-
- `reachy_controller.py` - ็ถๆ็ผๅญ TTL
|
| 924 |
|
| 925 |
---
|
| 926 |
|
| 927 |
-
##
|
| 928 |
-
|
| 929 |
-
### ้ฎ้ขๆ่ฟฐ
|
| 930 |
-
1. **ๆไธๆๅค้้ปๅก** - ๆไธๆๅค้ๅๅฏน่ฏๅ่ฝไธๆญฃๅธธ๏ผๅญๅจ้ปๅก้ฎ้ข
|
| 931 |
-
2. **้บฆๅ
้ฃ็ตๆๅบฆไฝ** - ้่ฆ้ ๅพ่ฟๆ่ฝ่ฏๅซ่ฏญ้ณ
|
| 932 |
-
|
| 933 |
-
### ๆ นๆฌๅๅ
|
| 934 |
-
1. **้ณ้ขๆญๆพ้ปๅก** - `_tap_continue_feedback()` ๅจๆ็ปญๅฏน่ฏๆจกๅผไธๆญๆพๆ็คบ้ณ๏ผ้ปๅกไบ้ณ้ขๆตๅค็
|
| 935 |
-
2. **AGC ่ฎพ็ฝฎไธไผๅ** - ReSpeaker XVF3800 ็้ป่ฎค่ฎพ็ฝฎไธ้ๅ่ฟ่ท็ฆป่ฏญ้ณ่ฏๅซ
|
| 936 |
|
| 937 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 938 |
|
| 939 |
-
|
| 940 |
-
```python
|
| 941 |
-
def _tap_continue_feedback(self) -> None:
|
| 942 |
-
"""Provide feedback when continuing conversation in tap mode.
|
| 943 |
-
|
| 944 |
-
Triggers a nod to indicate ready for next input.
|
| 945 |
-
Sound is NOT played here to avoid blocking audio streaming.
|
| 946 |
-
"""
|
| 947 |
-
# NOTE: Do NOT play sound here - it blocks audio streaming
|
| 948 |
-
if self.state.motion_enabled and self.state.motion:
|
| 949 |
-
self.state.motion.on_continue_listening()
|
| 950 |
-
```
|
| 951 |
-
|
| 952 |
-
#### 2. ๆทปๅ ๅผๅธธๅค็ๅฐ tap ๅ่ฐ (voice_assistant.py)
|
| 953 |
-
```python
|
| 954 |
-
def _on_tap_detected(self) -> None:
|
| 955 |
-
"""Callback when tap is detected on the robot.
|
| 956 |
-
|
| 957 |
-
NOTE: This is called from the tap_detector background thread.
|
| 958 |
-
"""
|
| 959 |
-
try:
|
| 960 |
-
self._state.satellite.wakeup_from_tap()
|
| 961 |
-
# ... motion feedback
|
| 962 |
-
except Exception as e:
|
| 963 |
-
_LOGGER.error("Error in tap detection callback: %s", e)
|
| 964 |
-
```
|
| 965 |
-
|
| 966 |
-
#### 3. ๅ
จ้ขไผๅ้บฆๅ
้ฃ่ฎพ็ฝฎ (voice_assistant.py) - ๆดๆฐไบ 2026-01-07
|
| 967 |
-
```python
|
| 968 |
-
def _optimize_microphone_settings(self) -> None:
|
| 969 |
-
"""Optimize ReSpeaker XVF3800 microphone settings for voice recognition."""
|
| 970 |
-
|
| 971 |
-
# ========== 1. AGC (Automatic Gain Control) Settings ==========
|
| 972 |
-
# Enable AGC for automatic volume normalization
|
| 973 |
-
respeaker.write("PP_AGCONOFF", [1])
|
| 974 |
-
|
| 975 |
-
# Increase AGC max gain for better distant speech pickup (default ~15dB -> 30dB)
|
| 976 |
-
respeaker.write("PP_AGCMAXGAIN", [30.0])
|
| 977 |
-
|
| 978 |
-
# Set AGC desired output level (default ~-25dB -> -18dB for stronger output)
|
| 979 |
-
respeaker.write("PP_AGCDESIREDLEVEL", [-18.0])
|
| 980 |
-
|
| 981 |
-
# Optimize AGC time constant for voice commands
|
| 982 |
-
respeaker.write("PP_AGCTIME", [0.5])
|
| 983 |
-
|
| 984 |
-
# ========== 2. Base Microphone Gain ==========
|
| 985 |
-
# Increase base microphone gain (default 1.0 -> 2.0)
|
| 986 |
-
respeaker.write("AUDIO_MGR_MIC_GAIN", [2.0])
|
| 987 |
-
|
| 988 |
-
# ========== 3. Noise Suppression Settings ==========
|
| 989 |
-
# Reduce noise suppression to preserve quiet speech (default ~0.5 -> 0.15)
|
| 990 |
-
respeaker.write("PP_MIN_NS", [0.15])
|
| 991 |
-
respeaker.write("PP_MIN_NN", [0.15])
|
| 992 |
-
|
| 993 |
-
# ========== 4. Echo Cancellation & High-pass Filter ==========
|
| 994 |
-
respeaker.write("PP_ECHOONOFF", [1])
|
| 995 |
-
respeaker.write("AEC_HPFONOFF", [1])
|
| 996 |
-
```
|
| 997 |
-
|
| 998 |
-
### ไฟฎๅคๆๆ
|
| 999 |
-
|
| 1000 |
-
| ๅๆฐ | ไฟฎๅคๅ | ไฟฎๅคๅ | ่ฏดๆ |
|
| 1001 |
-
|------|--------|--------|------|
|
| 1002 |
-
| ๆไธๆๆ็ปญๅฏน่ฏ | ้ปๅก | ๆญฃๅธธๅทฅไฝ | ็งป้ค้ปๅก้ณ้ขๆญๆพ |
|
| 1003 |
|
| 1004 |
---
|
| 1005 |
|
| 1006 |
-
##
|
| 1007 |
-
|
| 1008 |
-
###
|
| 1009 |
-
|
| 1010 |
-
|
| 1011 |
-
|
| 1012 |
-
**
|
| 1013 |
-
|
| 1014 |
-
|
| 1015 |
-
- `
|
| 1016 |
-
|
| 1017 |
-
|
| 1018 |
-
|
| 1019 |
-
|
| 1020 |
-
|
| 1021 |
-
|
| 1022 |
-
|
| 1023 |
-
|
| 1024 |
-
|
| 1025 |
-
|
| 1026 |
-
|
| 1027 |
-
|
| 1028 |
-
|
| 1029 |
-
|
| 1030 |
-
|
| 1031 |
-
|
| 1032 |
-
|
| 1033 |
-
|
| 1034 |
-
|
| 1035 |
-
|
| 1036 |
-
|
| 1037 |
-
|
| 1038 |
-
|
| 1039 |
-
|
| 1040 |
-
**
|
| 1041 |
-
|
| 1042 |
-
|
| 1043 |
-
|
| 1044 |
-
|
| 1045 |
-
|
| 1046 |
-
|
| 1047 |
-
|
| 1048 |
-
|
| 1049 |
-
|
| 1050 |
-
|
| 1051 |
-
|
| 1052 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1053 |
|
| 1054 |
---
|
| 1055 |
-
| ้บฆๅ
้ฃ็ตๆๅบฆ | ~30cm | ~2-3m | ๅ
จ้ขไผๅ AGC ๅๅข็ |
|
| 1056 |
-
| AGC ๅผๅ
ณ | ๅ
ณ้ญ | ๅผๅฏ | ่ชๅจ้ณ้ๅฝไธๅ |
|
| 1057 |
-
| AGC ๆๅคงๅข็ | ~15dB | 30dB | ๆๅ่ฟ่ท็ฆปๆพ้ณ |
|
| 1058 |
-
| AGC ็ฎๆ ็ตๅนณ | -25dB | -18dB | ๆดๅผบ่พๅบไฟกๅท |
|
| 1059 |
-
| ้บฆๅ
้ฃๅข็ | 1.0x | 2.0x | ๅบ็กๅข็็ฟปๅ |
|
| 1060 |
-
| ๅชๅฃฐๆๅถ | ~0.5 | 0.15 | ๅๅฐๅฏน่ฏญ้ณ็่ฏฏๆๅถ |
|
| 1061 |
-
| ๅๅฃฐๆถ้ค | ๅผๅฏ | ๅผๅฏ | ไฟๆ TTS ๆญๆพๆถ็ๆธ
ๆฐๅบฆ |
|
| 1062 |
-
| ้ซ้ๆปคๆณข | ๅ
ณ้ญ | ๅผๅฏ | ๅป้คไฝ้ขๅชๅฃฐ |
|
| 1063 |
-
|
| 1064 |
-
### XVF3800 ๅๆฐๅ่
|
| 1065 |
-
|
| 1066 |
-
| ๅๆฐๅ | ็ฑปๅ | ่ๅด | ่ฏดๆ |
|
| 1067 |
-
|--------|------|------|------|
|
| 1068 |
-
| `PP_AGCONOFF` | int32 | 0/1 | AGC ๅผๅ
ณ |
|
| 1069 |
-
| `PP_AGCMAXGAIN` | float | 0-40 dB | AGC ๆๅคงๅข็ |
|
| 1070 |
-
| `PP_AGCDESIREDLEVEL` | float | dB | AGC ็ฎๆ ่พๅบ็ตๅนณ |
|
| 1071 |
-
| `PP_AGCTIME` | float | ็ง | AGC ๆถ้ดๅธธๆฐ |
|
| 1072 |
-
| `AUDIO_MGR_MIC_GAIN` | float | 0-4.0 | ้บฆๅ
้ฃๅข็ๅๆฐ |
|
| 1073 |
-
| `PP_MIN_NS` | float | 0-1.0 | ๆๅฐๅชๅฃฐๆๅถ (่ถไฝ่ถๅฐๆๅถ) |
|
| 1074 |
-
| `PP_MIN_NN` | float | 0-1.0 | ๆๅฐๅชๅฃฐไผฐ่ฎก |
|
| 1075 |
-
| `PP_ECHOONOFF` | int32 | 0/1 | ๅๅฃฐๆถ้คๅผๅ
ณ |
|
| 1076 |
-
| `AEC_HPFONOFF` | int32 | 0/1 | ้ซ้ๆปคๆณขๅผๅ
ณ |
|
| 1077 |
-
|
| 1078 |
-
### ็ธๅ
ณๆไปถ
|
| 1079 |
-
- `satellite.py` - ็งป้ค้ปๅก็้ณ้ขๆญๆพ
|
| 1080 |
-
- `voice_assistant.py` - ๅ
จ้ข้บฆๅ
้ฃไผๅ
|
| 1081 |
-
- `reachy_controller.py` - AGC ๅฎไฝ้ป่ฎคๅผๆดๆฐ
|
| 1082 |
-
- `entity_registry.py` - AGC max gain ่ๅดๆดๆฐ (0-40dB)
|
| 1083 |
-
- `reachy_mini/src/reachy_mini/media/audio_control_utils.py` - SDK ๅ่
|
| 1084 |
|
| 1085 |
-
|
| 1086 |
-
|
| 1087 |
-
### SDK ๆฐๆฎ็ปๆๅ่
|
| 1088 |
|
| 1089 |
```python
|
| 1090 |
-
#
|
| 1091 |
class MotorControlMode(str, Enum):
|
| 1092 |
-
Enabled = "enabled" #
|
| 1093 |
-
Disabled = "disabled" #
|
| 1094 |
-
GravityCompensation = "gravity_compensation" #
|
| 1095 |
|
| 1096 |
-
# Daemon
|
| 1097 |
class DaemonState(Enum):
|
| 1098 |
NOT_INITIALIZED = "not_initialized"
|
| 1099 |
STARTING = "starting"
|
|
@@ -1102,70 +415,14 @@ class DaemonState(Enum):
|
|
| 1102 |
STOPPED = "stopped"
|
| 1103 |
ERROR = "error"
|
| 1104 |
|
| 1105 |
-
#
|
| 1106 |
-
class FullState:
|
| 1107 |
-
control_mode: MotorControlMode
|
| 1108 |
-
head_pose: XYZRPYPose # x, y, z (m), roll, pitch, yaw (rad)
|
| 1109 |
-
head_joints: list[float] # 7 ไธชๅ
ณ่่งๅบฆ
|
| 1110 |
-
body_yaw: float
|
| 1111 |
-
antennas_position: list[float] # [right, left]
|
| 1112 |
-
doa: DoAInfo # angle (rad), speech_detected (bool)
|
| 1113 |
-
|
| 1114 |
-
# IMU ๆฐๆฎ (ไป
ๆ ็บฟ็ๆฌ)
|
| 1115 |
-
imu_data = {
|
| 1116 |
-
"accelerometer": [x, y, z], # m/sยฒ
|
| 1117 |
-
"gyroscope": [x, y, z], # rad/s
|
| 1118 |
-
"quaternion": [w, x, y, z], # ๅงฟๆๅๅ
ๆฐ
|
| 1119 |
-
"temperature": float # ยฐC
|
| 1120 |
-
}
|
| 1121 |
-
|
| 1122 |
-
# ๅฎๅ
จ้ๅถ
|
| 1123 |
HEAD_PITCH_ROLL_LIMIT = [-40ยฐ, +40ยฐ]
|
| 1124 |
HEAD_YAW_LIMIT = [-180ยฐ, +180ยฐ]
|
| 1125 |
BODY_YAW_LIMIT = [-160ยฐ, +160ยฐ]
|
| 1126 |
-
YAW_DELTA_MAX = 65ยฐ #
|
| 1127 |
-
```
|
| 1128 |
-
|
| 1129 |
-
### ESPHome ๅ่ฎฎๅฎ็ฐ่ฏดๆ
|
| 1130 |
-
|
| 1131 |
-
ESPHome ๅ่ฎฎ้่ฟ protobuf ๆถๆฏไธ Home Assistant ้ไฟกใ้่ฆๅฎ็ฐไปฅไธๆถๆฏ็ฑปๅ๏ผ
|
| 1132 |
-
|
| 1133 |
-
```python
|
| 1134 |
-
from aioesphomeapi.api_pb2 import (
|
| 1135 |
-
# Number ๅฎไฝ (้ณ้/่งๅบฆๆงๅถ)
|
| 1136 |
-
ListEntitiesNumberResponse,
|
| 1137 |
-
NumberStateResponse,
|
| 1138 |
-
NumberCommandRequest,
|
| 1139 |
-
|
| 1140 |
-
# Select ๅฎไฝ (็ตๆบๆจกๅผ)
|
| 1141 |
-
ListEntitiesSelectResponse,
|
| 1142 |
-
SelectStateResponse,
|
| 1143 |
-
SelectCommandRequest,
|
| 1144 |
-
|
| 1145 |
-
# Button ๅฎไฝ (ๅค้/็ก็ )
|
| 1146 |
-
ListEntitiesButtonResponse,
|
| 1147 |
-
ButtonCommandRequest,
|
| 1148 |
-
|
| 1149 |
-
# Switch ๅฎไฝ (็ตๆบๅผๅ
ณ)
|
| 1150 |
-
ListEntitiesSwitchResponse,
|
| 1151 |
-
SwitchStateResponse,
|
| 1152 |
-
SwitchCommandRequest,
|
| 1153 |
-
|
| 1154 |
-
# Sensor ๅฎไฝ (ๆฐๅผไผ ๆๅจ)
|
| 1155 |
-
ListEntitiesSensorResponse,
|
| 1156 |
-
SensorStateResponse,
|
| 1157 |
-
|
| 1158 |
-
# Binary Sensor ๅฎไฝ (ๅธๅฐไผ ๆๅจ)
|
| 1159 |
-
ListEntitiesBinarySensorResponse,
|
| 1160 |
-
BinarySensorStateResponse,
|
| 1161 |
-
|
| 1162 |
-
# Text Sensor ๅฎไฝ (ๆๆฌไผ ๆๅจ)
|
| 1163 |
-
ListEntitiesTextSensorResponse,
|
| 1164 |
-
TextSensorStateResponse,
|
| 1165 |
-
)
|
| 1166 |
```
|
| 1167 |
|
| 1168 |
-
##
|
| 1169 |
|
| 1170 |
- [OHF-Voice/linux-voice-assistant](https://github.com/OHF-Voice/linux-voice-assistant)
|
| 1171 |
- [pollen-robotics/reachy_mini](https://github.com/pollen-robotics/reachy_mini)
|
|
|
|
| 1 |
+
# Reachy Mini Home Assistant Voice Assistant - Project Plan
|
| 2 |
|
| 3 |
+
## Project Overview
|
| 4 |
|
| 5 |
+
Integrate Home Assistant voice assistant functionality into Reachy Mini Wi-Fi robot, communicating with Home Assistant via ESPHome protocol.
|
| 6 |
|
| 7 |
+
## Local Reference Directories (DO NOT modify any files in reference directories)
|
| 8 |
+
1. [linux-voice-assistant](reference/linux-voice-assistant) - Linux-based Home Assistant voice assistant app for reference
|
| 9 |
+
2. [Reachy Mini SDK](reference/reachy_mini) - Reachy Mini SDK local directory for reference
|
| 10 |
+
3. [reachy_mini_conversation_app](reference/reachy_mini_conversation_app) - Reachy Mini conversation app for reference
|
| 11 |
+
4. [reachy-mini-desktop-app](reference/reachy-mini-desktop-app) - Reachy Mini desktop app for reference
|
| 12 |
+
5. [sendspin](reference/sendspin-cli/) - Sendspin client for reference
|
| 13 |
|
| 14 |
+
## Core Design Principles
|
| 15 |
|
| 16 |
+
1. **Zero Configuration** - Users only need to install the app, no manual configuration required
|
| 17 |
+
2. **Native Hardware** - Use robot's built-in microphone and speaker
|
| 18 |
+
3. **Home Assistant Centralized Management** - All configuration done on Home Assistant side
|
| 19 |
+
4. **Motion Feedback** - Provide head movement and antenna animation feedback during voice interaction
|
| 20 |
+
5. **Project Constraints** - Strictly follow [Reachy Mini SDK](reachy_mini) architecture design and constraints
|
| 21 |
+
6. **Code Quality** - Follow Python development standards with consistent code style, clear structure, complete comments, comprehensive documentation
|
| 22 |
+
7. **Feature Priority** - Voice conversation with Home Assistant is highest priority; other features are auxiliary and must not affect voice conversation functionality or response speed
|
| 23 |
+
8. **No LED Functions** - LEDs are hidden inside the robot; all LED control is ignored
|
| 24 |
+
9. **Preserve Functionality** - Any code modifications should optimize while preserving completed features; do not remove features to solve problems
|
| 25 |
|
| 26 |
+
## Technical Architecture
|
| 27 |
|
| 28 |
```
|
| 29 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
| 40 |
โ โ
|
| 41 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 42 |
โ โ Camera + Face Tracking (YOLO) โ โ
|
| 43 |
+
โ โ - 15Hz face detection and tracking โ โ
|
| 44 |
+
โ โ - look_at_image() calculates target pose โ โ
|
| 45 |
+
โ โ - Smooth return to neutral position after face lost โ โ
|
| 46 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 47 |
โ โ
|
| 48 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 49 |
โ โ Motion Controller (Head + Antennas) - 5Hz โ โ
|
| 50 |
โ โ - Face tracking offsets (secondary pose) โ โ
|
| 51 |
+
โ โ - Speech sway (voice-driven micro-movements) โ โ
|
| 52 |
+
โ โ - Breathing animation (idle breathing) โ โ
|
| 53 |
โ โ - on_wakeup โ on_listening โ on_speaking โ on_idle โ โ
|
| 54 |
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
| 55 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
|
|
| 65 |
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 66 |
```
|
| 67 |
|
| 68 |
+
## Completed Features
|
| 69 |
+
|
| 70 |
+
### Core Features
|
| 71 |
+
- [x] ESPHome protocol server implementation
|
| 72 |
+
- [x] mDNS service discovery (auto-discovered by Home Assistant)
|
| 73 |
+
- [x] Local wake word detection (microWakeWord)
|
| 74 |
+
- [x] Tap-to-wake (IMU acceleration detection, wireless version only)
|
| 75 |
+
- [x] Audio stream transmission to Home Assistant
|
| 76 |
+
- [x] TTS audio playback
|
| 77 |
+
- [x] Stop word detection
|
| 78 |
+
|
| 79 |
+
### Reachy Mini Integration
|
| 80 |
+
- [x] Use Reachy Mini SDK microphone input
|
| 81 |
+
- [x] Use Reachy Mini SDK speaker output
|
| 82 |
+
- [x] Head motion control (nod, shake, gaze)
|
| 83 |
+
- [x] Antenna animation control
|
| 84 |
+
- [x] Voice state feedback actions
|
| 85 |
+
- [x] YOLO face tracking (replaces DOA sound source localization)
|
| 86 |
+
- [x] 5Hz unified motion control loop
|
| 87 |
+
|
| 88 |
+
### Application Architecture
|
| 89 |
+
- [x] Compliant with Reachy Mini App architecture
|
| 90 |
+
- [x] Auto-download wake word models
|
| 91 |
+
- [x] Auto-download sound effect files
|
| 92 |
+
- [x] No .env configuration file required
|
| 93 |
+
|
| 94 |
+
## File List
|
| 95 |
|
| 96 |
```
|
| 97 |
reachy_mini_ha_voice/
|
| 98 |
โโโ reachy_mini_ha_voice/
|
| 99 |
+
โ โโโ __init__.py # Package initialization
|
| 100 |
+
โ โโโ __main__.py # Command line entry
|
| 101 |
+
โ โโโ main.py # ReachyMiniApp entry
|
| 102 |
+
โ โโโ voice_assistant.py # Voice assistant service
|
| 103 |
+
โ โโโ satellite.py # ESPHome protocol handling
|
| 104 |
+
โ โโโ audio_player.py # Audio player
|
| 105 |
+
โ โโโ camera_server.py # MJPEG camera stream server + face tracking
|
| 106 |
+
โ โโโ head_tracker.py # YOLO face detector
|
| 107 |
+
โ โโโ motion.py # Motion control (high-level API)
|
| 108 |
+
โ โโโ movement_manager.py # Unified movement manager (20Hz control loop)
|
| 109 |
+
โ โโโ models.py # Data models
|
| 110 |
+
โ โโโ entity.py # ESPHome base entity
|
| 111 |
+
โ โโโ entity_extensions.py # Extended entity types
|
| 112 |
+
โ โโโ reachy_controller.py # Reachy Mini controller wrapper
|
| 113 |
+
โ โโโ api_server.py # API server
|
| 114 |
+
โ โโโ zeroconf.py # mDNS discovery
|
| 115 |
+
โ โโโ util.py # Utility functions
|
| 116 |
+
โโโ wakewords/ # Wake word models (auto-download)
|
| 117 |
+
โโโ sounds/ # Sound effect files (auto-download)
|
| 118 |
+
โโโ pyproject.toml # Project configuration
|
| 119 |
+
โโโ README.md # Documentation
|
| 120 |
+
โโโ PROJECT_PLAN.md # Project plan
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
```
|
| 122 |
|
| 123 |
+
## Usage Flow
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
|
| 125 |
+
1. **Install App** - Install `reachy-mini-ha-voice` from Reachy Mini App Store
|
| 126 |
+
2. **Start App** - App auto-starts ESPHome server (port 6053), auto-downloads required models and sounds
|
| 127 |
+
3. **Connect Home Assistant** - Home Assistant auto-discovers device (mDNS) or manually add via Settings โ Devices & Services โ Add Integration โ ESPHome
|
| 128 |
+
4. **Use Voice Assistant** - Say "Okay Nabu" to wake, speak command, Reachy Mini provides motion feedback
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
---
|
| 131 |
|
| 132 |
+
## ESPHome Entity Implementation
|
| 133 |
+
|
| 134 |
+
### Completed Entities Summary
|
| 135 |
+
|
| 136 |
+
**Total: 43+ entities implemented**
|
| 137 |
+
- Phase 1-4: Basic controls, motor control, pose control, gaze control
|
| 138 |
+
- Phase 5-7: Audio sensors, diagnostics, IMU sensors
|
| 139 |
+
- Phase 8-12: Emotion control, microphone volume, camera, audio processing
|
| 140 |
+
- Phase 13: Sendspin audio output support
|
| 141 |
+
|
| 142 |
+
### Control Entities (Read/Write)
|
| 143 |
+
|
| 144 |
+
| Entity Type | Name | Description |
|
| 145 |
+
|-------------|------|-------------|
|
| 146 |
+
| `Number` | `speaker_volume` | Speaker volume (0-100) |
|
| 147 |
+
| `Select` | `motor_mode` | Motor mode (enabled/disabled/gravity_compensation) |
|
| 148 |
+
| `Switch` | `motors_enabled` | Motor torque switch |
|
| 149 |
+
| `Button` | `wake_up` / `go_to_sleep` | Wake/sleep robot actions |
|
| 150 |
+
| `Number` | `head_x/y/z` | Head position control (ยฑ50mm) |
|
| 151 |
+
| `Number` | `head_roll/pitch/yaw` | Head angle control |
|
| 152 |
+
| `Number` | `body_yaw` | Body yaw angle (-160ยฐ ~ +160ยฐ) |
|
| 153 |
+
| `Number` | `antenna_left/right` | Antenna angle control (ยฑ90ยฐ) |
|
| 154 |
+
| `Number` | `look_at_x/y/z` | Gaze point coordinates |
|
| 155 |
+
| `Select` | `emotion` | Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust) |
|
| 156 |
+
| `Number` | `microphone_volume` | Microphone volume (0-100%) |
|
| 157 |
+
| `Switch` | `agc_enabled` | Auto gain control switch |
|
| 158 |
+
| `Number` | `agc_max_gain` | AGC max gain (0-30 dB) |
|
| 159 |
+
| `Number` | `noise_suppression` | Noise suppression level (0-100%) |
|
| 160 |
+
| `Number` | `tap_sensitivity` | Tap detection sensitivity (0.5-4.0g) |
|
| 161 |
+
| `Switch` | `sendspin_enabled` | Sendspin switch |
|
| 162 |
+
|
| 163 |
+
### Sensor Entities (Read-only)
|
| 164 |
+
|
| 165 |
+
| Entity Type | Name | Description |
|
| 166 |
+
|-------------|------|-------------|
|
| 167 |
+
| `Text Sensor` | `daemon_state` | Daemon status |
|
| 168 |
+
| `Binary Sensor` | `backend_ready` | Backend ready status |
|
| 169 |
+
| `Text Sensor` | `error_message` | Current error message |
|
| 170 |
+
| `Sensor` | `doa_angle` | Sound source direction angle |
|
| 171 |
+
| `Binary Sensor` | `speech_detected` | Speech detection status |
|
| 172 |
+
| `Sensor` | `control_loop_frequency` | Control loop frequency (Hz) |
|
| 173 |
+
| `Text Sensor` | `sdk_version` | SDK version |
|
| 174 |
+
| `Text Sensor` | `robot_name` | Robot name |
|
| 175 |
+
| `Binary Sensor` | `wireless_version` | Wireless version flag |
|
| 176 |
+
| `Binary Sensor` | `simulation_mode` | Simulation mode flag |
|
| 177 |
+
| `Text Sensor` | `wlan_ip` | Wireless IP address |
|
| 178 |
+
| `Sensor` | `imu_accel_x/y/z` | Accelerometer (m/sยฒ) |
|
| 179 |
+
| `Sensor` | `imu_gyro_x/y/z` | Gyroscope (rad/s) |
|
| 180 |
+
| `Sensor` | `imu_temperature` | IMU temperature (ยฐC) |
|
| 181 |
+
| `Binary Sensor` | `echo_cancellation_converged` | Echo cancellation convergence status |
|
| 182 |
+
| `Camera` | `camera` | ESPHome Camera entity |
|
| 183 |
+
| `Text Sensor` | `sendspin_url` | Sendspin server URL |
|
| 184 |
+
| `Binary Sensor` | `sendspin_connected` | Sendspin connection status |
|
| 185 |
|
| 186 |
---
|
| 187 |
|
| 188 |
+
## Voice Assistant Enhancement Features
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 189 |
|
| 190 |
+
### Phase 14 - Emotion Action Feedback System ๐ก Partial
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
|
| 192 |
+
**Status**: Basic infrastructure ready, supports manual trigger, uses voice-driven natural micro-movements during conversation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 193 |
|
| 194 |
+
**Implemented**:
|
| 195 |
+
- โ
Emotion Selector entity (`emotion`)
|
| 196 |
+
- โ
Basic emotion action playback API (`_play_emotion`)
|
| 197 |
+
- โ
Emotion mapping: Happy/Sad/Angry/Fear/Surprise/Disgust
|
| 198 |
+
- โ
Integration with HuggingFace action library
|
| 199 |
+
- โ
SpeechSway system for natural head micro-movements during conversation
|
| 200 |
|
| 201 |
+
**Design Decisions**:
|
| 202 |
+
- ๐ฏ No auto-play of full emotion actions during conversation to avoid blocking
|
| 203 |
+
- ๐ฏ Use voice-driven head sway (SpeechSway) for natural motion feedback
|
| 204 |
+
- ๐ฏ Emotion actions retained as manual trigger feature via ESPHome entity
|
| 205 |
|
| 206 |
+
### Phase 15 - Face Tracking (Replaces DOA) โ
Complete
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 207 |
|
| 208 |
+
**Goal**: Implement natural face tracking so robot looks at speaker during conversation.
|
|
|
|
|
|
|
|
|
|
| 209 |
|
| 210 |
+
**Design Decision**:
|
| 211 |
+
- โ Original plan: DOA (Direction of Arrival) sound source tracking
|
| 212 |
+
- โ
Changed to: YOLO face detection - more stable and accurate
|
| 213 |
+
- Reason: DOA inaccurate at wakeup, frequent queries cause daemon crash
|
| 214 |
|
| 215 |
+
**Implemented Features**:
|
| 216 |
+
- โ
YOLO face detection using `AdamCodd/YOLOv11n-face-detection` model
|
| 217 |
+
- โ
Adaptive frame rate: 15fps during conversation, 3fps when idle without face
|
| 218 |
+
- โ
look_at_image() calculates target pose from face position
|
| 219 |
+
- โ
Smooth return to neutral position after face lost (1 second)
|
| 220 |
+
- โ
face_tracking_offsets as secondary pose overlay
|
| 221 |
+
- โ
Model download retry (3 attempts, 5s interval)
|
| 222 |
+
- โ
Conversation mode integration with voice assistant state
|
| 223 |
|
| 224 |
+
**Resource Optimization (v0.5.1)**:
|
| 225 |
+
- During conversation (listening/thinking/speaking): High-frequency tracking 15fps
|
| 226 |
+
- Idle with face detected: High-frequency tracking 15fps
|
| 227 |
+
- Idle without face for 10s: Low-power mode 3fps
|
| 228 |
+
- Immediately restore high-frequency tracking when face detected
|
| 229 |
|
| 230 |
+
### Phase 16 - Cartoon Style Motion Mode ๐ก Partial
|
|
|
|
| 231 |
|
| 232 |
+
**Goal**: Use SDK interpolation techniques for more expressive robot movements.
|
| 233 |
|
| 234 |
+
**Implemented**:
|
| 235 |
+
- โ
20Hz unified control loop (reduced from 100Hz to prevent daemon crash)
|
| 236 |
+
- โ
Pose change detection - only send commands on significant changes (threshold 0.001)
|
| 237 |
+
- โ
State query caching - 100ms TTL, reduces daemon load
|
| 238 |
+
- โ
Smooth interpolation (ease in-out curve)
|
| 239 |
+
- โ
Breathing animation - idle Z-axis micro-movement + antenna sway
|
| 240 |
+
- โ
Command queue mode - thread-safe external API
|
| 241 |
+
- โ
Error throttling - prevents log explosion
|
| 242 |
+
- โ
Connection health monitoring - auto-detect and recover from connection loss
|
| 243 |
|
| 244 |
+
**Not Implemented**:
|
| 245 |
+
- โ Dynamic interpolation technique switching (CARTOON/EASE_IN_OUT etc.)
|
| 246 |
+
- โ Exaggerated cartoon bounce effects
|
|
|
|
| 247 |
|
| 248 |
+
### Phase 17 - Antenna Sync Animation During Speech ๐ก Partial
|
|
|
|
|
|
|
| 249 |
|
| 250 |
+
**Goal**: Antennas sway with audio rhythm during TTS playback, simulating "speaking" effect.
|
|
|
|
|
|
|
| 251 |
|
| 252 |
+
**Implemented**:
|
| 253 |
+
- โ
Voice-driven head sway (`SpeechSwayGenerator`)
|
| 254 |
+
- โ
VAD detection based on audio loudness
|
| 255 |
+
- โ
Multi-frequency sine wave overlay (Lissajous motion)
|
| 256 |
+
- โ
Smooth envelope transitions
|
| 257 |
|
| 258 |
+
**Not Implemented**:
|
| 259 |
+
- โ Antenna sway with audio rhythm (currently only head sway)
|
| 260 |
+
- โ Audio spectrum analysis driven animation
|
| 261 |
|
| 262 |
+
### Phase 18 - Visual Gaze Interaction โ Not Implemented
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 263 |
|
| 264 |
+
**Goal**: Use camera to detect faces for eye contact.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 265 |
|
| 266 |
+
### Phase 19 - Gravity Compensation Interactive Mode ๐ก Partial
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
|
| 268 |
+
**Implemented**:
|
| 269 |
+
- โ
Gravity compensation mode switch (`motor_mode` Select entity)
|
|
|
|
|
|
|
|
|
|
| 270 |
|
| 271 |
+
**Not Implemented**:
|
| 272 |
+
- โ Teaching mode - record motion trajectory
|
| 273 |
+
- โ Save/playback custom actions
|
|
|
|
|
|
|
| 274 |
|
| 275 |
+
### Phase 20 - Environment Awareness Response ๐ก Partial
|
| 276 |
|
| 277 |
+
**Implemented**:
|
| 278 |
+
- โ
Tap-to-wake enters continuous conversation mode
|
| 279 |
+
- โ
Second tap exits continuous conversation mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 280 |
|
| 281 |
+
**Tap-to-wake vs Voice Wake**:
|
| 282 |
+
| Wake Method | Conversation Mode | Description |
|
| 283 |
+
|-------------|-------------------|-------------|
|
| 284 |
+
| Voice wake (Okay Nabu) | Single conversation | Need to say wake word for each conversation |
|
| 285 |
+
| Tap-to-wake | Continuous conversation | Auto-continue listening after TTS ends, tap again to exit |
|
| 286 |
|
| 287 |
+
**Not Implemented**:
|
| 288 |
+
- โ Shake detection - play dizzy action
|
| 289 |
+
- โ Tilt/fall detection - play help action
|
| 290 |
+
- โ Long idle - enter sleep animation
|
| 291 |
|
| 292 |
+
### Phase 21 - Home Assistant Scene Integration โ Not Implemented
|
|
|
|
|
|
|
|
|
|
|
|
|
| 293 |
|
| 294 |
---
|
| 295 |
|
| 296 |
+
## Completion Statistics
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 297 |
|
| 298 |
+
| Phase | Status | Completion | Notes |
|
| 299 |
+
|-------|--------|------------|-------|
|
| 300 |
+
| Phase 1-12 | โ
Complete | 100% | 40 ESPHome entities implemented (Phase 11 LED disabled) |
|
| 301 |
+
| Phase 13 | โ
Complete | 100% | Sendspin audio output support |
|
| 302 |
+
| Phase 14 | ๐ก Partial | 30% | API infrastructure ready, missing auto-trigger |
|
| 303 |
+
| Phase 15 | โ
Complete | 100% | YOLO face tracking fully implemented |
|
| 304 |
+
| Phase 16 | ๐ก Partial | 70% | Control loop + pose detection + breathing animation |
|
| 305 |
+
| Phase 17 | ๐ก Partial | 50% | Voice-driven head sway implemented |
|
| 306 |
+
| Phase 18 | โ Not done | 10% | Camera implemented, missing face detection |
|
| 307 |
+
| Phase 19 | ๐ก Partial | 40% | Mode switch implemented, missing teaching flow |
|
| 308 |
+
| Phase 20 | ๐ก Partial | 30% | Tap-to-wake implemented |
|
| 309 |
+
| Phase 21 | โ Not done | 0% | Not implemented |
|
| 310 |
|
| 311 |
+
**Overall Completion**: **Phase 1-13: 100%** | **Phase 14-21: ~45%**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
|
| 313 |
---
|
| 314 |
|
| 315 |
+
## Bug Fixes History
|
| 316 |
+
|
| 317 |
+
### v0.5.1 Bug Fixes (2026-01-08)
|
| 318 |
+
|
| 319 |
+
#### Issue 1: Music Not Resuming After Voice Conversation
|
| 320 |
+
**Problem**: Music doesn't resume after voice conversation ends.
|
| 321 |
+
**Root Cause**: Sendspin was incorrectly connected to `tts_player` instead of `music_player`.
|
| 322 |
+
**Fix**:
|
| 323 |
+
- `voice_assistant.py`: Sendspin discovery now connects to `music_player`
|
| 324 |
+
- `satellite.py`: `duck()`/`unduck()` now call `music_player.pause_sendspin()`/`resume_sendspin()`
|
| 325 |
+
|
| 326 |
+
#### Issue 2: tap_sensitivity Not Persisted
|
| 327 |
+
**Problem**: tap_sensitivity value set in ESPHome lost after restart.
|
| 328 |
+
**Fix**:
|
| 329 |
+
- `models.py`: Added `tap_sensitivity` field to `Preferences` dataclass
|
| 330 |
+
- `entity_registry.py`: Entity setter now saves to `preferences.json`
|
| 331 |
+
- Load saved value on startup
|
| 332 |
+
|
| 333 |
+
#### Issue 3: Audio Conflict During Voice Assistant Wakeup
|
| 334 |
+
**Problem**: Audio streaming (Sendspin or ESPHome audio) conflicts when voice assistant wakes up.
|
| 335 |
+
**Fix**:
|
| 336 |
+
- `audio_player.py`: Added `pause_sendspin()` and `resume_sendspin()` methods
|
| 337 |
+
- `satellite.py`: `duck()` now pauses Sendspin, `unduck()` resumes it
|
| 338 |
+
- Improved `pause()` method to actually stop audio output
|
| 339 |
+
|
| 340 |
+
#### Issue 4: AttributeError for _camera_server
|
| 341 |
+
**Problem**: `_set_conversation_mode()` referenced non-existent `_camera_server` attribute.
|
| 342 |
+
**Fix**: Changed `self._camera_server` to `self.camera_server` (removed underscore prefix)
|
| 343 |
+
|
| 344 |
+
#### Issue 5: tap_sensitivity Default Value Wrong
|
| 345 |
+
**Problem**: tap_sensitivity default was still 2.0g instead of expected 0.5g.
|
| 346 |
+
**Fix**: Use `TAP_THRESHOLD_G_DEFAULT` constant as default value
|
| 347 |
+
|
| 348 |
+
#### Issue 6: Sendspin Sample Rate Optimization
|
| 349 |
+
**Problem**: ReSpeaker hardware I/O is 16kHz (hardware limitation), but Sendspin might try higher sample rates.
|
| 350 |
+
**Fix**: Prioritize 16kHz in Sendspin supported formats list to avoid unnecessary resampling
|
| 351 |
+
|
| 352 |
+
### Daemon Crash Fix (2026-01-07)
|
| 353 |
+
|
| 354 |
+
**Problem**: `reachy_mini daemon` crashes during long-term operation.
|
| 355 |
+
|
| 356 |
+
**Root Cause Analysis**:
|
| 357 |
+
1. Each `set_target()` sends 3 Zenoh messages
|
| 358 |
+
2. Daemon control loop is 50Hz
|
| 359 |
+
3. Previous 20Hz control loop still too high (20Hz ร 3 = 60 msg/s > 50Hz capacity)
|
| 360 |
+
4. Pose change threshold too small (0.002) - almost every loop triggers `set_target()`
|
| 361 |
+
|
| 362 |
+
**Fix**:
|
| 363 |
+
- Control loop frequency: 20Hz โ 10Hz
|
| 364 |
+
- Pose change threshold: 0.002 โ 0.005
|
| 365 |
+
- Camera/face tracking frequency: 15fps โ 10fps
|
| 366 |
+
- IMU polling frequency: 50Hz โ 20Hz
|
| 367 |
+
- State cache TTL: 1s โ 2s
|
| 368 |
+
|
| 369 |
+
**Results**:
|
| 370 |
+
| Metric | Before | After | Improvement |
|
| 371 |
+
|--------|--------|-------|-------------|
|
| 372 |
+
| Control loop frequency | 20 Hz | 10 Hz | โ 50% |
|
| 373 |
+
| Max Zenoh messages | 60 msg/s | 30 msg/s | โ 50% |
|
| 374 |
+
| Expected stability | Hours before crash | Stable operation | Significant |
|
| 375 |
+
|
| 376 |
+
### Tap-to-Wake and Microphone Sensitivity Fix (2026-01-07)
|
| 377 |
+
|
| 378 |
+
**Problems**:
|
| 379 |
+
1. Tap-to-wake blocking - conversation not working properly after tap wake
|
| 380 |
+
2. Low microphone sensitivity - need to be very close for voice recognition
|
| 381 |
+
|
| 382 |
+
**Fixes**:
|
| 383 |
+
1. Removed audio playback in `_tap_continue_feedback()` to avoid blocking
|
| 384 |
+
2. Comprehensive microphone optimization:
|
| 385 |
+
- AGC enabled with max gain 30dB
|
| 386 |
+
- AGC desired level -18dB
|
| 387 |
+
- Base microphone gain 2.0x
|
| 388 |
+
- Noise suppression reduced to 0.15
|
| 389 |
+
- Echo cancellation and high-pass filter enabled
|
| 390 |
+
|
| 391 |
+
**Results**:
|
| 392 |
+
| Parameter | Before | After |
|
| 393 |
+
|-----------|--------|-------|
|
| 394 |
+
| Microphone sensitivity | ~30cm | ~2-3m |
|
| 395 |
+
| AGC max gain | ~15dB | 30dB |
|
| 396 |
+
| Noise suppression | ~0.5 | 0.15 |
|
| 397 |
|
| 398 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 399 |
|
| 400 |
+
## SDK Data Structure Reference
|
|
|
|
|
|
|
| 401 |
|
| 402 |
```python
|
| 403 |
+
# Motor control mode
|
| 404 |
class MotorControlMode(str, Enum):
|
| 405 |
+
Enabled = "enabled" # Torque on, position control
|
| 406 |
+
Disabled = "disabled" # Torque off
|
| 407 |
+
GravityCompensation = "gravity_compensation" # Gravity compensation mode
|
| 408 |
|
| 409 |
+
# Daemon state
|
| 410 |
class DaemonState(Enum):
|
| 411 |
NOT_INITIALIZED = "not_initialized"
|
| 412 |
STARTING = "starting"
|
|
|
|
| 415 |
STOPPED = "stopped"
|
| 416 |
ERROR = "error"
|
| 417 |
|
| 418 |
+
# Safety limits
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 419 |
HEAD_PITCH_ROLL_LIMIT = [-40ยฐ, +40ยฐ]
|
| 420 |
HEAD_YAW_LIMIT = [-180ยฐ, +180ยฐ]
|
| 421 |
BODY_YAW_LIMIT = [-160ยฐ, +160ยฐ]
|
| 422 |
+
YAW_DELTA_MAX = 65ยฐ # Max difference between head and body yaw
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 423 |
```
|
| 424 |
|
| 425 |
+
## Reference Projects
|
| 426 |
|
| 427 |
- [OHF-Voice/linux-voice-assistant](https://github.com/OHF-Voice/linux-voice-assistant)
|
| 428 |
- [pollen-robotics/reachy_mini](https://github.com/pollen-robotics/reachy_mini)
|