Overview
Deaf and hard-of-hearing individuals can easily miss critical sounds at home — a crying baby, a doorbell, a fire alarm. This project is an end-to-end mobile alerting system built to close that gap: it analyzes ambient sound with on-device AI and turns a detected event into a vibration + camera-flash alert the user can't miss.
System Architecture
The system has three components:
- Dinleyici (Listener / Edge Sensor) — A Flutter app that runs on a spare phone or tablet left in a shared space. It continuously records audio, converts it into a real-time mel-spectrogram, and classifies it on-device with a TensorFlow Lite model (background noise / baby cry / doorbell / fire alarm). Once the confidence score clears a threshold, it posts an alert to the backend.
- Backend — A Spring Boot (Java 17) REST API. It persists alerts to PostgreSQL, pushes notifications to the companion app through Firebase Cloud Messaging, and manages settings such as FCM token registration and temporary muting. Packaged with Docker and deployed on Render.
- Bildirilen (Reporter / Receiver device) — A Flutter companion app that runs on the hearing-impaired user's phone. It catches push notifications even while backgrounded or killed, and responds with an SOS vibration pattern plus a strobing camera flash so the alert is impossible to miss visually and haptically. It also shows an alert history and lets the user mute the system for 15 minutes / 1 hour / 24 hours.
Technical Highlights
- Sound classification runs fully on-device: raw PCM audio is turned into a mel-spectrogram from scratch in Dart (FFT + Hann window + mel filterbank) and fed to the TFLite model — no server round-trip needed for inference.
- The mel filterbank and window function are precomputed once in Python
with librosa (
gen_mel.py), base64-encoded, and embedded directly as Dart source, so the device never needs librosa itself. - A sliding-window audio queue processes continuous audio in 3-second frames without gaps.
- On the receiving end, a Firebase background message handler drives the vibration/flash alert even when the app is backgrounded or killed, with cleanup logic guaranteeing the flash is always turned back off.
- A layered backend (controller → service → repository) manages alert history and system settings.
