Problem Description:
Since the breakout of COVID-19, all courses go online. However, many of us faced a
situation that, we cannot hear the professor’s speech clearly. Since most of the
computers and webcams on the market have integrated mono microphones, this kind
of mono microphone cannot do a good job of noise canceling. Therefore, we plan to
make a system that can cancel noise and/or enhance voice in real-time using a mono
microphone.
To get a noise reduction module, we can follow these steps:
1. Data Collection: Generate big dataset of synthetic noisy speech by mixing clean
speech with noise
2. Training: Feed this dataset to the Neural Network on input and the clean speech
on the output
3. Real-time Inference: Produce a mask (binary, ratio or complex) which will leave
the human voice and filter out noise using high-performance framework like
frugally-deep.
Dataset:
We will use existing datasets and mix voice and noise to create new training data, like :
1. [ Ссылка ]
2. [ Ссылка ].
To generate model with high ability of generality, we collect multiple classes of noise, which
has high probability happened in our daily life, like cafeteria.
The link is given as follow:
[ Ссылка ]: [ Ссылка ]
Proposed Method:
We will use Neural Network to do it in the frequency domain. There are few existing
solutions like RNNoise, Wavenet and Krisp. But RNNoise is suitable only for stationary
noise, Wavenet has huge model size and cannot do real-time denoising. Krisp is a
commercial software and no technical details disclosed.
Besides, the main challenge is to do audio processing using RNN in real-time. So, we
will try to limit the model size and use a framework like frugally-deep to do inference
in C++. And we may downsample audio or use narrowband signals. Latency up to
200ms will be acceptable.
We set three milestones for this project:
1. A non-real time noise cancellation model in python based on existing
architecture such as RNNoise or Wavenet.
2. A non-real time noise cancellation module in python with self-made model.
3. A real time noise cancellation model in using frugally-deep.
Since it might be difficult and not necessary to implement a real-time noise reduction
demo like a virtual sound card, we will use processing time and CPU usage to
determine if this processing is real-time or not.
