logo dotConferences

Processing Images using the Web Audio API

Rafael Silva at dotJS 2017

Rafael shows how the Web Audio API filters are so versatile that they can handle arrays of any kind. Since is possible to get information of the image's pixels as an array we could use it as an audio input and filter it as sound blurring it or highlighting the boundaries.


To see the slides, you can click here

More details

  • First we load an image to a canvas element
  • Then we can get the information from each image pixel as a typed array that will contain red, green, blue and alpha successively.
  • As you may know we can manipulate audio on browser using web audio api. One of the most amazing thing is too use an array as data audio input. So you can basically synthesise sounds by creating waves/curves on the array.
  • If is possible to get data information as an array and Web Audio API supports arrays as input we can use each separated color of the image as an input to the web audio API.
  • So if we apply a low pass filter to the image it'll soften the input signal, the result of this soften is a blur effect since we are reducing transitions between pixels.
  • The high pass filter will remove the low frequencies giving us only the abrupt transitions from low to high values and vice-versa. That will basically attenuate the pixels that have the same color as neighbors letting us with the high transitions or boundaries of the image.

There is an article exploring all steps in more detail: https://medium.com/statuscode/filtering-images-using-web-audio-api-276555cca6ad

The code of each example is here: https://github.com/rssilva/web-audio-image-filtering