Client-side audio+video capture

05-01-2007 5:09 PM |

After a quick search through the beta docs and on the net, I could not find any information regarding whether Silverlight supports client-side audio+video capture. I currently use a custom ActiveX control and FireFox plugin to access DirectShow for Video and DirectSound for audio capture in C++/ASM. It would be useful to be able to create a cross-platform version using Silverlight.

If and when an audio capture API is available, in order to have feature parity with my existing C++/ASM code, I would need to be able to implement custom DSP code. This includes multiple audio stream mixing, FFT, DWT, FIR and IIR filters, etc. Will it be possible for C# Silverlight code to be able to call trusted+signed ActiveX code, with a similar cross-platform shared lib method for OSX+*nix? Or will C# be fast enough to run native DSP code (last we benchmarked, C# was at least ~2x slower than C++, and much slower compared to short ASM sequences (for example, batch converting int=>float float=>int). If enough DSP toolbox elements were part of Silverlight (optimized native code), it would allow for a wide range of custom DSP effects. The same idea applies to video/image processing.

John

Re: Client-side audio+video capture

05-05-2007 2:40 PM |

I have a feeling none of this will make Silverlight 1 or 1.1 unfortunately since it hasn't been mentioned yet. That being said I think MS needs to get on this ASAP because this is a huge reason why my company has used Flash for live web streaming simply because our actual broadcasting application is very simply coded and since Flash deployment market share is so large, we can ensure everyone has it.

Setting up a basic stream in Flash takes like 10 lines of code or so. This includes webcam and microphone. Very very nice. I really hope Silverlight includes this someday soon.

Re: Client-side audio+video capture

07-24-2007 7:42 PM |

I spoke with an MS rep after an onsite demo: audio+video capture is coming in the future. For the Win32 platform, we can use our AX plugin with JS and Agc. Would be nice to see DSP building blocks to allow for real-time cross-platform effects.

Re: Client-side audio+video capture

08-02-2007 4:51 AM |

This is good news. Thanks for sharing. Did the MS rep mention if this is gone be supported in the 1.1 Release?

Re: Client-side audio+video capture

08-02-2007 5:01 PM |

He did not state when the support will be available; he did state that real-time audio+video capture support is very complicated, so it will take time to get right (including cross platform). Perhaps someone from the MS team can comment here.

When writing my ActiveX control, I found that using the absolute minimum of DirectShow (video capture only), a custom clock (for the DirectShow video stream), DirectSound for audio, and direct use of the ASF compressor (via IWMWriter, including real-time indexing: this was indeed complicated (complex binary stream parsing and modification, from the spec), but MS should have this already in-house), I was able to greatly simplify the system (and have excellent control over Audio-Video sync, which can be problematic for a variety of reasons, including but not limited to buggy drivers).

Re: Client-side audio+video capture

09-06-2007 5:19 AM |

This is really good news, because client-side audio and video capture would be essential for an app we would like to create. 

Re: Client-side audio+video capture

11-02-2008 9:17 PM |

Hi synced,

                 I am trying to develop a web app. in asp.net to record audio from client and store audio respone on server. could you please provide some guidance on how to go about it.

                 Thanks a lot in advance!