What I think you should do is look at Windows Server 2003 Enterprise, and the Windows Server 2003 R2 SDK.

The server has the Advanced Fast Start feature built in, and that's specifically designed for low latency streams. There's several articles about it in the SDK, and actually the built in help file for the Media Server gives you pretty much all the info you need. You can have the same low latency that you can achieve with Flash (you have to be sure to read some follow up stuff about how to encode your media). Not being all that familiar with Flash on the client side of things I don't know anything really about how to make a player that adjusts to different scenarios. The Windows Media player has the functionality built in to work with either the low latency streams, Fast Start streams and slow buffer streams. I'm not sure yet if this is supported currently in Silverlight, from my few hours of experience with 1.1 alpha it isn't.

The reason that you want the SDK and the Enterprise edition is cos only 2003 Server Enterprise supports Windows Media Server plugins. These plugins are easy to make and to debug, all in your favorite .NET language, and give you realtime control over all aspects of your server, and even individual clients (referred to as "players" I believe). I do know a little bit about Flash on the server side, and I can tell you that it does not come close to the power of the Windows Media Server on 2003 Enterprise