[Icecast] Running Icecast2 behind Amazon Elastic Load Balancer (ELB)

[Matt Gray]

Hello,

I'm writing to see if anyone here has had any success running Icecast2
(specifically I am using 2.4) on Amazon EC2 behind an Elastic Load Balancer?

I have the ELB configured to do TCP loadbalancing rather than HTTP. I am
able to connect and to stream music via the ELB, but when I disconnect, the
listener count in Icecast does not decrease. On examining the state of the
sockets table after disconnection I see that the socket from the ELB to
Icecast is in the CLOSE_WAIT state *, and it remains in that state
indefinitely. When I connect to the Icecast server directly, connections
are established and closed as expected. I'm using Debian Wheezy in EC2.

Any suggestions or workarounds? It is my understanding that Icecast should
be closing the sockets in the CLOSE_WAIT state.

Thanks,

Matt

*
***@ip-10-0-0-60:~# netstat -atnp | grep 8000
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN
9615/icecast2
tcp 1000 0 10.0.0.60:8000 10.0.0.79:35915
ESTABLISHED 9615/icecast2 <-- This is my liquidsoap source server
tcp 1 0 10.0.0.60:8000 10.0.0.69:14665
CLOSE_WAIT 9615/icecast2 <-- This is the listener connection via ELB that
I believe should now be closed.


ps: Apologies for the email disclaimer below, I can't remove it.
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

[Thomas B. Rücker]

Hi Matt,

Post by Matt Gray
Hello,
I'm writing to see if anyone here has had any success running Icecast2
(specifically I am using 2.4) on Amazon EC2 behind an Elastic Load Balancer?

Why do you think you need a load balancer in the first place?

Icecast saturates the network before you notice any other significant
resource usage.

I'm happy to help you with your problem if there is a good reason, but
would like to first make sure that this needs to be addressed.


Cheers

Thomas

[Matt Gray]

Hi Thomas,

We need a load balancer to allow for a single URL "entry point" to multiple
Icecast servers - we intend to host multiple Icecast servers behind the
load balancer so that we can expand / contract capacity as required - we
would add servers into the load balancer pool. This also allows us to scale
beyond the number of listeners a single Icecast server can support, while
still maintaining the "illusion" of a single streaming server.

It also provides redundancy / failover capability by allowing us to balance
connections across Icecast servers in multiple AWS availability zones.

I'm aware of some "playlist based" loadbalancing techniques using .m3u or
HTTP redirects to the icecast servers, but this is not ideal for us, as a)
we need to support some legacy players (eg. set top boxes) that don't
support redirects or m3u's, and b) using ELB allows us to very easily
provision backend health checks, multi availability zone redundancy, and
auto scaling of Icecast servers.

Hope that answers your question,

Cheers,

Matt

Post by Thomas B. Rücker
Hi Matt,

Post by Matt Gray
Hello,
I'm writing to see if anyone here has had any success running Icecast2
(specifically I am using 2.4) on Amazon EC2 behind an Elastic Load Balancer?

Why do you think you need a load balancer in the first place?
Icecast saturates the network before you notice any other significant
resource usage.
I'm happy to help you with your problem if there is a good reason, but
would like to first make sure that this needs to be addressed.
Cheers
Thomas
_______________________________________________
Icecast mailing list
http://lists.xiph.org/mailman/listinfo/icecast

--
*Matt Gray*
System Administrator

*T:* [phone]
*E:* ***@7digital.com
*W:* www.7digital.com
<http://www.google.com/url?q=http%3A%2F%2Fwww.7digital.com&sa=D&sntz=1&usg=AFrqEzeLlO75mVeA-8Yi-niLjEpFokXaYA>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

[Thomas B. Rücker]

Post by Matt Gray
Hi Thomas,
We need a load balancer to allow for a single URL "entry point" to
multiple Icecast servers - we intend to host multiple Icecast servers
behind the load balancer so that we can expand / contract capacity as
required - we would add servers into the load balancer pool. This also
allows us to scale beyond the number of listeners a single Icecast
server can support, while still maintaining the "illusion" of a single
streaming server.

That is only really relevant if you expect to saturate the upstream
bandwidth of the instance running Icecast. Which nowadays is in the
10-20k listener range at the very least. Clustering through DNS also
removes that limitation.
For context, such listener numbers are only achieved by the largest
on-line radio stations.
Does an ELB have more upstream bandwidth than a single instance? Would
be one question. If not (iow >>1Gbit) you need a different approach anyway.

Post by Matt Gray
It also provides redundancy / failover capability by allowing us to
balance connections across Icecast servers in multiple AWS
availability zones.

Redundancy/failover might be worth looking at. I'm not exactly sure if
adding another SPOF is the right approach, but thankfully it's not my
decision.
Out of curiosity, is ELB able to span several AZ?

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.
I'm not sure how well it will cope with the extremely long lived
connections though. In my experience adding the complexity of a reverse
proxy decreases Icecast availability and reliability rather than improve it.

Post by Matt Gray
I'm aware of some "playlist based" loadbalancing techniques using .m3u
or HTTP redirects to the icecast servers, but this is not ideal for
us, as a) we need to support some legacy players (eg. set top boxes)
that don't support redirects or m3u's, and

I'd not go for a playlist as it relies on clients. I'd probably just use
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
distribution. One stable and fully transparent URL. That's also easier
to implement for reliable and robust availability in my opinion. Turns
out Amazon even offers it as a service: Amazon Route 53.

Post by Matt Gray
b) using ELB allows us to very easily provision backend health checks,
multi availability zone redundancy, and auto scaling of Icecast servers.

OK, that sentence ticks a row on my bingo card. ;-)
You're set on using ELB for some reason and I won't argue further with
it. I'll just warn that it's probably not ideal in both HTTP and TCP
mode. You should do significant testing to ensure things work as they do
without it in the way.
Why it doesn't close the TCP connection after the original connection is
gone is not clear to me. It might be a bug or a feature of ELB. You
should actually check AWS documentation or contact Amazon support about
that. You can chose to ignore it and the system will eventually kill
them, but you should make sure to have enough of head room for
connections available on the Icecast machine (ulimit and server config
limits).

I know I sound quite defensive about this topic, but the point I'm
trying to make is, that it adds not well known behaviour, as it's not
fully transparent, and constitutes another single point of failure. Also
there are suitable other options. (I'd even make the same point for a
plain web server)


Cheers

Thomas

[Matt Gray]

Post by Thomas B. Rücker

Post by Matt Gray
Hi Thomas,
We need a load balancer to allow for a single URL "entry point" to
multiple Icecast servers - we intend to host multiple Icecast servers
behind the load balancer so that we can expand / contract capacity as
required - we would add servers into the load balancer pool. This also
allows us to scale beyond the number of listeners a single Icecast
server can support, while still maintaining the "illusion" of a single
streaming server.

That is only really relevant if you expect to saturate the upstream
bandwidth of the instance running Icecast. Which nowadays is in the
10-20k listener range at the very least. Clustering through DNS also
removes that limitation.

Unfortunately this is not the case in EC2, as the upstream bandwidth is not
as high as one would expect hosting one's own server (the AWS decision is
out of my hands unfortunately). With an m3.medium we are able to support
about 2000 listeners per server.

Post by Thomas B. Rücker
For context, such listener numbers are only achieved by the largest
on-line radio stations.
Does an ELB have more upstream bandwidth than a single instance? Would
be one question. If not (iow >>1Gbit) you need a different approach anyway.

ELB has much greater upstream bandwidth than an of the amazon instance
sizes. It's effectively unlimited, it grows as connections increase.

Post by Thomas B. Rücker

Post by Matt Gray
It also provides redundancy / failover capability by allowing us to
balance connections across Icecast servers in multiple AWS
availability zones.

Redundancy/failover might be worth looking at. I'm not exactly sure if
adding another SPOF is the right approach, but thankfully it's not my
decision.
Out of curiosity, is ELB able to span several AZ?

ELB is not a SPOF (at least an entire Amazon region has to go down for an
ELB to completely fail), because it scales to cope with increased
connections. ELB is able to span AZs within a region.

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.

Post by Thomas B. Rücker
I'm not sure how well it will cope with the extremely long lived
connections though. In my experience adding the complexity of a reverse
proxy decreases Icecast availability and reliability rather than improve it.

Post by Matt Gray
I'm aware of some "playlist based" loadbalancing techniques using .m3u
or HTTP redirects to the icecast servers, but this is not ideal for
us, as a) we need to support some legacy players (eg. set top boxes)
that don't support redirects or m3u's, and

I'd not go for a playlist as it relies on clients. I'd probably just use
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
distribution. One stable and fully transparent URL. That's also easier
to implement for reliable and robust availability in my opinion. Turns
out Amazon even offers it as a service: Amazon Route 53.

Yes we've looked at Route 53, perhaps this is an option we will go with.

Post by Thomas B. Rücker
b) using ELB allows us to very easily provision backend health checks,

Post by Matt Gray
multi availability zone redundancy, and auto scaling of Icecast servers.

OK, that sentence ticks a row on my bingo card. ;-)
You're set on using ELB for some reason and I won't argue further with
it.

We are certainly not set on it, because of the issues that I've explained
(which might well be to do with ELB) I'm just trying to to understand the
behaviour of Icecast in that environment.

Post by Thomas B. Rücker
I'll just warn that it's probably not ideal in both HTTP and TCP
mode. You should do significant testing to ensure things work as they do
without it in the way.
Why it doesn't close the TCP connection after the original connection is
gone is not clear to me. It might be a bug or a feature of ELB. You
should actually check AWS documentation or contact Amazon support about
that.

I have also taken this up with AWS support and their stance is that sockets
in CLOSE_WAIT are waiting for the backend server to close them, and there
is nothing that the ELB can do about that.

Post by Thomas B. Rücker
You can chose to ignore it and the system will eventually kill
them, but you should make sure to have enough of head room for
connections available on the Icecast machine (ulimit and server config
limits).

Unfortunately we can't, because of the issues you mention and also we need
to use the disconnection event to log listener usage etc.

Post by Thomas B. Rücker
I know I sound quite defensive about this topic, but the point I'm
trying to make is, that it adds not well known behaviour, as it's not
fully transparent, and constitutes another single point of failure. Also
there are suitable other options. (I'd even make the same point for a
plain web server)

You've every right to be defensive :) I'm just trying to understand what
causes the CLOSE_WAIT states. You are right, there are plenty of other
options.

In general, what are people using to provide a high availability icecast
service?

Thanks for your time putting together such a comprehensive response.
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

[Thomas B. Rücker]

Post by Matt Gray
Hi Thomas,
We need a load balancer to allow for a single URL "entry point" to
multiple Icecast servers - we intend to host multiple Icecast servers
behind the load balancer so that we can expand / contract capacity as
required - we would add servers into the load balancer pool. This also
allows us to scale beyond the number of listeners a single Icecast
server can support, while still maintaining the "illusion" of a single
streaming server.

That is only really relevant if you expect to saturate the upstream
bandwidth of the instance running Icecast. Which nowadays is in the
10-20k listener range at the very least. Clustering through DNS also
removes that limitation.
For context, such listener numbers are only achieved by the largest
on-line radio stations.
Does an ELB have more upstream bandwidth than a single instance? Would
be one question. If not (iow >>1Gbit) you need a different approach anyway.

Post by Matt Gray
It also provides redundancy / failover capability by allowing us to
balance connections across Icecast servers in multiple AWS
availability zones.

Redundancy/failover might be worth looking at. I'm not exactly sure if
adding another SPOF is the right approach, but thankfully it's not my
decision.
Out of curiosity, is ELB able to span several AZ?

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.
I'm not sure how well it will cope with the extremely long lived
connections though. In my experience adding the complexity of a reverse
proxy decreases Icecast availability and reliability rather than improve it.

Post by Matt Gray
I'm aware of some "playlist based" loadbalancing techniques using .m3u
or HTTP redirects to the icecast servers, but this is not ideal for
us, as a) we need to support some legacy players (eg. set top boxes)
that don't support redirects or m3u's, and

I'd not go for a playlist as it relies on clients. I'd probably just use
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
distribution. One stable and fully transparent URL. That's also easier
to implement for reliable and robust availability in my opinion. Turns
out Amazon even offers it as a service: Amazon Route 53.

Post by Matt Gray
b) using ELB allows us to very easily provision backend health checks,
multi availability zone redundancy, and auto scaling of Icecast servers.

OK, that sentence ticks a row on my bingo card. ;-)
You're set on using ELB for some reason and I won't argue further with
it. I'll just warn that it's probably not ideal in both HTTP and TCP
mode. You should do significant testing to ensure things work as they do
without it in the way.
Why it doesn't close the TCP connection after the original connection is
gone is not clear to me. It might be a bug or a feature of ELB. You
should actually check AWS documentation or contact Amazon support about
that. You can chose to ignore it and the system will eventually kill
them, but you should make sure to have enough of head room for
connections available on the Icecast machine (ulimit and server config
limits).

I know I sound quite defensive about this topic, but the point I'm
trying to make is, that it adds not well known behaviour, as it's not
fully transparent, and constitutes another single point of failure. Also
there are suitable other options. (I'd even make the same point for a
plain web server)


Cheers

Thomas