Serving static content under heavy load

Question

CMS

On one project, the authorities came up with the idea of ​​giving partner sites a banner + a small JS file (when you click on the banner, the script receives the current coordinates of the user and transfers them to our site in the parameters). Banner + JS is a BN package. For each partner there may be different or only their own packages.
And the problem is that the first partner has 2.5 million uniques a week. And there are plans to find several more such partners, i.e. the load will grow immediately and by many.

And here's the problem, how to organize such a distribution in the best way?
The project itself is written in PHP.
As an option, we considered CDN, but we still have the logic to show only a certain number of times, the ability to pause the display, etc.
Searching the web, I found a good presentationwww.highload.ru/papers2009/12189.html

As a result, I came up with the following system. And I would like to hear the opinion of experienced habravchans, how correct this solution is, and perhaps there is something more elegant.



SS - Server of statics, stores and distributes statics
LB - Load balancer for starters, apparently software.
MS - (Main server) Server of the main project
CM - (Content manager server). SS control server, receives commands from MS and sends commands to SS.

Message types

SHOW, HIDE, DELETE

Tasks

SS

1. Store, delete and give away BN content
2. Each has Nginx, PHP, MemcacheDB database installed. All have the same software configuration
3. Also has access to MemcacheDB database on CM
4. MemcacheDB stores current counters and statuses for all BNs in its own database and periodically synchronizes them with CM
5. Has a set of scripts for processing incoming messages from CM

CM

1. Also stores master copies of all current BNs
2. Installed Nginx, PHP, MemcacheDB database.
3. Can receive messages from MS and send to SS
4. Can sync SS with its internal set of BNs
5. Maintains a common table for all BNs, with counters, current statuses, etc.
6. It also maintains a table of available (IP?) addresses of all SSs. And in case of adding/removing SS`a, in principle, it should be enough to register/remove its address in this table.

Such a schedule. Unfortunately, there is little experience in this kind of tasks, the opinion of the habrand people is very interested.
Can not take a steam bath and switch to cloud hosting? Or on such loadings there hefty price grows?

Thanks in advance!

Answer 1

You've got something.
If partners, unlike you, can easily withstand such a load, give them banners and a js file for storage.

Answer 2

You can save on LB by transferring the load to DNS or Routers.
Give from stats.
In general, I don’t see a problem with 2.5 million (although it’s not clear how many hits there will be) the average Dedik will withstand it without problems - it’s small banners that will directly give them from memory, the problem will be only in the OS settings and channel width.
If these are hits, then these are only 56 hits per second (for two files) - even Apache can handle it, I'm not talking about nginx.
You will rest only on the channel width and the performance of the OS and network card.

Answer 3

One dual-processor (2 Xeon 2.4Ghz, Pentium-4 times) Linux Debian 4.0 + SCSI on Nginx withstood 100 million static hits per day. Now processors are much faster, and SSD has appeared.