Why does Squid 2.7 crash when given a large external rules file for url_regex?

Question

Squid

There is squid 2.7 running on Ubuntu. The computer has 2 gigabytes of RAM. On average, over 5 years of work, the black list of sites has been filled with about 1,500 sites. Entertainment, social networks, porn... Broken into categories, the maximum list - 700 sites.

A file with a categorical order came from a higher organization: to filter the attached list, and a file with a list of 10 megabytes, 552,000 sites long.

The first attempt to make a solution in the forehead through:

acl url_blnav url_regex "/home/....../domains.txt"
http_access deny url_blnav

led to the fact that the squid ate 4.5 gigabytes of virtual memory for the list and went into a deep swap, completely ceasing to respond to requests.

I reduced the list to 380 thousand sites by removing sites with common words, such as sex porno erotic, etc. Then I tried splitting the file into parts - this reduces memory consumption (up to 2.5 gig), but still does not fit into the RAM. As a result, the squid still goes into the swap and shuts up. on my usual rules, the squid occupied ~ 650-700 megabytes of memory.

How can I put a blocklist for 380,000 sites into a squid?

Add memory - I can not. I can’t discuss the issue of reducing the number of sites (there is so much heresy where 100% people won’t go) either.

separate question - whether the upgrade on a squid 3.3 or 3.2 will help?

Answer 1

As far as I remember, squid will create a separate acl in memory for each line in domains.txt. You can try to use SquidGuard ( http://www.squidguard.org/ ) it is just for working with large lists.

Answer 2

Perhaps in this situation it makes sense to make a whitelist instead of a blacklist? Or, in general, do filtering not using squid, but, for example, using dns, which squid accesses.

Answer 3

Upgrading to 3.x is the first thing I would do. 2.x is very outdated. With a probability of 0.8 your problem will be solved by an upgrade.