The Squid Team are pleased to announce the release of Squid-3.2.0.16 for testing.
This new release is available for download from http://www.squid-cache.org/Versions/v3/3.2/ or the mirrors.
While this release is not deemed ready for production use, we believe it is ready for wider testing by the community.
We welcome feedback and bug reports. If you find a bug, please see http://wiki.squid-cache.org/SquidFaq/BugReporting for how to submit a report with a stack trace.
Although this release is deemed good enough for use in many setups, please note the existence of open bugs against Squid-3.2.
The 3.2 change history can be viewed here.
Squid 3.2 represents a new feature release above 3.1.
The most important of these new features are:
Most user-facing changes are reflected in squid.conf (see below).
Details in Advisory SQUID-2011:1
Squid locates the authority-URL details available in an HTTP request as defined by RFC 2616 and validates that all found representations are textually equivalent. In the case of intercepted traffic the client destination IP is also compared to the Host: authority domains DNS entries.
When the Host: authority contradicts another authority source Squid will log "SECURITY ALERT: Host: header forgery detected" and respond with a 409 Conflict error status page.
The new "workers" squid.conf option can be used to launch multiple worker processes and utilize multiple CPU cores. The overall intent is to make multiple workers look like one to an outside observer, while providing knobs to customize each worker behavior if needed.
By default, all worker processes are configured identically and do what a single Squid instance would have done. Squid.conf macro substitutions and conditionals (see below) can be used to customize individual worker configurations. In the paragraphs below, "can share" implies "will share by default".
Workers can share HTTP, HTTPS, SNMP, ICP, and HTCP listening addresses. Configuration related to ICP and HTCP clients must be adjusted to avoid source address conflicts: Modify the IP address and/or the port used for the protocol. Workers do not share DNS addresses by default because the OS assigns each worker a unique DNS port.
Workers can share logs.
Workers can share caches. Memory cache is automatically shared when multiple workers are used. Cache_dir are shared when configured with the rock storage type. Cache_dir of other types must be adjusted to point each disk-caching worker to its own disk area. ICP and HTCP responses are based on the responding worker cache state.
Cache manager statistics are reported from a worker point of view, for now. Though some reports are combined. SNMP statistics are combined across all workers.
Startup, reconfiguration, shutdown, and log rotation are handled as for a monolithic Squid. Abnormally terminated workers are restarted while other workers continue serving traffic.
Added support for process_name and process_number macros as well as simple if-statement conditionals in squid.conf. These features allow individual worker customization in SMP mode. For details, search for "Conditional configuration" and "SMP-Related Macros" sections in squid.conf.documented.
The helper multiplexer's purpose is to relieve some of the burden Squid has when dealing with slow helpers. It does so by acting as a middleman between squid and the actual helpers, talking to Squid via the multiplexed concurrent variant of the helper protocol and to the helpers via the non-concurrent variant.
Helpers are started on demand, and in theory the muxer can handle up to 1k helpers per instance. It's up to squid to decide how many helpers to start.
The muxer knows nothing about the actual messages being passed around, and as such can't really (yet?) compensate for broken helpers. It is not yet able to manage dying helpers, but it will.
To configure the multiplexer add its binary name (usually /usr/share/libexec/helper-mux.pl) in front of the name of whichever helper is being multiplexed. It takes the helper binary path and parameters as its own command parameters. The concurrency setting already existing in Squid is used to configure how many child helpers it may run.
For example, a traditional configration is
url_rewrite_program /your/redirector.sh
url_rewrite_children 5
the alternative multiplexer configuration is:
url_rewrite_program /usr/share/libexec/helper-mux.pl /your/redirector.sh
url_rewrite_children 1 concurrency=5
Helpers which are already concurrent protocol enabled gain little benefit from the multiplexer on most systems. However on some systems where Squid spawning helpers causes excess memory usage the reduction in direct helper spawned by Squid can result in a great reduction in resource use.
The helper can be controlled using various signals:
Traditionally Squid has been configured with a fixed number of helpers and started them during it's start and reconfigure phases. This forces the hard configuration problem of how many helpers will be needed to be solved before starting Squid in production use.
The on-demand helpers feature allows greater flexibility and resolves this problem by allowing maximum, initial and idle thresholds to be configured. Squid will start the initial set during start and reconfigure phases. However over the operational use new helpers up to the maxium will be started as load demands. The idle threshold determins how many more helpers to start if the currently running set is not enough to handle current request loads.
For example, a traditional configration is
auth_param ntlm /usr/libexec/squid/ntlm_auth
auth_param ntlm children 200
the alternative on-demand configuration could be:
auth_param ntlm /usr/libexec/squid/ntlm_auth
auth_param ntlm children 200 startup=10 idle=2
The example still permits up to 200 helpers to be running at once under peak traffic loads. But only starts 10 when Squid is initialized resulting in a faster boot up. When client requests threaten to overload the running helpers an additional 2 will be started.
NOTE: if no startup and idle values are specified the traditional behaviour of starting the maximum number of helpers will occur.
To improve the understanding of what each helper does and where it should be used the helper binaries which are bundled with Squid have undergone a naming change in this release.
Below is a list of the old helper names and what their names have changed to. For several helpers the directory name used in --enable-X-helpers configure option has also changed.
This group of helpers have been bundled to demonstrate how to code URL re-writers:
The man(8) and man(1) pages bundled with Squid are now provided online for all versions and beginning with 3.2 they are available in languages other than English (where translated).
Details in The Squid wiki
3.1 began the Internationalization of Squid with the public facing error pages. This move begins the Localization of the internal administrator facing manuals.
Automatic detection and use of the pthreads library available from Solaris 10
The result of this addition means that faster more efficient AUFS cache storage mechanisims are now available in Solaris 10.
Support is experimental at this stage due to lack of feedback on the results of enabling it. We recommend giving AUFS a try for faster disk storage and encourage feedback.
The Surrogate extensions to HTTP protocol enable an origin web server to specify separate cache controls for a reverse proxy acting on its behalf. Previously this was closely tied with the ESI feature support in Squid. This release opens Surrogate support to all reverse proxies.
Reverse proxy requests sent on to the web server include the HTTP header Surrogate-Capabilities: specifying the capabilities of the reverse proxy along with an ID which can be used to target reponses with a Surrogate-Control: HTTP header used instead of the Cache-Control: header.
The default surrogate ID is generated automatically from the Squid site-unique hostname as found by the automatic detection or manual configuration of visible_hostname although can be configured separately with the httpd_accel_surrogate_id option.
Security Considerations: Websites sould be careful of accepting any surrogate ID. Older releases of Squid leak the Surrogate-Control headers to external servers. This 3.2 series of Squid will now prevent this leakage of its own ID destined responses, however it is possible and for some uses desirable to receive external reverse-proxies Surrogate-Capabilities: headers.
NOTE: Several operating system distributions historically package Squid with a forced value of visible_hostname localhost. If this is done on a Surrogate enabled install a manual re-configuration is required to prevent an unacceptable surrogate ID of 'localhost' being generated.
The advanced logging modules introduced in Squid-2.7 are now available from Squid-3.2.
This feature is documented at http://wiki.squid-cache.org/Features/LogModules
The new infrastructure currently supports several different channels types (modules) ranging from direct filesystem logging (stdio, daemon) to network logging (syslog, UDP and TCP). The daemon logging interface allows for a custom helper to be written to process logs in real-time.
Upgrading: the access_log was previously logged via what is now called the stdio module. This is still supported and used by default if no module is named. For best performance particularly in SMP environments we recommend the daemon be used. The provided log_file_daemon helper performs the traditional logging to local filesystem.
Additional to this the cache.log can now be limited to a smaller number of files stored. Traditionally cache.log.N has been fixed at the same number of rotated files as access.log.N through the logfile_rotate setting. The debug_options setting can now be used to configure the number of debug cache.log files to rotate through with a rotate=N option. This is particularly useful for logging a single cache.log at relatively high debug levels on a high-traffic system. Or one which is required to store a long period of access.log and needs to conserve disk space.
The referer_log and useragent_log directives have been converted to built-in log formats. These logs are now created using an access_log line with the format "referrer" or "useragent". They also now log all client requests, if there was no Referer or User-Agent header a dash (-) is logged.
In mobile environments, Squid may need to limit Squid-to-client bandwidth available to individual users, identified by their IP addresses. The IP address pool can be as large as a /10 IPv4 network (4 million unique IP addresses) and even larger in IPv6 environments. On the other hand, the code should support thousands of connections coming from a single IP (e.g., a child proxy).
The implementation is based on storing bandwidth-related "bucket" information in the existing "client database" hash (client_db.cc). The old code already assigned each client IP a single ClientInfo object, which satisfies the client-side IP-based bandwidth pooling requirements. The old hash size is increased to support up to 32K concurrent clients if needed.
Client-side pools are configured similarly to server-side ones, but there is only one pool class. See client_delay_pools, client_delay_initial_bucket_level, client_delay_parameters, and client_delay_access in squid.conf. The client_delay_access matches the client with delay parameters. It does not pool clients from different IP addresses together.
Special care is taken to provide fair distribution of bandwidth among clients sharing the same bucket (i.e., clients coming from the same IP address). Multiple same-IP clients competing for bandwidth are queued using FIFO algorithm. If a bucket becomes empty, the first client among those sharing the bucket is delayed by 1 second before it can attempt to receive more response data from Squid. This delay may need to be lowered in high-bandwidth environments.
Support for libecap version 0.2.0 has been added with this series of Squid. Bringing better support for body handling, and logging.
The Squid Cache Manager has previously only been accessible under the cache_object:// URL scheme. Which has restricted its reporting to tools which can send arbitrary URI to the proxy.
This version of Squid now provides access through the http:// and https:// URL schemes allowing web browsers access without having to use the cachemgr.cgi gateway and enabling the use of HTTPS security were desired.
The cache manager is available under the path prefix /squid-internal-mgr/. For example the URL http://example/com/squid-internal-mgr/menu will bring up the manager menu. This means there are some configuration changes required to lock down manager access. The manager ACL needs changing to:
acl manager url_regex -i ^cache_object:// ^https?://[^/]+/squid-internal-mgr/
The manager prefix /squid-internal-mgr/ with no action attempts to load an optional template MGR_INDEX which may be installed amongst in the Squid error templates. This template is not supplied with Squid but intended to be supplied by separate cache manager applications as their front page embedding all scripts, accessors or redirects required for their initial GUI display.
Version 3.2 of the CGI cache manager tool now presents XHR scripted probes to detect proxies presenting these manager index pagess and provides direct HTTP/HTTPS web links to those managers.
There have been changes to Squid's configuration file since Squid-3.1.
This section gives a thorough account of those changes in three categories:
This option allows Squid administrator to add custom ICAP request headers or eCAP options to Squid ICAP requests or eCAP transactions.
Same as depricated icap_send_client_ip but applies to both ICAP and eCAP.
Same as depricated icap_send_client_username but applies to both ICAP and eCAP.
Same as depricated icap_uses_indirect_client but applies to both ICAP and eCAP.
New setting for client bandwith limits to specifies the number of client delay pools used.
New setting for client bandwith limits to determine the initial bucket size as a percentage of max_bucket_size from client_delay_parameters.
New setting for client bandwith limits to configures client-side bandwidth limits.
New setting for client bandwith limits to determines the client-side delay pool for the request.
New setting to disable extra Host: header security on interception proxies. Impacts cache integrity/reliability and client browser security.
IMPORTANT: disabling this directive only allows Squid to change the destination IP to another source indicated by Host: domain DNS or cache_peer configuration. It does not affect Host: validation.
Renamed from persistent_request_timeout.
New setting for SMP support to map Squid processes onto specific CPU cores.
Replacement for maximum_single_addr_tries, but instead of only applying to hosts with single addresses. This directive applies to all hosts, extending the number of connection attempts to each IP address.
New setting to configure maximum number of bytes packet size to advertise via EDNS. Set to "none" (the initial default) to disable EDNS large packet support.
Part of conditional SMP support syntax. see if
Part of conditional SMP support syntax. see if
Whether to lookup the EUI or MAC address of a connected client.
New option to toggle whether the ICAP 206 (Partial Content) responses extension. Default is on.
New conditional syntax for SMP multiple-worker. If-statements can be used to make configuration directives depend on conditions.
The else part is optional. The keywords if, else and endif must be typed on their own lines, as if they were regular configuration directives.
Ported from 2.7. Specify the file I/O daemon helper to run for logging.
Places an upper limit on how stale content Squid will serve from the cache if cache validation fails
Controls which objects to keep in the memory cache (cache_mem)
'always' Keep most recently fetched objects in memory (default)
'disk' Only disk cache hits are kept in memory, which means
an object must first be cached on disk and then hit
a second time before cached in memory.
network Only objects fetched from network is kept in memory
Controls whether the memory cache is shared among SMP workers.
Currently, entities exceeding 32KB in size cannot be shared.
Renamed from pconn_timeout.
Controls whether the indirect client address found in the X-Forwarded-For header is used for spoofing instead of the directly connected client address. Requires both --enable-follow-x-forwarded-for and --enable-linux-netfilter
Number of main Squid processes or "workers" to fork and maintain. In SMP mode, each worker does nearly all what a single Squid daemon does (e.g., listen on http_port and forward HTTP requests).
0: "no daemon" mode, like running "squid -N ..."
1: "no SMP" mode, start one main Squid process daemon (default)
N: start N main Squid process daemons (i.e., SMP mode)
New setting to limit time spent waiting for data writes to be confirmed.
New stdio module to send log data directly from Squid to a disk file. This is the historic behaviour of Squid before logging modules were introduced, and remains the default used when no module is selected. It is recommended to upgrade logging to the faster daemon: module.
New daemon module to send each log line as text data to a file I/O daemon handling the slow disk I/O. New installs, or installs with no logs configured explicitly will use this module by default.
New tcp module to send each log line as text data to a TCP receiver.
New udp module to send each log line as text data to a UDP receiver.
New format referrer to log with the format prevously used by referer_log directive.
New format useragent to log with the format prevously used by useragent_log directive.
New type random. Pseudo-randomly match requests based on a configured probability.
Renamed myip to localip. It matches the IP which the client connected to.
Renamed myport to localport. It matches the port which the client connected to.
The localip/localport differ from earlier releases where they matched a mix of of an invalid IP and port 0, the client destination IP/port or the Squid listening IP/port. This definition is now consistent across all modes of traffic received by Squid.
The manager ACL requires adjustment to cover new cache manager access:
acl manager url_regex -i ^cache_object:// ^https?://[^/]+/squid-internal-mgr/
New options for Basic, Digest, NTLM, Negotiate children settings. startup=N determins minimum number of helper processes used. idle=N determines how many helper to retain as buffer against sudden traffic loads. concurrency=N previously called auth_param ... concurrency as a separate option.
Removed Basic, Digest, NTLM, Negotiate auth_param ... concurrency setting option.
min-size option ported from Squid-2
htcp-* options collapsed into htcp= taking an optional comma-separated list of flags. The old form is deprecated but still accepted.
New configuration parameter clientside_mark
Allows packets leaving Squid on the client side to be marked with a Netfilter mark value in the same way as the existing clientside_tos feature.
This feature is only available for Netfilter environments.
Support URL format tags. For dynamically generated URL in denial redirect.
Support the full range of 200-599 HTTP status codes. 3xx status only available when redirecting to a URI. Other status only available when supplying an error template body.
New format tags and option parameters:
%SRCEUI48 EUI-48 / MAC address of client from ARP lookup.
%SRCEUI64 EUI-64 of clients with SLAAC address.
%EXT_LOG log= message returned by previous external ACL calls. An updated version may be returned.
%EXT_TAG tag= value returned by previous external ACL calls. Tag may not be altered once set.
children-max=N determins maximum number of helper processes used.
children-startup=N determins minimum number of helper processes used.
children-idle=N determines how many helper to retain as buffer against sudden traffic loads.
Deprecated children=N in favor of children-max=N.
act-as-origin ported from 2.7. This option corrects several HTTP header issues when operating as a reverse proxy and cache. Notably the externally visible aging of objects stored in the server-side cache.
vhost is deprecated. accel mode, reverse proxy, now defaults to always enable HTTP/1.1 virtual domain support.
no-vhost option is added to disable the new reverse proxy behaviour.
Deprecated in favor of adaptation_send_client_ip which applies to both ICAP and eCAP.
Deprecated in favor of adaptation_send_username which applies to both ICAP and eCAP.
Deprecated in favor of adaptation_uses_indirect_client which applies to both ICAP and eCAP.
%<a Server or Peer IP address from the last server connection (next hop).
%>bs Number of HTTP-equivalent message body bytes received from the next hop.
icap::%>bs Number of message body bytes received from the ICAP server.
%sn Unique sequence number per log line. Ported from 2.7
%>eui EUI logging (EUI-48 / MAC address for IPv4, EUI-64 for IPv6). Both EUI forms are logged in the same field. Type can be identified by length or byte delimiter.
%err_code The ID of an error response served by Squid or a similar internal error identifier
%err_detail Additional err_code-dependent error information.
%>la Rename of %la to indicate being a client connection detail.
%>lp Rename of %lp to indicate being a client connection detail.
%<p Server or Peer port number from the last server connection (next hop).
Memory limits have been revised and corrected from 3.1.4 onwards.
Please check and update your squid.conf to use the text none for no limit instead of the old 0 (zero).
All users upgrading need to be aware that from Squid-3.3 setting this option to 0 (zero) will mean zero bytes of memory get pooled.
New options mark and tos and miss
tos retains the original QOS functionality of the IP header TOS field.
mark offers the same functionality, but with a netfilter mark value.
These options should be placed immediately after qos_flows.
The tos value is optional in order to maintain backwards compatability.
The preserve-miss functionality is available with the mark option and requires no kernel patching. It does, however, require libnetfilter_conntrack. This will be included by default if available (see the --without-netfilter-conntrack configure option for more details).
miss sets a value for a cache miss. It is available for both the tos and mark options and takes precedence over the preserve-miss feature.
Added ACL support for control over when the limit applies and when it is avoided.
New option max-stale= to provide a maximum staleness factor. Squid won't serve objects more stale than this even if it failed to validate the object.
This parameter is now compatible with persistent server connections. The IPv6 magic 'to_ipv6' hacks needed in 3.1 are now no longer necessary.
New configuration parameter tcp_outgoing_mark
Allows packets leaving Squid on the server side to be marked with a Netfilter mark value in the same way as the existing tcp_outgoing_tos feature.
This feature is only available for Netfilter environments.
This parameter is now compatible with persistent server connections.
New options startup=N, idle=N, concurrency=N
Now only available to be set in Windows builds.
Obsolete. Replaced by DNS parallel lookups.
Replaced by common format option on an access_log directive.
Obsolete.
Obsolete.
Obsolete.
Obsolete. Replaced by automatic detection of the %>A logformat tag.
Obsolete. Use a custom log with %<A format tag to receive server FQDN or peer name.
The behaviour controlled by this directive is no longer possible. It has been replaced by connect_retries option which operates a little differently.
Renamed to server_idle_pconn_timeout
Renamed to client_idle_pconn_timeout
Replaced by the referrer format option on an access_log directive.
Replaced by url_rewrite_children ... concurrency=N option.
Replaced by the useragent format option on an access_log directive.
There have been some changes to Squid's build configuration since Squid-3.1.
This section gives an account of those changes in three categories:
Specified without any parameters all helpers will be auto-built.
With an explicit empty list ="" protocol support will be built but no helpers.
With an explicit list protocol support and just those helpers will be built.
Specified without any parameters all helpers will be auto-built.
With an explicit empty list ="" protocol support will be built but no helpers.
With an explicit list protocol support and just those helpers will be built.
Specified without any parameters all helpers will be auto-built.
With an explicit empty list ="" protocol support will be built but no helpers.
With an explicit list protocol support and just those helpers will be built.
Specified without any parameters all helpers will be auto-built.
With an explicit empty list ="" protocol support will be built but no helpers.
With an explicit list protocol support and just those helpers will be built.
Add an additional string in the output of "squid -v".
Enable Support for handling EUI operations. This includes ARP lookups for MAC (EUI-48) addresses and the ACL arp type tests.
Build helpers for logging I/O.
Build helpers for some basic URL-rewrite actions. For use by url_rewrite_program. If omitted or set to =all then all bundled helpers that are able to build will be built. If set to a specific list of helpers then only those helpers will build. Currently one demo helper fake is provided in shell and C++ forms to demonstrate the helper protocol usage and provide exemplar code.
Location to display in documentation for the default cache. Updated to indicate /var/cache/squid in accordance with the filesystem layout standards. Squid-3 no longer builds an implicit disk cache at this location, so the change is not expected to have any effect on existing builds other than fixing some mysterious lack of core dumps. The old /var/cache location was often non-writable which blocked core dumps creation.
Disables the libnetfilter_conntrack library being used for the new qos_flows option mark. default is to auto-detect the library and use where available.
No longer takes a list of arguments. This option now is restricted to building Squid with or without authentication support.
The new --enable-auth-X/--disable-auth-X parameters determine which authentication protocols and helpers are built.
Replaced by --enable-eui
replaced by --enable-auth-basic.
replaced by --enable-auth-digest.
replaced by --enable-auth-negotiate.
replaced by --enable-auth-ntlm.
Obsolete.
Obsolete.
Some squid.conf and ./configure options which were available in Squid-2.6 and Squid-2.7 are made obsolete in Squid-3.2.
blankpassword option for basic scheme removed.
Not safe for general use. An external_acl_type helper may be used to bypass authentication if that is suitable.
Not safe for general use. An external_acl_type helper may be used to bypass authentication if that is suitable.
http11 Obsolete.
Format tag %{Header} replaced by %>{Header}
Format tag %{Header:member} replaced by %>{Header:member}
Replaced by request_header_access and reply_header_access
no-connection-auth replaced by connection-auth=[on|off]. Default is ON.
transparent option replaced by intercept
http11 obsolete.
Replaced by adapted_http_access
Replaced by http_port disable-pmtu-discovery= option
Obsolete.
Replaced by url_rewrite_bypass
Obsolete.
Obsolete.
Replaced by qos_flows local-hit=
Obsolete.
Obsolete.
Replaced by qos_flows parent-hit=
Replaced by qos_flows sibling-hit=
read-only option replaced by no-store.
Obsolete.
Replaced by automatic detection.
Obsolete.
Replaced by automatic detection.
Obsolete.
Obsolete.
Obsolete. Enabled by default.
Obsolete.
Obsolete.
Obsolete.
Replaced by automatic detection.
Replaced by automatic detection.
Replaced by automatic detection.
Obsolete. Enabled by default.
Obsolete.
Obsolete. Disabled by default.
Some squid.conf and ./configure options which were available in Squid-2.7 are not yet available in Squid-3.2
If you need something to do then porting one of these from Squid-2 to Squid-3 is most welcome.
urllogin option not yet ported from 2.6
urlgroup option not yet ported from 2.6
Not yet ported from 2.6
COSS storage type is lacking stability fixes from 2.6
COSS overwrite-percent= option not yet ported from 2.6
COSS max-stripe-waste= option not yet ported from 2.6
COSS membufs= option not yet ported from 2.6
COSS maxfullbufs= option not yet ported from 2.6
idle= not yet ported from 2.7
monitorinterval= not yet ported from 2.6
monitorsize= not yet ported from 2.6
monitortimeout= not yet ported from 2.6
monitorurl= not yet ported from 2.6
Not yet ported from 2.6
Not yet ported from 2.6
Not yet ported from 2.6
%ACL format tag not yet ported from 2.6
%DATA format tag not yet ported from 2.6
Not yet ported from 2.7
urlgroup= not yet ported from 2.6
Not yet ported from 2.7
Not yet ported from 2.6
Not yet ported from 2.6
Not yet ported from 2.6
Not yet ported from 2.6
stale-while-revalidate= not yet ported from 2.7
ignore-stale-while-revalidate= not yet ported from 2.7
negative-ttl= not yet ported from 2.7
Not yet ported from 2.7
Not yet ported from 2.7
Not yet ported from 2.7
Not yet ported from 2.7
Not yet ported from 2.7
Not yet ported from 2.7
Not yet ported from 2.7