Changes between Version 1 and Version 2 of WebserverSetup

Timestamp:

08/12/11 14:12:22 (13 years ago)

Author:

joe

Comment:

--

WebserverSetup

@@ -131 +131 @@
 3. If it didn't have a DNS name, you can try using the `whois` command to see who institution the IP range is registered to.
 
-
+=== Blocking by Browser / User-Agent ===
+
+You can also reject by the client's browser identification string, although, they can always change or hide it, so this isn't always good solution.  It can be a hint to search crawlers or `wget` users that their activity isn't appreciated, though.
+
+As we have hundreds of thousands of requests per day for the same file, all coming from thousands of IP addresses, by a single strange browser (that I've tried asking companies with similarly named software if their stuff acts as a browser, and if so, to stop it), we use the following to block access:
+
+{{{
+<Location "/images/latest_eit_304.gif">
+    BrowserMatch ^CompanionLink badClient
+    Order allow,deny
+    Allow from all
+    Deny from env=badClient
+</Location>
+}}}
+
+== Restricting Access to Local Users ==
+
+If you have data that's under embargo, and you want to make it available only to local users, you can limit access to a directory such as `/embargoed` to [http://httpd.apache.org/docs/2.0/mod/mod_access.html#allow specific IP addresses or to group of IP addresses]:
+
+{{{
+<Location "/embargoed/">
+    Order deny,allow
+    Deny from all
+    Allow from 10.1.1.1/24
+    Allow from 10.11.12.167
+</Location>
+}}}
+
+If you want to allow the data to be accessed by local users, by also from outside with the proper username and password, you can do:
+
+{{{
+<Location "/embargoed/">
+    Order deny,allow
+    Deny from all
+    Allow from 10.1.1.1/24
+    Allow from 10.11.12.167
+
+    AuthType Digest
+    AuthName "Embargoed Data"
+    AuthDigestDomain /embargoed/
+    AuthDigestFile /path/to/htdigest/file
+    Require valid-user
+
+    Satisfy any
+</Location>
+}}}
+
+Note that in Apache 2.2, `AuthDigestFile` should be changed to `AuthUserFile`.  You can use `AuthType Basic` instead, but then the passwords are sent in the clear, and you'll have to [http://httpd.apache.org/docs/2.0/howto/auth.html change some of the lines].
+
+
+
+
+== Debugging Slow Web Servers ==
+
+Entries into the webserver's access logs only occur once the client disconnects.  This means that if you have lots of connections sitting open, you can't use the access logs to see what's going on.  Apache does have a way to get some information about what's going on, however.  Look for a section in your server config mentioning `ExtendedStatus` or `server-status`, and change it to read something like:
+
+{{{
+<IfModule mod_status.c>
+    <Location "/server-status">
+        SetHandler server-status
+        Order deny,allow
+        Deny from all
+        Allow from 127.0.0.1
+        Allow from your-ip-address
+    </Location>
+    ExtendedStatus On
+</IfModule>
+}}}
+
+Obviously, set `your-ip-address` to an appropriate value, and you can set more than one `Allow` line to allow connections from more than one machine.  You can then (after restarting the webserver) request the page `http://servername/server-status`, which down at the bottom will give a report that includes what requests are being processed, what IP asked for it, and how long it's been processing, so you can try to identify what might be having problems, or what connecting IP address might be doing strange things.  You have to do this in advance of the request; as it requires a web server restart to turn on this feature, you can't just turn it on when you have a problem connection (unless it's a flood of problem connections that you're actively monitoring).
+
+ 
+