URL filtering

URL filtering

Users spend increasing time on the web, surfing their favorite sites, clicking on email links or utilizing a variety of web-based SaaS applications for both personal and business use. While this could be incredibly useful to drive business productivity, this kind of unfettered web activity exposes organizations to a range of security and business risks, such as propagation of threats, possible data loss and potential lack of compliance.

Traditionally, companies have used URL filtering as a tool to prevent employees from accessing unproductive sites. With today’s URL filtering, firms enable secure web access and protection from increasingly sophisticated threats, including malware and phishing sites.

Suppose you type the name of your favorite social networking site on the web browser and it displays a message like “The policy of this organization doesn’t allow you to browse that website” and does not let you access the site from office, there is a URL filter that has been put in place by your Organization. So, a URL filter is used to basically categorize the websites on the internet and either allow/block the access to the web users of the organization either by referring to an already categorized central database (maintained by URL filtering vendors) or by classifying the websites in real time. URL filtering can also be made applicable only during certain times of a day or days of a week, if required.

Why is URL Filtering required?

URL filtering is required to stop the users of an organization from accessing those websites during working hours that:

  • Drain their productivity.
  • Let’s them view objectionable content from their workplace.
  • Are bandwidth intensive and hence create a strain on resources.

How URL Filtering is done?

URL Filtering is basically done by maintaining a highly categorized database of most of the websites in the internet and either allowing access to them or dis-allowing access to the internet users of an organization either at all times or during certain times of a day. The policies of which categories of sites is to be allowed/dis-allowed to the users of an organization could be set by the Enterprise through a web-based interface provided by URL filters. So, there is a local hardware appliance or software application running on a server that connects to a central database of the URL filtering vendors which enable to block individual websites.

There might be a local database, which is updated fully or partially from the central database. But updating them completely might have its own productivity problems like bandwidth or memory usage. Some vendors update such databases (local) as and when the users visit the websites (it typically takes only few milliseconds to do so).

A website can be categorized in a single category or multiple categories and the blocking can be done appropriately. For example, websites can be allowed to be accessed if they are categorized as sports but not if they are categorized as sports and gambling.

Generally, websites are rated based on their domain names (In addition to the URL’s) as one domain can have multiple URL’s that tend to increase frequently. Optionally, even the IP addresses of the domain names can be included while rating the domains. The sub-domains also need to be classified in-addition to the main domains (For blogs, etc.) and the intermediate pages need to be classified in addition to the primary pages or based on primary pages. (Like translation sites or sites that display images from other websites) Websites in multiple languages may also be needed to be categorized similarly.

Categorizing websites in Real-time:

Since the internet is so huge, it is practically impossible to categorize the entire list of websites present in it. So, when certain sites are being accessed by the user, the URL filtering systems categorize them ‘on the fly’ or in Real-time. This typically takes only a couple of hundred milliseconds and the local databases are automatically updated along with the central database.

Human Intervention:

There are times when the learning machines are not able to classify websites and all such websites are categorized by human professionals, who actively participate in training them, analyzing the results and abnormalities etc. Site submissions are also accepted from all the users, which is reviewed by professionals for classification (for the websites that are not already classified).

Advantages and Disadvantages of URL Filtering:

As mentioned earlier, URL filtering helps organizations improve productivity by making sure that employee time is not spent in unnecessary activities during office hours. These URL filtering can also help by preventing malicious code/spyware, phishing etc. which may be potentially harmful to the organization. Some vendors also help block Peer-to-Peer software’s and Instant messaging which use more resources, wastes time and is also a security threat.

Over-blocking can cause issues with users. (Example, some commercial spyware needs to be installed for certain applications to work and blocking them might deny access to those applications to the users) And over-blocking can also result in more help-desk tickets that need to be attended to, and resolved by the support team. If that happens frequently, then both the time of the user and the support team is utilized excessively. Sometimes, there is a problem with certain websites that have been already classified and then become threat sites/ avoidable sites at a later stage.

URL filtering allows you to control access to Internet websites by permitting or denying access to specific websites based on information contained in an URL list. You can maintain a local URL list on the router, and you can use URL lists stored on Websense or Secure Computing URL filter list servers. URL filtering is enabled by configuring an Application Security policy that enables it. Even if no Application Security policy is configured on the router, you can still maintain a local URL list and an URL filter server list that can be used for URL filtering when a policy is created that enables it.

An effective URL filtering solution is essential for an enterprise because granting your employees unfettered access to the Internet opens your company to a multitude of problems.

Some of them are-

Security: Grave risk to the companies’ security.

Legal Trouble: Liability of inappropriate content.

Productivity: Loss of employee productivity due to Internet abuse.


With the advent of Web 2.0 technologies, websites are now mash-ups of content that is aggregated from many other sites. This scenario adds complexity to filtering websites based on domain names alone and also opens up new avenues of attack for hackers and virus writers who are increasingly becoming successful at compromising syndicated feeds. If just one feed of data is compromised, all the websites that pull in that feed will deliver malicious code to their trusted users.

Inappropriate Content

Companies can be liable for the data that is transferred into their internal networks and servers. In an ideal world, you should be able to trust all your employees that they will only use the Internet for the jobs they do. The reality is unfortunately very different, and one single rogue user’s actions can create serious legal problems if his or her actions are not monitored. Pornography at work and downloading of copyrighted materials are a real and dangerous threat to an Organization.

If a good URL filtering solution is in place, the organization is protected against this type of threat. Pornography and copyrighted material is blocked using standard filters. A good URL filtering solution from a trusted vendor will also protect you legally because it shows your commitment to stopping this type of crime in your workplace.

Employee Productivity

A security breach due to lack of URL filtering will undoubtedly affect your employee productivity; however, an ineffective or badly configured filter can also have a negative effect on productivity if it blocks legitimate content that is needed by your employees to do their job.

GajShield provides URL filtering as a feature which aims at effectively filtering out unwanted websites and URLs to provide a secure and safe internet presence to the enterprises.

GajShield provides various features for effective URL filtering in organizations like Automated web categorization engine to block non-productive websites, URL filtering for HTTP and HTTPS protocols, MIME type and file extension blocking,

In Addition to this, GajShield’s URL filtering solution supports 85+ categories to be able to block millions of websites falling under those categories as per the requirement of enterprises, user and group based URL filtering to selectively block or give access to a certain websites along with download size restriction policies, Time-based access schedule, mechanism to prevent download of streaming media, images, popup ads, etc., support to block malware, phishing and pharming URLs, support to block Java Applets, cookies, Active X, URL exemption/whitelisting, Automatic updates to URL categories and more. All these features effectively helps the organization to provide restrictive internet access to its users and thereby curbs illegal use.