Sourceforge… I’m disappointed

In a hurry to acquire the latest Filezilla installer, I clicked download on the Sourceforge page and then ran the installer for what I thought would be Filezilla. Having forgotten about something I learnt a few months back (maybe even over a year ago), many Sourceforge downloads have now been polluted with adware and ‘unwanted’ software which could be perceived as malware or spyware. In my haste I clicked next.. next and next only realising at the last second that I had just allowed Norton 360 and nefarious browser plugins to be installed. I wasn’t happy…

Removing the software even required me to reboot which isn’t something it asked me to do after installation. Suffice to say, removal of the software seemed to get rid of most of the rubbish and then a quick scan with MalwareBytes seems to have done the rest. It is shameful that Sourceforge have allowed this to happen.

I also found this on my searches which has a more detailed explanation of some of the software being installed.

Broadband Usage Limits

Most of the mainstream ISP’s such as Virgin Media, Talk Talk, BT and Sky all offer broadband connections with an ‘unlimited’ or ‘unmetered’ bandwidth consumption. ISP’s who do this will tend to hide in the small print that they will traffic manage your connection during peak periods. This basically means that if you exceed a certain threshold, i.e. you download more than 10GB of data between the hours of 6pm and 10pm they will traffic shape your bandwidth from it’s maximum of 80mb down to 20mb (for example). Virgin Media are renowned for doing this and although they offer a 152mb connection, they sometimes apply traffic management policies if you’re using a lot of bandwidth during peak times and they have other sneaky tricks such as using transparent proxies and hijacking your DNS to point to their own caches for things like Netflix and Youtube. This has it’s own issues but is best reserved for another post!

On the other side of all this are smaller ISP’s who can’t afford to offer unlimited bandwidth but instead offer you a fully unfiltered and non-traffic managed connection. The catch is of course that you are often given a small bandwidth allowance on their FTTC packages (not so much on their ADSL packages) and to get a higher allowance, you will have to pay a premium. A couple of the smaller ISP’s operating in this manner are Xilo and AAISP (Andrews & Arnold). The former offer unlimited bandwidth ADSL packages but these are comparatively slow to the FTTC packages they also offer but with a cap. Taking their 500GB option will cost you over £50/month. The latter ISP have usage caps on all of their residential connection offerings (both ADSL & FTTC) and to get a 300GB bandwidth allowance (the maximum available) with them will cost you an extra £20 on top of your normal monthly cost for the connection.

I’m not a heavy user in terms of downloads month on month so generally these limits don’t pose too much of an issue for me. However, I’m beginning to watch a lot more Sky Go, Now TV, 4OD etc and just tonight I bought GTA5 for the PC which when I went to download it on Steam comes out at a huge ***60GB***. As I’m with AAISP, I immediately had to top-up my allowance by 50GB (£10) to allow me to continue with the download as I was quickly coming towards the end of my 100GB monthly allowance. Coupled with all of this, I would like to take monthly backups of VM’s running in a datacenter to my local NAS here at home, but this just isn’t feasible when some of the VM’s attached disks total more than triple my monthly bandwidth allowance!

I have proposed a possible solution to AAISP before in that they offer a vaster amount of bandwidth allowance, i.e. 1TB a month but throttle my 80mb FTTC connection to 20mb down but allow burst traffic to 80mb. As I’m not a heavy user for most of the month, this would probably suit me down to the ground. In order to get around this issue, I’ve had to order a new connection from a mainstream provider which has an ‘unlimited allowance’, however in general I concede that if you were to try to take advantage of this 24/7 you might find very quickly that you get complaint letters from your ISP. So I think rather that a higher solid limit be set and agreed to and then there is no differing interpretations of what unlimited means.

Cable Trunking

One of the few disadvantages to being a geek is the fact that hiding the myriad of cables you inevitably have running around can sometimes prove to be difficult. After some brief investigation into the best method of hiding said cables, I came across some cheap but seemingly effective trunking at Clas Ohlson. The ‘D-line’ trunking perfectly suited my requirements for hiding away 3 disobedient solid core Ethernet cables and at £5.99, you can’t really go wrong!

I also needed some floor trunking to go across the doors for the same cables as they were becoming a bit of a trip hazard. Thankfully, I managed to get a couple of these from Clas Ohlson as well which has tidied things up nicely!

Observium Custom Agent Module

Observium has been my graphing system of choice for a long while now, originally brought in to replace Cacti and Munin. Due to the brief documentation provided on their site on how to add new graphs using the agent system, I embarked on discovering this for myself (with the help of their guide) and as a result, I have put together the following guide.

Pre-Requisites:

1) The Observium agent must be fully installed and operational on the server you’re trying to monitor (check it’s listening on TCP/36602);
2) You’ll need a little bit of patience as it can be a little bit fiddly to get working, but very much worth the effort.

Create module on monitored server


#!/bin/bash

procs=$(ps -ef|grep "[h]ttpd" | wc -l)

echo "<<<app-apache_procs>>>"

echo $procs

It’s very important to start the script with the following (substituting *yourapp* with a name such as ‘apache_procs’):


echo "<<<app-*yourapp*>>>"

Once you have created this, make sure the file is executable and that it’s in the ‘/usr/lib/observium_agent/local’ directory so it can be executed by the agent:


# chmod +x myapp.sh && mv myapp.sh /usr/lib/observium_agent/local

You can now test this is working by connecting with telnet to the agent port of the server in question:


# telnet localhost 36602
...
<<<app-apache_procs>>>
12
Connection closed by foreign host.
...
#

Create Observium side scripts for collection and generation of graph data

Now that we have the easy bit out of the way, it’s time to create the relevant scripts for the module on the Observium server itself so that it is able to generate graphs and store RRD data. These consist of the poller include, application graph include and html page include.

In our guide, we use the base directory of ‘/opt/observium’ and you may need to change this to suit your setup. Create the following files on the Observium server:

./includes/polling/applications/apache_procs.inc.php


<?php
if (!empty($agent_data['app']['apache_procs']))
{
$rrd_filename = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-apache_procs-".$app['app_id'].".rrd";
list ($procs) = explode("\n", $agent_data['app']['apache_procs']);
if (!is_file($rrd_filename))
{
rrdtool_create($rrd_filename, " \
DS:procs:GAUGE:600:0:125000000000 ");
}
rrdtool_update($rrd_filename,  "N:$procs");
}
?>

./html/includes/graphs/application/apache_procs.inc.php


<?php
if (!empty($agent_data['app']['apache_procs']))
{
$rrd_filename = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-apache_procs-".$app['app_id'].".rrd";
list ($procs) = explode("\n", $agent_data['app']['apache_procs']);
if (!is_file($rrd_filename))
{
rrdtool_create($rrd_filename, " \
DS:procs:GAUGE:600:0:125000000000 ");
}
rrdtool_update($rrd_filename,  "N:$procs");
}
?>

./html/pages/device/apps/apache_procs.inc.php


<?php

/**
* Observium Network Management and Monitoring System
* Copyright (C) 2006-2014, Adam Armstrong - http://www.observium.org
*
* @package    observium
* @subpackage applications
* @author     Adam Armstrong <adama@memetic.org>
* @copyright  (C) 2006-2014 Adam Armstrong
*
*/

$app_graphs['default'] = array('apache_procs' => 'Processes');

// EOF

Conclusion

If you have carried out the steps above correctly then you should see a graph populate underneath the ‘Apps’ tab in Observium for the host you’re monitoring (it can take 5 minutes for the broken graphs to appear and a further 10 minutes for any sort of useful data to appear on the graph, so be patient!)

If you don’t see anything under the apps tab or your graphs are broken after 10 minutes then you can debug the process by running the poller manually with the debug switch as such:


# cd /opt/observium
# ./poller.php -h <monitored_host> -d

The above command will show lots of output including any database queries it executes relating to MySQL. It should also show the creation or update of the RRD files relevant to your various agent modules.

If everything is working, you should see something similar to below (please note there is significant amounts of data on this graph already as it had been populating for a good few hours):

apache_procs