Code Project: Network Map Webpage, Making it Better

I wrote a bit about my Network Map Webpage recently. It’s part of a larger home dashboard project I’m working on, but as part of that I’ve updated things a bit to make them more streamlined and easier to use. The biggest problem with the page as it was originally coded is that it shows everything. I’ve cycled most of my regularly used electronics onto the network so they could be captured by an arp scan, though not all of them are on all the time. For example, I still have a Raspbery Pi and Arduino set up to capture temperature data. I also have several Next Thing CHIP devices, though Next Thing has gone out of business. In total, between my IOT stuff and laptops, phones and tablets and the duplicate IPs from the network extender, I have 55 devices in the raw table.

So I set out to make this more manageable at a glance. My original query in my PHP code looked something like this:

SELECT ip, arpscans.mac, arpscans_known_macs.device_name, arpscans_known_macs.device_description, last_seen, device_owners.user_name FROM arpscans LEFT JOIN arpscans_known_macs on arpscans_known_macs.mac = arpscans.mac LEFT JOIN device_owners on = arpscans_known_macs.device_owner ORDER BY ip

By slipping in “WHERE last_seen >= NOW() – INTERVAL 5 MINUTE” just before ORDER BY, I can make the code return only currently connected devices. The ARP scan runs every 5 minutes, anything that has a last seen time stamp within 5 minutes is assumed to still be attached. This interval could be shorted to almost real time, but I don’t really need that much of a check.

I can also view all disconnected devices with a simple change of the above command, making it “WHERE last_seen <= NOW() – INTERVAL 5 MINUTE”. This wouldn’t work if I were still keeping historical data, but I essentially only capture the last seen data for any device. Essentially what this does is return everything not seen in the last 5 minutes.

I also broke out my PHP code that builds my table from my query into it’s own PHP function. This was I could set the variable $SQL for the active devices, call the function to build the table, then set $SQL for inactive devices and build a second table, under the first.

I immediately scrapped this, because it was ugly. Plus, sometimes I do want to see “everything”.

Enter some GET calls and an if/else statement.

	if($_GET['show'] == "active") {
	// SQL for selecting active devices
	$tabletitle="Active Devices";
	$sql = "SELECT ip, arpscans.mac, arpscans_known_macs.device_name, arpscans_known_macs.device_description, last_seen, device_owners.user_name FROM arpscans LEFT JOIN arpscans_known_macs on arpscans_known_macs.mac = arpscans.mac LEFT JOIN device_owners on = arpscans_known_macs.device_owner WHERE last_seen >= NOW() - INTERVAL 5 MINUTE ORDER BY ip";
	elseif($_GET['show'] == "inactive") {
	// SQL for selecting active devices
	$tabletitle="Inactive Devices";
	$sql = "SELECT ip, arpscans.mac, arpscans_known_macs.device_name, arpscans_known_macs.device_description, last_seen, device_owners.user_name FROM arpscans LEFT JOIN arpscans_known_macs on arpscans_known_macs.mac = arpscans.mac LEFT JOIN device_owners on = arpscans_known_macs.device_owner WHERE last_seen <= NOW() - INTERVAL 5 MINUTE ORDER BY ip";
	else {
	// SQL for Selecting all devices
	$tabletitle="All Devices";
	$sql = "SELECT ip, arpscans.mac, arpscans_known_macs.device_name, arpscans_known_macs.device_description, last_seen, device_owners.user_name FROM arpscans LEFT JOIN arpscans_known_macs on arpscans_known_macs.mac = arpscans.mac LEFT JOIN device_owners on = arpscans_known_macs.device_owner ORDER BY ip";

Basically, if nothing, or a random string is passed by the URL variable “show”, then it goes to the end, and displays everything when accessing the page at index.php. If it passes index.php?show=active, it sets $SQL for showing active devices and if it gets index.php?show=inactive, it shows inactive devices. It also sets a variable called $tabletitle which is just echoed out into some header tags. I then added links across the top of the page to each of these filters.

This allows for a quick and easy toggle of which data is pulled and displayed.

Additionally, I updated the way the Add Device form works. Previously, the form would fill in the MAC, a Device Name and a Device Description, then it would POST to another PHP page that would insert the data into the table, then forward on back to the index page with a header redirect. I’m not going to get into too much detail on it here, but I also integrated the Network Map into my dashboard framework with a header, navigation, sidebar, and footer. It also uses a table based navigation system, so in order to view the network map, I am hitting “index.php?page=4”. Pages basically all need to be wrapped in this structure to work properly, so in order to make things flow better, the Add Device form now POSTs back to the Network Map page itself, which checks to see if the POST variables are set, and if they are, it inserts the new information, before pulling the table.

This also meant slightly altering my page calls to look for “index.php?page=4&show=active” and “index.php?page=4&show=inactive”.

Eventually I want to move the Add Device form to appear at the top of the page, so the whole thing is all handled in one single page.

Lastly, I made up a quick block of code in it’s own page, that simple counts and displays the number of currently connected devices on the network. This block is embedded on the front page of my Dashboard Framework and links to the full Network Map page. The general idea on the Dashboard is to have widgets like this that show quick glance information, with links to detailed information.

I have not built a lot of them yet, but one of the others I have built works somewhat similar to the ARP scanning system. A script makes a call to my TT-RSS instance for each of the segmented accounts I have, then dumps the unread count into a table on the server. The widget shows how many unread articles each topic/account has. I am still really bad about only actually reading the Basic feed (mostly Toys and Video Games).

But I will get into the Dashboard Widgets thing a bit more in a future post probably.

Code Project: Network Map Webpage

I want to start off by saying, there isn’t going to be a ton of code here, and if there is code, it’s going to be super dirty. I’m fairly good at making code for “private use” that is pretty insecure, and not so great at code that’s scrubbed up and user friendly to distribute to others.

I’ve been working a bit on some local code projects, specifically for my little private “Dashboard” that runs on my file server. One project I’ve wanted to try for a while is a dynamic network tracker tool. I’ve looked into some options available, and they all seem to run as a plug in for some complicated 3rd party analytics software that often has some goofy complicated set up procedure that’s beyond “apt-get” or even just dumping a bunch of files in a web server directory.

This project is both kind of simple and not. It was fairly simple in set up and execution, but it’s somewhat complex in design. The first job was getting a list of currently connected devices on the network. This is easily done via the command line with an arp-scan request.

sudo arp-scan --localnet

The output of which looks something like this:

Using a pipe, I can shove all of this into a text file, which contains everything above.

sudo arp-scan --localnet | scan.txt

The trick is, how to display this output on a webpage. One way would be to pull it from a database. Pulling data from MySQL is pretty easy, dumping it to a pretty looking table is also easy. The harder part is getting the output of arp-scan to MySQL in a useful manner.

This is where Python comes into play. I am sure there are other methods or languages available, but like Python, and mostly know how to use Python. Essentially, I wrote a script that would open the file, scan.txt, that was created above. I am only concerned with lines that contain IP addresses, so I used the function “is_number()” to check if the first character of each line is numeric, if it is, it runs through a couple of operations.

Firstly, the output of arp-scan is tab delimited, so I can use the “split” function on “\t”, and dump the result into an array. This gives me an array of the IP address, MAC address, and Manufacturer. This sticks a new line in with the Manufacturer, so I did a “replace” on \n in the third item of the list. Lastly, I wanted the IPs to be uniformly formatted, so I write a little function that would add in leading zeros to the IP octets.

Finally, the Python builds an SQL statement from the line’s list, and make a call to the server to insert the values. A modified version of this code that just displays the resulting SQL commands instead of executing them is below.


# Open a file

def is_number(s):
                return True                                                         except ValueError:
                return False

def format_ip(ipstring):
        octets = ipstring.split(".")
        for i in octets:
        return octets[0]+"."+octets[1]+"."+octets[2]+"."+octets[3]
        #return ipstring

def add_zero(shortstring):                                                          return "0"+shortstring

import MySQLdb

mydb = MySQLdb.connect(

mycursor = mydb.cursor()

fo = open("scan.txt", "r")
#print ("Name of the file: ",

# read each line of the list
for line in fo:
        #check for lines that contain IP addresses
        if is_number(line[0]):                                                              #Convert lines into list
                line_list = line.split("\t")
                #remove line delimitors
                #Make IP Octets 3 digits
                line_list[0] = format_ip(line_list[0])
                SQL = "INSERT INTO arpscans (ip, mac, mfg) VALUES ("+line_l$                print SQL                                                   

It’s not super pretty, but it was a quick way to make sure everything came out looking correct. The table I used is called “arpscans” and contains columns called, “ip”, “mac”, “mfg”, and “last_seen”. The time stamp is an automatically generated time stamp.

I then created a shell script that would run the arp-scan piped into scan.txt then runt he python script. I set up this script in the root crontab to run once every half hour. Root is required to run the arp-scan command, so my user crontab wouldn’t cut it. Everything ran fine when I manually did a run of the script using sudo. The PHP on the other end out output the latest values based on the time stamp to a webpage.

This is where I ran into my first major hurdle. The script wasn’t running in cron. After a lot of digging and futzing, I found that basically, when cron runs the script, it works off of different environmental variables. I had to specify in ,y bash file, specifically where each command existed. The end result looks something like this:

#!/usr/bin/env bash
/usr/sbin/arp-scan --localnet > /home/ramen/scripts/arp_sql/scan.txt
/usr/bin/python /home/ramen/scripts/arp_sql/

Eventually the scan was running and posting data automatically as expected. After a few days, I ran into my second major issue. There was simply put, way too much data for my crappy old “server” to handle. The webpage slowed to a crawl as the table contained something like 9000+ entries. It’s possible and likely that my query was also rubbish, but rather than stress more figuring it out, I modified all of the code again.

Instead of adding a new entry for every MAC address every scan, I changed it to check if there already was an entry, and simply update the last_seen time. I had originally designed the system with the idea of getting legacy data for attached devices, but decided I only really cared about a generic history.

The new webpage table now displays all devices, current and previously seen, with the last seen date.

A few issues came up on the output end as well, though none of them were super hard to correct. One, I wanted a way to sort the table by clicking the headers. There are several scripts you can toss in your code to do this online.

I also wanted more data about each device, so I added a form where I could fill in more data about each device. Specifically, the network name, if there was one, a description of what the device is, the User of the device (which family member or if it’s just a network device). This also checks and updates based on MAC address.

I also ran into an issue with MAC addresses and my Network extender. When devices are connected to the Network Extender, the first half of the MAC is replaced with the first part of the Extender’s MAC, though they retain the last half. I may eventually write some code to detect and merge these entries, but for now, I’ve simply been labeling them in the description as “(Extender)”, so I know it’s the same device on the other connection.

The final end result looks something like this:

I used to have the network super organized before I moved, but the new router doesn’t work nicely with my Pi DHCP server, so I have not gotten things quite as nicely sorted as I would like. Everything in the picture is sorted, but above .100, it’s a mess. I also can’t assign IPs to some devices at all, like the DirecTV gear or my Amazon Echos, which is really annoying.

One of my future projects will hopefully correct this, as I want to put a second router on the network with DD-WRT, between the ISP gateway and everything else.

Overall, it’s been a fun little exercise in coding that combined a lot different techniques together in a fun way.

Raspberry Pi Project 02 – Ad Blocking DNS and DHCP

Raspberry_Pi_LogoI’ve been at a bit of a loss for a while on what to do with my RaspberryPi B+.  I have a B+ and two RaspberryPi B 2s.  The B2s are so much more useful and powerful than the B+ that the poor B+ has kind of been tossed to the wayside.  I was using it as a server for ZNC, an IRC Bouncer.  When I set up my milti-monitor set up, I tried running X-Chat on the B+ with ZNC but the lag was too much.  I’m not sure if the load was from XChat or from ZNC or from Synergy, but the short of it is, the B+ wasn’t useful to my array of machines and needed a headless activity assigned to it.

Sure, I could let it continue to run ZNC, but dedicating two Pis to IRC seems like overkill, so I moved ZNC to the B2 that I run Xchat on and freed the B+ for whatever I wanted.

Pi-Hole for DNS

I started out by setting up Pi-Hole.  It has a fancy name and is simple to install, but at the core, Pi-Hole is just a DNS server that filters sites based on some predefined public lists.  Specifically, it filters ads.  There is a whole debate to be made on the pluses and minuses of ad filtering, but it is what it is and I’m not getting into that debate here.  Pi-Hole has a white list anyway, for “good ads”.  Mostly, it’s a way to filter on mobile, where it’s not easy to install an ad blocker.

Installing Pi-Hole is as simple as entering the following into the command line over SSH or even on the RaspberryPi itself:

sudo curl -L | bash

Though you will also need to set up the Pi with a static IP and then later point your computer’s DNS records to the IP of the Pi.

On a bit of a side note, DNS is Domain Name Server.  The easy analogy is that it is a phone book to the Internet.  Servers where websites live all have IP addresses assigned in various ways, but in order to keep things simple for humans, these servers get friendly names, like or  The DNS is where your computer looks up “” to find out what IP address it’s located at.  In this case, ads come from specific servers and providers, often not related to the core domain.  So may serve ads from  With Pi-Hole, your computer looks up both domains when you connect to, but the RaspberryPi Pi-Hole simply replies “I don’t know where is located”.  You computer then fetches what it can from, but displays nothing from


Sort of tangentially related, I recently changed ISPs.  I went from a 3Mbps DSL to 50Mbps Cable modem.  I keep my home network crazy organized since there are a ton of devices on it.  For every person in my house there are something like 4 devices, and this doesn’t count game consoles, media devices, or my own little IOT devices.  On the old modem, I set up MAC assignments on the router so that wireless devices like my phone, where I can’t assign a static IP, get assigned a specific IP.  I went through and set all of this up on the new modem, but none of the assignments seem to have taken.  I’m not entirely sure why, I have theories, but I’ve instead decided to use this as an excuse to re-purpose my B+ and to learn more about managing a DHCP server.

So what is DHCP.  DHCP stands for Dynamic Host control Protocol.  I mentioned earlier that DNS was the way to connect the firstly domain name to an unfriendly IP address.  DHCP assigns those IP addresses.  Some static devices like routers or desktop computers that never go anywhere can pick and self assign an IP but things like phone that move between networks need to be given a temporary IP address as they come and go.  DHCP handles this.  Every device on a network has a unique IP address, though small networks like the one in your home, all tend to share IP ranges since to the outside world they appear to be one node/thing/device.  DHCP can be used to give out reserved “static like” IPs to these roaming devices though based on the device’s MAC Address.  A MAC Address is a device unique identifier.  For example, when the DHCP server or router sees a specific MAC, it can say “I know you, you get IP”.  When a strange device shows up, it will simply say “Here, this time you are, which is the next free IP in my pool of IPs”.

The set up for DHCP is a bit more involved than for Pi-Hole.  There is a detailed guide below but I’ll run through the cliff notes version.  There is a lot more that can be done but I’m not going to get super detailed here.  This is essentially for a simple home network with one subnet.  The first steps are to set up and configure the server, it will likely error out in the middle, just keep going with it.

sudo apt-get install isc-dhcp-server

sudo nano /etc/dhcp/dhcpd.conf

This is where it gets a little trickier.  You will need to edit the options in the config file.

option domain-name “your_domain”
option domain-name-servers

Your Domain can be more or less anything really, it’s often just “Workgroup” by default in Windows.  It should be all one word though.  Domain Name Servers are specific however.  If you are using Pi-Hole, set the IP of the Pi running Pi-Hole into this list.  You can add additional DNS servers with a comma between each one, some suggestions,, and are Google’s DNS servers.  OpenDNS uses and  It’s a good idea to have more than one in case there are issues.

The only other thing you NEED to set is the ip information and range found under

subnet netmask
{ range;
option routers;

You will need to replace the IP information in this block.  the subnet should be the first three octets of your network followed by a .0.  The most common ones in a home network are,, and  The range is the range of IPs to give out.  If you are going to use static IPs, i would recommend setting this range beyond your static IP ranges.  For example, I have a spreadsheet blocking off reservations through, I added some extra for my router to serve as a backup DHCP server and set the range on the RaspberryPi as  This means, when a device connects, it will be assigned an IP starting at the next available at .175.

The final option is routers, chances are really good that you have only one, and chances are the IP ends in .1 or .254.  This will assign the router IP so that devices know how to get out to the Internet.

If you want to ensure your Pi DHCP server is assigning IPs over anything else on the network (ie your router) you will need to find the line that reads “If this DHCP is the official DHCP server…” and uncomment the line “authoritative”.

Finally at the bottom, there are blocks to assign static IPs using a format like:

hardware ethernet MACADDRESS;

An easy way, if you are running Windows, to find out MAC addresses of devices is to run Netscan.  This will give you a list of everything connected to your home network.  There are a few ways to decipher which IP/MAC is which.  Some will have the MAC printed on them, often near the network port.  In the case of phones or tablets, it’s simplest to simply disable the WiFi or turn them of and rescan to see which disappears.  Some may show up with names you may recognize.  You can also sometimes search for the first 3 sets of numbers (IE 45:3b:a3) which is manufacturer specific to decipher that “That’s a Sony device, the only Sony thing I own is the Blu Ray player”.

Setting reserve assignments is entirely optional.  The main purpose is to better organize your home network.  If you only have a half dozen devices, it really probably isn’t necessary at all.

Once you are done editing, CTRL+O (for Output) to save the file, and CTRL+X to close it. If you get an error that you can’t write the file then you forgot to do a”sudo” and you will need to do it all over again.

Wrap Up

Things are not quite finished yet.  You will need to start the DHCP server, since it error-ed out earlier.  You can do this using the following commands:

service isc-dhcpd-server stop
service isc-dhcpd-server start

HOWEVER, I still got an error when I did this.  A little searching and I found a similar issue and fix which I used.

sudo pico /etc/default/ifplugd

Then change this:

ARGS=”-q -f -u0 -d10 -w -I”

To match this:

ARGS=”-q -f -u0 -d10 -w -I”

then reboot and after the reboot start the server with:

sudo reboot

service isc-dhcpd-server start

Everything should be working now.  You can run Netscan later and see if IPs are being assigned into the range you chose.  This may not happen immediately since IPs have a lease time and devices may hold on to the old IP for a bit.



Configuring the Raspberry Pi as a DHCP Server under Raspbian Wheezy

How To : Use The Raspberry Pi As A Wireless Access Point/Router Part 3…B!

Tools I Use: Netscan and Fing

I wanted to do some occasional posts on some tools I use for various technical tasks.  Partially just to suggest some useful stuff, partially so I have some posts to reference anytime I reference said stuff.

I wanted to start off with Netscan and Fing, which serve the same basic purpose on two different platforms.  Both of these tools will scan the local IP range and return a list of every device connected to the network.  Netscan is what I use on windows, Fing is what I use on Android.

I use these tools very frequently, several times a week on average.  So what use is scanning the local network anyway?  I have two main uses, though both come down to Device Discovery.

Firstly, basic device discovery.  I’ve hooked something new to the network and I need to access it.  A lot of what I connect is headless with no easy way of discovering the IP aside from a scan.  An Arduino, a Raspberry Pi, a networked Webcam, all of these things need to be found once connected.  The scan is also useful for getting the MAC address of devices on the network.  The IP is dynamic on a network by network basis, a MAC address is a unique identifier.  Knowing the MAC address is useful for building firewall rules and setting up static IPs assigned by the router for devices like phones or laptops where assigning IPs on the device can get hairy.

The other reason for doing a network based scan is intrusion detection.  Generally speaking, I don’t expect to see hackers or anything on my home network.  This is more for checking things like “if my kids’ devices are connected” or occasionally if one of my kids has a new device borrowed or whatever that I am not aware of on the network.

Ultimately I want to set up a little network monitoring system on a server to do these sorts of checks in real time but both of these tools have served me well for years as doing the job quickly and simply.

Both are also useful for poking around foreign networks.  You can see what machines are on an open WiFi hotspot and see if they have any open shared files.  Though some open hotspots are smart enough to block such scans.

Why Can’t I Hold All These Devices?

There are 5 people in my household.  At this point, each of these people has at least a laptop and a tablet and the majority of them have a handheld smartphone style device (only one is actually a phone with data).  There are several game consoles and media devices, a couple of additional desktops, some security cameras, my Pi projects, etc etc.  A quick rundown gives me 33 Devices, though there are more that are not frequently active.

At some point, it became necessary for me to take control and actively manage my home network.  I was getting issues with double assigned IPs from DHCP, I have files on shared drives which need to have static IPs, I needed to implement security and filtering on the network for the kids to keep them from doing things they shouldn’t be doing online as well as tracking usage.

It also helps with security because I know what devices are online and if there are “outside” devices on the network.

So, a quick basic rundown of networking.  Every device, from PCs to Xboxes to iPods, get a unique IP address, most commonly on home networks this will be 192.168.1.XXX.  This is ahow data is pushed around, data has a header that says “I need to go to 192.168.1.XXX, where is that?” and routers and switches push this data around appropriately until some device says “Here I am, send it to me”.  This is really really generalized but it’s the basic idea.

These IPs can often be set up to be static (always the same) on the device, it can be assigned randomly from a pool by the Router (DHCP) or it can be assigned to be static by the Router based on the Device MAC Address.

Every device also has a MAC address.  MAC Addresses are unique to the device interface.  Think of it as a fingerprint.  I say “Device Interface” because if a machine has multiple network interfaces, say, WiFi and a Network cable jack, these will have different MAC addresses despite being one device.  In most routers, you can set up a table of MAC Addresses and tell the router “If you see this MAC, assign it the IP X”.

This is really useful for things like Laptops, Phones and Tablets.  For a machine like a desktop PC or a server that never leaves the network, it may be better to assign the IPs on the device, that is, the Device connects and says “i want to be assigned IP YYY”.  If you have a Wireless device assigned with a static IP, it can cause trouble when that device travels out of network, the static IP may not be available at say, your friend’s house or a coffee shop.  The remote location may use a different IP scheme, they may have their Router assigned to a different IP, there may be another device already using that particular IP.

So why assign IPs?

File servers really need static IPs.  If other devices are connecting to another machine to get say, photos, that other machine, the server, needs to always be in the same place.  Imagine how hard it would be to go to your friend’s home if their home was always in a different location and every building looked identical.

This also avoids IP conflicts.  This is less common since the Router is supposed to not double assign IPs but occasionally if a device disconnects and reconnects while another device is reconnecting, the IP may accidentally become double assigned, which means those data packets go nowhere.  This would be like trying to go to your friend’s address but there are two homes with the same address across the street from each other.

Assigned IPs is also great for security.  Limiting the IP range of DHCP, or limiting the number of devices that can connect keeps the network from getting over loaded by random people, though with a WiFi password this wouldn’t happen anyway.  You could also limit the capabilities of IPs connected through DHCP.  With scanning software you can also know at any time what is connected to your network.  It can also help diagnose issues.  If your unable to get your Wii to watch netflix, you can run a scan and if everything shows up but the Wii, you know the issue is probably on the Wii itself.

It’s also been good for my own experience in better learning methods for managing small networks and configuring the router.  I started off with a list of devices in a spread sheet.  I then gathered all the MAC addresses through a combination of scanning the network or checking the device itself.  Most devices will show you the MAC address in the settings somewhere and if all else fails it’s often printed on a sticker on the back side.

i then sorted out blocks of IPs based on device and sorted everything into these blocks.  This helps organize things.  The only thing that changes is the 4th octet of the IP, so everything is 192.168.1.XXX.  From here I use the following schema:

  • 01-09 = System Devices, the Router, the Wifi Access point, my NAS.
  • 10-20 = Game Consoles and media Devices
  • 21-29 = Desktops, of which there are 4.
  • 30-39 = Handheld Devices belonging to the Kids
  • 40-49 = Laptops
  • 50-59 = My Devices
  • 60-69 = Reserved for IP cameras
  • 70-79 = Reserved for Raspberry Pis and other Internet of Things style devices.
  • 100+ is used for DHCP assignments

This can be modified based on personal needs of course.  The idea is essentially that if nothing shows up under the 100+ IP range, I can know at a glance that nothing unknown is attached to the network.