Raspberry Pi

Linux & Open Source

Dead Memory Cards and Using Docker

Ramen Junkie No Comments

More often that it feels like it should, something in technology breaks or fails. I find that this can be frustrating, but often ultimately good, especially for learning something new, and forcing myself to clean up something I’ve been meaning to clean up. I have a Raspberry Pi I’ve been using for a while for several things as a little web server. It’s been running probably for years, but something gave out on it. I’m not entirely sure it’s the SD card or the Pi itself honestly, because I’ve been having a bit of trouble trying to recover through both. It’s sort of pushed me to try a different approach a bit.

But first I needed a new SD card. I have quite a few, most are “in use”. I say “in use” because many are less in use and more, underused. This has resulted in doing a bit of rebuild on some other projects to make better use of my Micro SD cards. The starting point was a 8 GB card with just a basic Raspbian set up on it.

So, for starters, I found that the one I have in my recently set up music station Raspberry Pi is a whopping 128gb. Contrary to what one might thing, I don’t need a 128gb card in my music station, the music is stored on the NAS over the network. It also has some old residual projects on it that should really be cleaned out.

So stuck the 8GB card in that device and did the minor set up needed for the music station. Specifically, configure VLC for Remote Control over the network, then add the network share. Once I plugged it back into my little mixer and verified I could remote play music, I moved on.

This ended up being an unrelated side project though, because I had been planning on getting a large, speedy, Micro SD card to stick in my Retroid Pocket. So I stuck that 128GB card in, the Retroid and formatted it. This freed up a smaller, 32GB card.

I also have a 64GB that is basically not being used in my PiGrrl Project I decided to recover back for use. The project was fun, but the Retroid does the same thing 1000x better. So now it’s mostly just a display piece on a shelf. Literally an overpriced paperweight. I don’t want to lose the PiGrrl configuration though, because it’s been programmed up to work with the small display and IO Control Inputs. So I imaged that card off.

In the end though, I didn’t end up needing those Micro SD cards though, I opted for an alternative option to replace the Pi, with Docker on my secondary PC. I’ve been meaning to better learn Docker, though I still find it to be a weird and obtuse bit of software. There are a handful of things I care about restoring that I used the Pi for.

  • Youtube DL – There seem to be quite a few nice Web Interfaces for this that will work much better than my old custom system.
  • WordPress Blog Archives – I have exported data files from this but I would like to have it as a WordPress Instance again
  • FreshRSS – My RSS Reader. I already miss my daily news feeds.

YoutubeDL was simple, they provided a nice basic command sequence to get things working.

The others were a bit trickier. Because the old set up died unexpectedly, The data isn’t easily exported for import, which means digging out and recovering off of the raw database files. This isn’t the first time this has happened, but its a lot bigger pain, which isn’t helped by not being entirely confident in how to manipulate Docker.

I still have not gotten the WordPress archive working actually. I was getting “Connection Reset” errors and now I am getting “Cannot establish Database connection” issues. It may be for nothing after the troubles I have had dealing with recovering FreshRSS.

I have gotten FreshRSS fixed though. Getting it running in Docker was easy peasy. Getting my data back, was… considerably less so. It’s been plaguing me now when I try to fix it for a few weeks now, but I have a solution. It’s not the BEST solution, but it’s… a solution. So, the core thing I needed were the feeds themselves. Lesson learned I suppose, but I’m going to find a way to automate a regular dump of the feeds once everything is reloaded. I don’t need or care about favorited articles or the articles contents. These were stored in a MySQL database. MySQL, specifically seems to be what was corrupted and crashed out on the old Pi/Instance because I get a failed message on boot and i can’t get it to reinstall or load anymore.

Well, more, I am pretty sure the root cause is the SD card died, but it affected the DB files.

My struggle now, is recovering data from these raw files. I’ve actually done this before on a surver crash years ago, but this round has lead to many many hurdles. One, 90% of the results looking up how to do it are littered with unhelpful replies about using a proper SQL dump instead. If I could open MySQL, I sure as hell would so that. Another issue seems to be that the SQL server running on the Pi was woefully out of date, so there have been file compatibility problems.

There is also the issue that the data may just flat out BE CORRUPTED.

So I’ve spun up and tried to manually move the data to probably a dozen instances of MySQL and MariaDB of various versions, on Pis, in Docker, on WSL, in a Linux install. Nothing, and I mean NOTHING has worked.

I did get the raw data pulled out though.

So I’ve been brute forcing a fix. Opening the .ibd file in a text editor gives a really ugly chuck of funny characters. But, strewn throughout this, is a bunch of URLs for feeds and websites and well, mostly that. i did an open “Replace” in Notepad++ that stripped out a lot of the characters. Then I opened up Pycharm, I did a find and replace with blanks on a ton of other ugly characters. Then I write up this wuick and dirty Python Script:

# Control F in Notepad++, replace, extended mode "\x00"
# Replace "   " with " "
# replace "https:" with " https:"
# rename to fresh.txt

## Debug and skip asking each time
file = "fresh.txt"
## Open and read the Log File supploed
with open(file, encoding="UTF-8") as logfile:
    log = logfile.read()

datasplit = log.split(" ")
links = []

for each in datasplit:
    if "http" in each:
        links.append(each)

with open("output.txt", mode="w", encoding="UTF-8") as writefile:
    for i in links:
        writefile.write(i+"\n")

Which splits everything up into an array, and skims through the array for anything with “http” in it, to pull out anything that is a URL. This has left me with a text file that is full of duplicates and has regular URLs next to Feed URLS, though not in EVERY case because that would be too damn easy. I could probably add a bunch of conditionals to the script to sort out anything with the word “feed” “rss”, “atom” or “xml” and get a lot of the cruft removed, but Fresh RSS does not seem to have a way to bulk import a text list, so I still get to manually cut and paste each URL in and resort everything into categories.

It’s tedious, but it’s mindless, and it will get done.

Afterwards I will need to reset up my WordPress Autoposter script for those little news digests I’ve been sharing that no one cares about.

Slight update, I added some filtering ans sorting to the code:

# Control F in Notepad++, replace, extended mode "\x00"
# Replace "   " with " "
# replace "https:" with " https:"
# rename to fresh.txt


## Debug and skip asking each time
file = "fresh.txt"
## Open and read the Log File supploed
with open(file, encoding="UTF-8") as logfile:
    log = logfile.read()

datasplit = log.split(" ")
links = []

for each in datasplit:
    if "http" in each:
        if "feed" in each or "rss" in each or "default" in each or "atom" in each or "xml" in each:
            if each not in links:
                links.append(each[each.find("http"):])

links.sort()

with open("output.txt", mode="w", encoding="UTF-8") as writefile:
    for i in links:
        writefile.write(i+"\n")
IOT Projects

Battery Testing with Raspberry Pi

Ramen Junkie No Comments

Recently I purchased a portable phone battery on clearance. I want to use it to modify my defunct Pokemon Go Gotcha band, which no longer holds a charge, to run off the battery pack. I realized that a battery pack could also be useful for powering other small electronics, such as the Raspberry Pi.

The problem is, I wanted to get an idea of how long the battery would last powering the Raspberry Pi. Figuring this out isn’t really all that hard. The tricky part is that I can’t stand over the Pi and watch it until it dies.

So instead I set up a simple cronjob task to do the job for me. I had a spare SD card, so I threw a basic fresh install of Raspbian on the card. I configured SSH and WiFi, then did a quick run of “crontab -e”, then droped the following at the end.

*/5 * * * * date >> /home/pi/date.txt

Simple.

Every 5 minutes, the Pi will now write the Date and Time to the file “date.txt”. It will do this until it can’t which would be after the battery dies, killing the Pi.

After charging the battery pack over night, I stuck the Pi on it and left it.

After checking back periodically, when I found the Pi was dead, I plugged it into a regular power source to retrieve the data. The result were both better than I had hoped, and not as great as I had hoped. The Pi started spitting out Time Stamps again after I plugged it back in, so I ended up having to skim through the file to find the time gap. I went ahead and truncated the data down to hour stamps until I came across the time jump from when the Pi had died and when I had plugged it back in.

So it turns out that the Battery pack will drive power for the Pi for around 12 hours. I also want to test this under a bit of a load and test how long it will power an Arduino writing to a remote database.

IOT Projects

Raspberry Pi Rack

Ramen Junkie 1 Comment

Even before buying any Raspberry Pi computers, I had this sort of vision for neat rack to put several of them in. The idea evolved a bit as the logistics of it got ironed out but after picking up a second Pi (and planning to get a third), I went and built the thing.

Raspberry Pi Rack

Sorry for the mediocre pictures, I didn’t think to take photos until things were running and I didn’t really want to unplug it. It currently holds a Raspberry Pi B+ and a Raspberry Pi2 B, the B+ is running a ZNC server to keep me connected to several IRC channels and the Raspberry Pi 2B is running a small OpenSIM server, though I’ll probably re-purpose it since in testing performance seems to top out at around 2000 Prim cubes and 4 Avatars. The instance I’ve got running on my VPS is much more suitable. On the other hand, it could be a good place to “archive” builds.

Raspberry Pi Rack

The build itself is pretty straight forward. I used (roughly) the following materials picked up at Lowes:

  • 3 pieces of 10×8 plexiglass cut to 5×7 size.
  • 6 6″ #8 thread rods.
  • 8 (2 packages) of #8 rounded caps.
  • 3 packages of #8 nuts
  • 6″ Micro USB cables from Amazon
  • Cat 5 Cable cut and crimped as needed
  • 1 Netgear Switch I had already
  • 1 Choetech 40W Smart power from Amazon.
  • The smaller screws holding the Pis in are 4mm i believe, and it took two packages of them (16 screws and 32 nuts).

The hardest part was working with the Plexiglass pieces. I could have cut it with the Dremel but the edge would have been all crooked for sure. Instead I scored and snapped them, which did leave a jagged edge on one edge but it could easily be sanded off. On a starting note, the Plexiglass comes with a plastic protective layer, this should be left ON until final assembly to protect the clear surface as much as possible.

Anyway, to cut the sheets down, I used a square and a box cutter to but a line where I wanted the break to be on one side, then clamped the sheet down on the edge of the workbench between the surface and a 2×4. The 2×4 was mostly to help protect the surface of the plexiglass from the clamps. The cut edge should be on the up side, right at the edge of the surface and the 2×4 (or whatever) at the edge on top. The Plexiglass should snap more or less cleanly off. I had some small chips left hanging on the shorter 1″ breaks to shorten the shelves to 7″ from 8″. I used a small hammer while the pieces was still clamped down to chip these down a bit.

The real trick was drilling the holes. In the end, I found the best results came from clamping all of the plastic sheets together between some boards and drilling them all at once. There should be a piece of wood completely covering the bottom of the drilling spot to help support the plexiglass as the drill penetrates through.

I had some trial and error trying to drill my sheets separately, so the holes are not perfect. For the mounting holes I laid the Pi itself where I wanted it and uses a pen to mark the holes. I probably could have done a bit better with a paper guide though.

Assembly was pretty easy, just a lot of tedious screwing of nuts as the thread rods were fed through the holes. Each shelf takes at a minimum, one nut above and below to hold them in place and the rounded caps go on the top and bottom. The power supply and switch are not mounted, they are simply sandwiched in place between the shelves, though the supporting long bars were positioned around the Switch to prevent it from sliding left or right or backwards.

The Power supply I picked because it has two standard wall outlets on one side. I used the Dremel to cut holes in the top sheet to allow these plugs to be accessible. I wanted the unit to be as self contained as possible, these plugs give me a place to plug the Switch in. It’s important when choosing a power supply that it has enough power on all ports to power a Pi. A USB HUB doesn’t work since it will distribute power across all the ports. Several ports I looked at had 2 “high output” ports for iPads and iPhones but the rest were lower output.

The USB power cables were 6″ jumpers I found on Amazon and the CAT 5 cable were small jumpers I made myself with ends and a crimper. When I add the back two Pis I’ll need different cables though, likely cables with a 90 degree connector and definitely longer ones.

Raspberry Pi Rack

The whole package sits nicely on top of my desktop box next to my Synology NAS.

Coding

Security Phase 2 – Doors and Windows (Planning)

Ramen Junkie No Comments

I just wanted to start off by saying, this isn’t a how to at all, it’s more some general ideas I’ve been looking over for how to add door and window security to my recently set up video monitoring system.  Basically, I want a log of when and if the doors and windows are opened and closed.  I’d prefer not to run a bunch of wires, so wireless sensors are in order.

What I ultimately want is for an indicator light on a webpage to change based on the status of the door or window, a timestamp log to be created and possibly for some sort of email or text alert to occur.  ULTIMATELY I may even use this whole project as an excuse to finally develop my Phone app skills and build an app that I can view it all on, but that’s farther down the road.  A basic web page is fine for now, and it’s something I can manage.

My worry with wireless sensors was that I’d have to constantly change the batteries.  Until I realized that it would be trivial to design the sensor so that it only transmits when the window is open, and while I am not building my own sensors (I probably could) I imagine the makers of such sensors realize that you could save a ton of battery life by designing the sensors to only work when open.  I did a bit of searching on Amazon and found a6 pack of sensors for less than ten dollars.  It appears that each sensor has it’s own receiver/indicator which emits an audible tone.

This is actually probably alright for my needs.

While I am not an Electrical Engineer, I am familiar with basic circuit flow and basic electronics.  I don’t have any of these sensors yet but I can make some basic assumptions based on what I know.   When the window/door is opened, some trigger is closed and the sensor starts transmitting some sort of radio signal.  The fob thing which is always listening will receive the signal and activate some sort of internal speaker. 

There are a couple of issues that need to be solved here.  One, while the sensors hopefully are designed to conserve battery power, the FOBs are probably not.  I also want the signal to go to a computer and not sound a mostly useless audible alarm.  The power issue isn’t a huge issue.  With the right transformer brick providing the proper voltage, it wouldn’t be hard to rewire the contacts to allow for the units to be plugged into the wall, all at once, off of one power supply.  They look to be small enough that they could be mounted inside an electrical box in a small array and wired for power.

A similar method can be used to transform the speaker signal into a trigger for some sort of simple IO board.  To activate the speaker, some sort of voltage is applied to the speaker, the speaker can easily be removed and the contact points that would previously have fed the speaker could be wired to am IO board.  When the voltage is supplied to the "speaker" it will instead be read by the IO board, triggering a signal that "the Window/Door is open."

The main puzzle I have right now is, what would work best for the IO board.  I see two options here, and both would function differently, and I may even require both.  I could go with an Arduino board or a Raspberry Pi.  Both of these options seem to have advantages and disadvantages, and there may even be a third option which corrects the disadvantages.

arduino

Let’s look at the Arduino.  The Arduino is ideal for dealing with the IO board.  It is, by design, meant to work with this sort of "hands on" electronics systems.  It even has build in contacts for working with two wire IO interfaces.  I don’t know anything about Process, the programming language of the Arduino, but I’ve been meaning to learn and I am capable of learning it.  The problem is on the output.  Arduino is not designed to work with PCs in any simple way.  From some quick research online, the Arduino would require a serial interface and likely special software running on the server to poll the Arduino at set intervals.  Which brings up another issue.  As near as I can tell, the Arduino is a "pull" interface, meaning the server would have to pull the status from the Arduino, as opposed to "push" where the Arduino would push the data to the server as it changes.

Raspberry_Pi_Photo

So there is also the Raspberry Pi.  The Raspberry Pi is a computer itself, which makes interfacing with the server simple and easy.  It runs a simplified OS but connecting the Pi to the network is simple and telling it to push even a text file status update to the server via FTP or some similar network protocol is trivial.  the issue comes in the need to read the sensors.  I’m not entirely sure the Raspberry Pi is even capable of reading an analogue IO interface right out of the box.  Some searching suggests it is possible through an additional interface board of some sort but not right out of the box.  Looks like there are a handful of GPIO points build into the PI.  The PI also feels like much more than is really needed for this simple application.

Its kind of a tricky decision, I’ve been wanting to play around with both the Pi and the Arduino so I may just pick up one of each and see what works.  Both are around the same price and the whole system in the end will end up costing around $50-$75, which is almost nothing.  The final step once the sensor data lands on the server is updating the web page but I already am familiar enough with building webpages and simple polling scripts that this step is the easy part.  Whatever I end up going with I’ll certainly document the results in the future.