Dead Memory Cards and Using Docker

More often that it feels like it should, something in technology breaks or fails. I find that this can be frustrating, but often ultimately good, especially for learning something new, and forcing myself to clean up something I’ve been meaning to clean up. I have a Raspberry Pi I’ve been using for a while for several things as a little web server. It’s been running probably for years, but something gave out on it. I’m not entirely sure it’s the SD card or the Pi itself honestly, because I’ve been having a bit of trouble trying to recover through both. It’s sort of pushed me to try a different approach a bit.

But first I needed a new SD card. I have quite a few, most are “in use”. I say “in use” because many are less in use and more, underused. This has resulted in doing a bit of rebuild on some other projects to make better use of my Micro SD cards. The starting point was a 8 GB card with just a basic Raspbian set up on it.

So, for starters, I found that the one I have in my recently set up music station Raspberry Pi is a whopping 128gb. Contrary to what one might thing, I don’t need a 128gb card in my music station, the music is stored on the NAS over the network. It also has some old residual projects on it that should really be cleaned out.

So stuck the 8GB card in that device and did the minor set up needed for the music station. Specifically, configure VLC for Remote Control over the network, then add the network share. Once I plugged it back into my little mixer and verified I could remote play music, I moved on.

This ended up being an unrelated side project though, because I had been planning on getting a large, speedy, Micro SD card to stick in my Retroid Pocket. So I stuck that 128GB card in, the Retroid and formatted it. This freed up a smaller, 32GB card.

I also have a 64GB that is basically not being used in my PiGrrl Project I decided to recover back for use. The project was fun, but the Retroid does the same thing 1000x better. So now it’s mostly just a display piece on a shelf. Literally an overpriced paperweight. I don’t want to lose the PiGrrl configuration though, because it’s been programmed up to work with the small display and IO Control Inputs. So I imaged that card off.

In the end though, I didn’t end up needing those Micro SD cards though, I opted for an alternative option to replace the Pi, with Docker on my secondary PC. I’ve been meaning to better learn Docker, though I still find it to be a weird and obtuse bit of software. There are a handful of things I care about restoring that I used the Pi for.

  • Youtube DL – There seem to be quite a few nice Web Interfaces for this that will work much better than my old custom system.
  • WordPress Blog Archives – I have exported data files from this but I would like to have it as a WordPress Instance again
  • FreshRSS – My RSS Reader. I already miss my daily news feeds.

YoutubeDL was simple, they provided a nice basic command sequence to get things working.

The others were a bit trickier. Because the old set up died unexpectedly, The data isn’t easily exported for import, which means digging out and recovering off of the raw database files. This isn’t the first time this has happened, but its a lot bigger pain, which isn’t helped by not being entirely confident in how to manipulate Docker.

I still have not gotten the WordPress archive working actually. I was getting “Connection Reset” errors and now I am getting “Cannot establish Database connection” issues. It may be for nothing after the troubles I have had dealing with recovering FreshRSS.

I have gotten FreshRSS fixed though. Getting it running in Docker was easy peasy. Getting my data back, was… considerably less so. It’s been plaguing me now when I try to fix it for a few weeks now, but I have a solution. It’s not the BEST solution, but it’s… a solution. So, the core thing I needed were the feeds themselves. Lesson learned I suppose, but I’m going to find a way to automate a regular dump of the feeds once everything is reloaded. I don’t need or care about favorited articles or the articles contents. These were stored in a MySQL database. MySQL, specifically seems to be what was corrupted and crashed out on the old Pi/Instance because I get a failed message on boot and i can’t get it to reinstall or load anymore.

Well, more, I am pretty sure the root cause is the SD card died, but it affected the DB files.

My struggle now, is recovering data from these raw files. I’ve actually done this before on a surver crash years ago, but this round has lead to many many hurdles. One, 90% of the results looking up how to do it are littered with unhelpful replies about using a proper SQL dump instead. If I could open MySQL, I sure as hell would so that. Another issue seems to be that the SQL server running on the Pi was woefully out of date, so there have been file compatibility problems.

There is also the issue that the data may just flat out BE CORRUPTED.

So I’ve spun up and tried to manually move the data to probably a dozen instances of MySQL and MariaDB of various versions, on Pis, in Docker, on WSL, in a Linux install. Nothing, and I mean NOTHING has worked.

I did get the raw data pulled out though.

So I’ve been brute forcing a fix. Opening the .ibd file in a text editor gives a really ugly chuck of funny characters. But, strewn throughout this, is a bunch of URLs for feeds and websites and well, mostly that. i did an open “Replace” in Notepad++ that stripped out a lot of the characters. Then I opened up Pycharm, I did a find and replace with blanks on a ton of other ugly characters. Then I write up this wuick and dirty Python Script:

# Control F in Notepad++, replace, extended mode "\x00"
# Replace "   " with " "
# replace "https:" with " https:"
# rename to fresh.txt

## Debug and skip asking each time
file = "fresh.txt"
## Open and read the Log File supploed
with open(file, encoding="UTF-8") as logfile:
    log = logfile.read()

datasplit = log.split(" ")
links = []

for each in datasplit:
    if "http" in each:
        links.append(each)

with open("output.txt", mode="w", encoding="UTF-8") as writefile:
    for i in links:
        writefile.write(i+"\n")

Which splits everything up into an array, and skims through the array for anything with “http” in it, to pull out anything that is a URL. This has left me with a text file that is full of duplicates and has regular URLs next to Feed URLS, though not in EVERY case because that would be too damn easy. I could probably add a bunch of conditionals to the script to sort out anything with the word “feed” “rss”, “atom” or “xml” and get a lot of the cruft removed, but Fresh RSS does not seem to have a way to bulk import a text list, so I still get to manually cut and paste each URL in and resort everything into categories.

It’s tedious, but it’s mindless, and it will get done.

Afterwards I will need to reset up my WordPress Autoposter script for those little news digests I’ve been sharing that no one cares about.

Slight update, I added some filtering ans sorting to the code:

# Control F in Notepad++, replace, extended mode "\x00"
# Replace "   " with " "
# replace "https:" with " https:"
# rename to fresh.txt


## Debug and skip asking each time
file = "fresh.txt"
## Open and read the Log File supploed
with open(file, encoding="UTF-8") as logfile:
    log = logfile.read()

datasplit = log.split(" ")
links = []

for each in datasplit:
    if "http" in each:
        if "feed" in each or "rss" in each or "default" in each or "atom" in each or "xml" in each:
            if each not in links:
                links.append(each[each.find("http"):])

links.sort()

with open("output.txt", mode="w", encoding="UTF-8") as writefile:
    for i in links:
        writefile.write(i+"\n")

Battery Testing with Raspberry Pi

Recently I purchased a portable phone battery on clearance. I want to use it to modify my defunct Pokemon Go Gotcha band, which no longer holds a charge, to run off the battery pack. I realized that a battery pack could also be useful for powering other small electronics, such as the Raspberry Pi.

The problem is, I wanted to get an idea of how long the battery would last powering the Raspberry Pi. Figuring this out isn’t really all that hard. The tricky part is that I can’t stand over the Pi and watch it until it dies.

So instead I set up a simple cronjob task to do the job for me. I had a spare SD card, so I threw a basic fresh install of Raspbian on the card. I configured SSH and WiFi, then did a quick run of “crontab -e”, then droped the following at the end.

*/5 * * * * date >> /home/pi/date.txt

Simple.

Every 5 minutes, the Pi will now write the Date and Time to the file “date.txt”. It will do this until it can’t which would be after the battery dies, killing the Pi.

After charging the battery pack over night, I stuck the Pi on it and left it.

After checking back periodically, when I found the Pi was dead, I plugged it into a regular power source to retrieve the data. The result were both better than I had hoped, and not as great as I had hoped. The Pi started spitting out Time Stamps again after I plugged it back in, so I ended up having to skim through the file to find the time gap. I went ahead and truncated the data down to hour stamps until I came across the time jump from when the Pi had died and when I had plugged it back in.

So it turns out that the Battery pack will drive power for the Pi for around 12 hours. I also want to test this under a bit of a load and test how long it will power an Arduino writing to a remote database.

A Pile of Used Tech

I recently had an idea occur to me that I might be able to pick up used Raspberry Pis off of eBay more affordably than buying them new. I didn’t really find a ton of savings, but I did pick up an auction for a lot of various parts for fairly cheap.

I am not sure what I’m going to do with all of this, but it seemed like a deal for around $50. I was worried that it wouldn’t all be included, but it was. Not everything is what I had hoped though. Two of the three Raspberry Pi 2s seem to be dead. I’ve tried several trouble shooting methods so far. They turn on, but don’t seem to real SD cards at all.

The arduino is a genuine Arduino, which is nice, but its a fairly older model. Not a huge issue, but it is what it is. The screens were a nice bonus. I’ve been looking into getting a screen of some sort of my Pis, possibly for a RetroPi handheld build. I have not tested the larger screen yet, it seems to work off of a funky daisy chain of an extra board and some cables. I did get the smaller screen working… ish.

It’s a nice little touch screen that fits nicely on top of the Pi. I have not had a chance to properly troubleshoot it, but the touch works kind of funky on it. For one, it seems to function more like a track pad than a straight touch. Two, the mouse cursor only wants to move along a diagonal axis across the screen.

This all kind of feels like a configuration issue however, so there is some hope. Plus I am not sure I really need a touch interface for a RetroPi handheld build.

There’s some other fun stuff that I have not had a chance to mess with yet. There were a ton of ultra sonic sensors. I’m not sure what exactly these could be useful for, but I am wondering if they would be able to do 3D imaging of an object or a space.

There’s some funky board with a digital display on it that seems to be some sort of power board. I am not sure I’m going to have a use for this at all.

Lastly, there is a Raspberry Pi camera module. I have not had a chance to test it out yet, but like the screen, this was something I’ve been wanting to try out.

Saving a Slice of Raspberry Pi

Raspberry_Pi_LogoSo of all the things on the network, the DNS and DHCP server are pretty important.  Especially because my experience has been the router the ISP provided is kind of mediocre at doing the DHCP job.  So when the Raspberry Pi I’ve tasked with doing the work started flaking out I was a little worried.

After some investigation, it seems the SD card had lost a sector, or whatever the equivalent thing is to a sector on flash memory.  These things do have a bit of a limited life and this particular one was a small 4GB card that was a little older.  Four gigabytes isn’t even above the recommended size for the Pi, but I knew I wasn’t planning on putting a lot on it.  Problems arose when it started dying though.  There actually isn’t much using the PiHole, some of the things my wife does with coupons and such I suspect might get blocked and I don’t want the headache of dealing with trying to manually whitelist things and complaints that this and that isn’t working.  The DHCP is another issue.  When the DHCP server disappears, it seems anything using it simply can’t connect to the internet.  I have a lot of static assignments but there’s quite a few assigned by the server.

I may have to look into setting up a secondary backup server, maybe on one of the CHIPs even.

I started out simply re-imaged Raspbian onto a fresh card and setting things up again following the tutorial I had posted.  Everything went pretty smoothly except that I was now out my old config file and would have to go through the hassle of recreating all of the changes I’d made adding and removing devices.

So I went with a new strategy, why not just clone the old card to a new one.  Unfortunately, Win32 Disk Imager, the recommended tool for imaging these SD cards, couldn’t read the whole card, it got hung up and failed partway in.  So I turned to my laptop and Linux hoping for something more robust, which I found in ‘dd’.  It probably stands for “Disk Duplicator”, but don’t quote me on that.

I started running dd to make an image of the old card to put onto a new card, unfortunately, it also failed, like Win32DiskImager.

Back to Square One…

After some searching, I found this post suggesting something called ddrescue.  Most of it wasn’t anything I needed aside from the following.

sudo apt-get install gddrescue
sudo ddrescue -v /dev/sdb /dev/sdc

A few things of note.  My laptop, like many, has a build in SD slot, I added a second using my USB card reader.  I opened the disk manager to get the path to each card, /dev/sdb and/dev/sdc.

This command also hit an error at the same point as both Win32DiskImager and the “dd” command, but it pushed on through, recovering everything else.  At this point it’s a matter of hoping that one bad spot wasn’t int he middle of an important file.

NextThing CHIP and Raspberry Pi Zero

Raspberry Pi Zero and Next Thing's CHIPI’ve already briefly discussed my NextThing CHIP computers before. A few weeks ago I was purchasing a Raspberry Pi 3 and found that the store had Pi Zeros in stock as well, so I tossed one on because, hey, $5, and they are apparently hard to find.

These two little devices are pretty similar, but at the same time so different.

Both are small computers, not just microcontrollers like an Arduino, but little Linux machines. Both have GPIO pins, both can be used as a regular “desktop” if you wanted, both cost less than $10… Mostly…

Raspberry Pi ZeroThe cost is where things get tricky, especially for the Raspberry Pi Zero. Upfront, the Pi Zero is only $5, which looks great on paper, and it’s still pretty cheap. That five dollars gets you a small flat circuit board with a few low profile ports on it. Chances are you’ll need a power supply, if you want to be able to configure it at all you’ll also need a way to plug a standard USB hub for keyboard and mouse into a micro USB port. The display out is a mini HDMI port, which you’ll likely also need an adaptor for. The Pi Zero also doesn’t have any sort of case on it. All of this is stuff that you may or may not need, you might have on hand, or, if you’re programming many Zeros, you can reuse the adapters and parts on all of the Pi Zeros you want, once configured.

The point is, after picking up even some cheap accessories, you’re probably looking at closer to $15 for a Pi Zero. Still cheap, just not quite AS cheap.

Next Thing CHIPThe CHIP on the other hand, is $9. It has built in WiFi and Bluetooth, it comes with a little plastic case for the underside of the board, and it comes with an RCA adapter. The RCA isn’t ideal and an HDMI or VGA attachment are both coming (for $10-$15) but the RCA is usable for setting up WiFi easily for SSH or VNC access. It even has built in storage. For $9, you’re pretty much up and running right out of the box. The Pi Zero is actually kind of useless all on it’s lonesome.

That doesn’t necessarily mean the CHIP is better.

Both The Raspberry Pi Zero and Next Thing’s CHIP run an arm based version of Debian (by default, but this could be changed), thus both have a lot of common in terms of what sorts of software can be used. The CHIP however seriously lacks the Raspberry Pi’s strong community. I’ve not been able to find any decent resources for how to even use the GPIO pins, for example. I had hoped to use one of my CHIPs for a project in the garage but I’ve hit some roadblocks.

chipdesktopI’ve actually had lots of roadblocks with the CHIP. Initially they were unstable, requiring a firmware update. I’ve had many strange problems getting software packages to work properly, even when following tutorials put out by others. The closest I’ve come to finishing a project is making a homemade Alexa clone. Even after buying identical parts (Mic and Button really), after trying many fixes and even wiping the thing out and starting from scratch, I still get errors with the Audio about “Period too large”.

Everything being built in is really convenient though. I honestly haven’t come up with any ideas yet for what to do with my Pi Zero. It ends up with some goofy dongles all over it to make it functional and I have 4 full sized Raspberry Pis at my disposal for other projects. I’ve seen an interesting Game Boy emulator build using the Zero I will probably try but so far, it’s pretty much just been running quietly on my workspace counter.

Personally, I think that ultimately it comes down to there is a point where these things become too cheap or small to be useful, at least for my needs. The sorts of projects a smaller board like this may be useful for feel like they would be better suited for an Arduino. Most of the projects I use my Pis for amount to “Hardware VMs”. Single task servers that can easily be repurposed or swapped out (using SD cards).