100DaysOfCode

100 Days of Python, Projects 66-70 #100DaysofCode

Ramen Junkie 0 Comments

Whew, I didn’t really think I’d get to 9 parts in this series, and I am only around 2/3rds of the way through even.  I actually may change up the format later with the last 20 projects that are listed as “Professional”.  Maybe one post each.

The bulk of this round is wrapping up the Flask projects and building a simple blog that runs on Python.  It’s been fun.  I’ve been a bit busier than normal slow my pace has slowed, but that’s ok too.  Day 66 in particular felt like it took longer than it really should have, given how little it felt like it was doing.

Day 66 – Build RESTful APIs

Kind of a different sort of project.  It’s building something, but not really anything, with any sort of interface.  All of the interaction is done through Postman (or the URL if one wants), and the responses are all JSON of some sort.  

We built 6 API interfaces to work with a database of Coffee shops.

  • One returned all of the shops.
  • One returned all of them in a particular city.
  • One returned a specific shop only.
  • One updated the price of coffee.
  • One added a new Coffee Shop.
  • One deleted a Coffee Shop.

Including built in error handling for if a Coffee Shop didn’t exist etc.  It all feels like it could be useful later in the Blog Project.  Not too useful on it’s own accord.

Day 67 – Blog Capstone Project Part 3

Back to the Blog Project, and after this round it’s a LOT more Blog Like, though incredibly insecure.  The Security Part looks to be the subject of the next couple of lessons though, so it’s all good.  We basically started with the last Blog Project from Day 59, and added the ability to add, edit, and delete posts.  Also with a Database back end.

One tricky moment here.  The idea was to use the same form page for New Post and Edit Post.  Originally I had nested these together, and had some if/else cases to check if it was Editing or Adding, and kind of got stuck on how to handle the date, since the date doesn’t get changed on updates.

But then I had a bit of an epiphany moment, the kind you get often with coding, where you band against a problem, convinced it’s the way to solve things, only to realize there is an obvious, easy method.

I split them up.  No more if/else, and no more worrying about what the DB is doing at the end.  Because two separate @app.route and related functions, can route to the same HTML template (the Edit Form).  The only thing needed was to pass a header variable along to change the title.  

It was a total “Duuuuuuh” moment.

Anyway, no security though, because there are no users or security keys and anyone can just come in and post anything and delete anything and it would be total chaos as an actual production Blog platform.

Day 68 – Authentication with Flask

Oops, I spoiled what this topic was going to be in the last project write up.  This whole section is working with Authentication using a Database for persistence.  It’s a simple website, a home page, a user creation page, a log in page, and a secrets page.  Users can create an account, and once registered, they can access the Secrets page and download a PDF.  

Part of this exercise is also restricting access for not logged in users and another part is proper security and handling passwords with hashing and salting.

Of all of the projects so far, I feel like this may be one of the most important ones, despite it’s basic simplicity.  I’ve been planning to work out the best way to share some of the projects built so far, and using Flask was a good “first step”, but being security conscious, I would rather not throw a bunch of web based Flask pages up with no restrictions on access.  Like last session, there was the Library App or the Top Ten Movies page.  Sure, I could put them up on the web-server, map some sub domain to them, but anyone can edit them, without proper, persistent security and restricted access.  

Now, I can make that happen.  Plus, with the modulator of routing and such, I can easily slip them into parts of the overall Blog Project, in time.

Day 69 – Blog Capstone Project Part 4

And here it is, the culmination of the Python based Flask Blog.  It’s neat.  I like it, well, I like the basic functionality of it.  The layout is a bit plain but that’s fixed.  I doubt it ever replaces WordPress, I love WordPress, but I could definitely see uses for this finished product.  I may actually modify it to work as a sort of “Twitter Replacement” since Twitter is currently burning down.  Using what I have learned, I could easily set this up to take a post, then “Syndicate” it out via API calls to Twitter, Facebook, Mastodon, or anywhere.  While keeping my own archive of posts.

My current, next up To Do Items:

  • Create RSS Feed Page  
  • Create Admin Page  
  • Enable/Disable Comments for Posts  
  • Allow Admin to delete Comments  
  • Allow Users to delete their own comments  
  • Add Mailer.class and add email notifications  
  • Add Pagination to Home Page  
  • Add Tags and Category Options to Posts

Also, on a side note, this actually isn’t the first time I’ve built a “Blog Platform”. I built a basic one a few years ago in HTML and PHP. Heck, the system I used int he early 2000s with SHTML Pages was sort of a “Blog Platform”.

I’d recommend, for anyone going through this course, check the notes on the Blog courses.  There are a lot of good suggestions for improvements, especially in stopping things like Java Script injection in the comments.  When you go to Angela’s (The Instructor) Blog link, it immediately throws out a JavaScript pop up that someone dropped in the comments.

Day 70 – Hosting with Heroku

So, there isn’t really a Day 70 project.  I did go through the section, and the most useful part for me was the more robust SQL solution mentioned in the last section.  The first few parts were about GIT and Github, which I am already using.  The middle part was about using Heroku, which I have heard of and used a bit before but for the long term, I don’t need to use a freemium hosting service in a jankey way.  I have a VPS, and later I will figure out how to get Apache to play nice with Flask.

For now, I’ve split the Blog Project into it’s own repository and worked up my initial planned ToDo List.  But I need to keep my focus on the course for now.

100DaysOfCode

100 Days of Python, Projects 58-65 #100DaysofCode

Ramen Junkie 0 Comments

Judging by the comments on the lessons, The lessons are getting harder, though frankly, I am finding they are a bit easier.  I feel like my experience with Web Design is a lot of this.  I also have been looking a bit into how to properly host some of these apps to share here, on my Code Projects Page.  I believe I could set them up to run on a production Flask environment, with different ports, then just map sub directories n the domain to different ports.

But I am doing my best not to get distracted.  Yes, I have been building a few smaller projects here and there lately, but I don’t want to get TOO distracted.

Day 58 – Bootstrap Overview

Not a lot of really say here, like the Web Foundational section, this was a chunk of the Web Dev course offered by the same Instructor.  I may look into picking it up if it’s on sale for cheap, since at this point I’ve done like half of it.  My next plan for a major learning push is going to be Javascript.  I really need to get familiar with Javascript for some work projects.  

The project was essentially using different Bootstrap bits to format a silly Tinder knockoff website, a site for Dogs called Tindog.  Bootstrap is definitely useful, but I generally try to avoid it because it makes things look very samey.  I suppose familiarity in design is useful at times though.

Day 59 and 59 – Blog Capstone Project Part 2

The last part of the Intermediate+ section was the bones of a Clean Blog running in Python Flask.  These two days expanded on the Blog concept a bit, and a few later lessons bring it back around again.  I am almost interested in trying to use the blog once finished, except that I already have plenty of proper outlets for Blogging.  Wordpress works fine.  

The core concept was using Bootstrap to format the blog up nicely, as well as setting it up to reuse Header and Footer files.  It’s a common method for webdesign, and it’s one I have been using since Geocities when I discovered something called SHTML which would let me write Blog Texts in HTML, then encapsulate them into a common top and bottom.

Day 61 – Flask Forms

An intro to easier to use Flask based forms and libraries.  The project itself was to construct a simple log in page, which lead to a Rick Roll when unlocked.  I could see this piece being useful later to add in on the more complete Blog project.

Which brings to a bit of an aside topic.  A lot fo these projects feel disconnected, but really, they are demonstrating a proper way of handling larger Code Projects.

Break it into parts.

We built a simple blog page that pulled posts from a CSV file.

We formatted that blog page.

We created a log in form, that could easily be slipped into a blog page.

Later lessons work with more dynamic forms and data and SQL Lite databases.

As the pieces get laid out, the end goal becomes more and more clear.  My prediction is that the later Blog Capstone project will amount to, “Take everything from the last ten weapons and mash it into a functioning Blog app.

Day 62 – Coffee Shop Wifi Tracker

Like I said, building on the last.  For this lesson, we built a little table that would list Coffee Shops in  pretty Bootstrap table with ratings for Coffee Quality, Wifi Quality, and Power Outlet availability.  Instead of just a form that takes an input and verifies it’s good, it actually inserts new data into the CSV that holds all the coffee shops.

Day 63 – Book Collection App

I may revisit this one later to build an app for my other collections.  And to make it more robust.  The core though is once again, building on the Coffee App, essentially.  Instead of just a tracked list with an Add Form, we added persistence by learning about SQL Lite databases.  So the data persists even if the server is shut down.  

I took a few extra non required steps on this app and reused the Coffee Shop code to make this app look much nicer.  The core app and instructions just had a very basic page with an unordered list.  I turned it into a table with a bit more formatting and color.

Day 64 – Top Ten Movies App

This whole project was essentially the same as the Book Collection App, except it was intnded from the start to look prettier.  The main difference is that the add process had a few extra steps to pull from the API of The Movie DataBase website to get data about each film, and have the user select from a list of results.  In the instructions for the class, the idea was to make multiple API calls for the list, then specific movie data, but I had already worked ahead on the project and instead it pulls the data, the user selects the movie, then it’s just done.  It was a bit complicated to get the data to pass around between pages since it was a dictionary and not just a single variable, but I got it working using Global variables.

I mostly try to avoid Global variables, but it felt like this was a good use case and made sense to use given the way Flask works.

I am thinking I may integrate the log in app created previously with this website and use it to test out mapping some of these projects to a domain to make it a for real, live website. Also, you can’t tell it in the snapshot but the posters do this cool flip over effect when you mouse over them to show more information. Not really a Python thing, but it’s still a neat effect.

Day 65 – Web Design Principles

No project for this day, it’s another slice of the Web Developer course from the same instructor.  I’m definitely going to keep an eye out for a deep discount, especially with Black Friday coming, since I’ve basically completed half of that course now.

I was hoping to fit all of this more advanced Flask based projects into one page but it feels like this post is getting pretty long so I’m going to go ahead and break it up here.

Programming Projects

FreshRSS and RSS Feed Posts

Ramen Junkie 0 Comments

Keen observers (ha ha ha no one reads this), might have noticed that a few posts of links showed up in the feed.  These are basically, stories I read in my RSS reader that I found interesting, and wanted to share, or at least, keep track of.  The posts as of now are a little ugly, and I’ll probably clean up the formatting over time, but I wanted to go ahead and write a bit about the process.  I’ll have the Code on Github at some point.

As for the factors, firstly, this is something I’ve wanted to have on my blog for a while.  Like a long while.  I might even try to see if there are ways to better slit up the links by topic later.  A fair number of blogs I subscribe to have these sort of link digest posts, and I’ve always just liked the idea.  It’s also good for personal reference to when I may have read something.  It is limited as it only comes from y RSS Reader.

Speaking of my RSS Reader.  I’ve moved on from TinyTinyRSS, for a few reasons.  One, the interface is a little meh, honestly.  Maybe the newer version is better but it’s only available in Docker, and Docker is such a PItA to use.  Also, while looking for alternatives, it sounds like the folks who make TTRSS are kind of a bunch of gatekeeping jerk types, and I’d rather not support that.  I also find the need to keep the update daemon running with Screen to be a pain.  So I’ve moved over to FreshRSS, which I just run locally on a Raspberry Pi.  I may move it to a publicly accessibly machine at some point, but I am not entirely convinced that TT-RSS wasn’t the entry point for my previous server malware woes.

So, like TT-RSS, Fresh RSS has a way to get an RSS feed out of your Favorited posts.  In the past I’ve used tools like IFTTT to automate posting these links around, but I don’t use IFTTT anymore for reasons I’m not going into.  Fortunately, I’ve been working to become a pretty good Python coder for the last month or so.  So instead I wrote a script.  

It’s not even a particularly complicated script.  There are only two things it really needs to do, get new articles, and then post them to WordPress. Since the script runs locally, on the same Raspberry Pi even, it easily can reach and pull the RSS feed.  One nice thing I noticed with Fresh RSS, the feed included a time interval, so just getting new posts was super simple, because the interval is just “24” for “24 hours”.  The script eventually will run on a cronjob at the exact same time daily.  Anyway, after pulling the RSS, the entries are already in an easily usable Dictionary.  which gets fed into the construction of the WordPress Post.

def get_feed(feed_url):
    NewsFeed = feedparser.parse(feed_url)
    return NewsFeed

The posting part was pretty easy as well, WordPress has an API, and Python also has a library that can use that API.  It just needs some log in information and a post payload to send.  

def make_post(NewsFeed):
    wp = Client(f'https://{wp_url}/xmlrpc.php', wp_user, wp_pass)
    post = WordPressPost()
    post.title = f"{cur_date} - Link List"
    post.terms_names = {'category': ['Link List'], 'post_tag': ['links', 'FreshRSS']}
    post.content = f"<p>Blogging Intensifies Link List for {cur_date}</p>"
    for each in NewsFeed.entries:
        post.content += f'{each.published[5:-15].replace(" ", "-")} - <a href="{each.links[0].href}">{each.title}</a></p>'

The trickiest part was formatting the date a bit prettier.  I mentioned cleaning up the formatting a bit, I’m thinking maybe a simple invisible table, so the date and the links don’t wrap oddly like they do now.   i also added a check that if there are no new favorited posts, it will skip making a post.  Otherwise I’ll end up with empty posts on days I forget to check my feed reader

While writing the script, at first I was just outputting a text copy of the post to the console until satisfied.  Eventually, I pushed out a real post, then verified that things worked.  The next day, was just a straight test by opening the project, then running it again.  The third day, I copied the files and installed the lobraries needed, then posted from the Pi.  Phase 4 of this will be to set up Cron to run it automatically.  If that works then it will certainly, “just run” for the foreseeable future.

100DaysOfCode

100 Days of Python, Projects 54-57 #100DaysofCode

Ramen Junkie 0 Comments

Back to web development again, but with a different twist this time.  Instead of scraping things, we’re learning Flask, to produce little Python based Websites.  In doing these exercises, I find I am kind of wondering why one would use Python over say, Apache, or NGINX or even IIS.  I can sort of see where it’s useful, and maybe later we will get to more of it’s usefulness.  My primary issue is that the HTML code part of it ends up being VERY specifically Flask based.  Like flask looks for images and CSS in specific folders.  Plus if you use any sort of variables, they all get passed to the HTML in a very particular way.

I had considered that it might be useful for sharing some of the code I have written through my web server, but in my research, things like Tkinter and Turtle don’t work at all through Flask.  I was kind of hoping it was smart enough to produce little Browser pop ups or something to render the graphics out.

This section isn’t super complex so far, but it wraps up the Intermediate+ section with a little interlude for Bootstrap in between, so I figure it’s a good little chunk to keep in it’s own write up.

As usual, the code is all on Github.

Day 54 – Intro to Flask

There was literally no project today really.  We created a basic “Hello World” Flask server, then created some Decorator Functions.  It was interesting, but not really that exciting to write up.  I do somewhat question the usefulness of a Decorator a bit, versus just having a function that takes an input and modifies it directly.

## Day 55 – Higher Lower Game Returns

The Day 55 Lessons were a bit better.  We covered Decorators a bit more and how to handle URLs in Flask, which brings me back to the “Is this better” I mentioned in the opening, since once again, the code will get weird to use outside of Flask.

I had a lot of fun with the project though.  It’s a web version of the “Higher-Lower” Game from way back on Day 14.  You Pick a number, it tells you if it’s higher or lower, only with web pages.  It was essentially a way to learn about using Dynamic URLs in Flask, but spiced up for fun.  I added a nav bar to mine so the user didn’t have to type a URL and could just click the next number to guess.  I also used a bunch of silly GIFs from my favorite musicians instead of Cat GIFs on each page.  

It’s kind of useless, but it was fun to build.

Day 56 – Personal Website

This day was mostly about how to quickly import existing code to Flask.  It involved a couple of practice projects and a “real” project.  The first Practice was taking the Lesson 41-44 website and importing it to Flask.

The second practice was to use someone else’s template and import it to Flask, as well as modifying and simplifying that code.

The final project was to build a simple “Name Card” website with some social links.  Essentially, it was a repeat of the second practice, but actually replacing images and information.  I kind of prefer the previously made CV website and it’s easier to hose on the web so I’m going to stick with that for now.

Day 57 – Blog Capstone Project Part 1

This project picks up in Day 59 with the start of the Advanced Section of the course.  The basic idea here was to build a simple blog interface that would read some generic JSON Posts and display them, and then let users click into each blog post to read more.

I’m particularly proud of my result, which only uses one HTML file, that varies if the user clicked on a blog post or not.  I feel like it was a pretty slick solution.  The starter files also included a file to make a “Post class”.  Using this class was not part of the assignment, but I suspect it will come up later, so I went ahead and built it, though I didn’t use it to read the blog posts.

If this comes out alright, I may actually use it somewhere, I’ve been looking for something to put on Joshmiller.net.  Though I also don’t really NEED another Blog outlet.  I barely maintain the one I regularly use now.

100DaysOfCode

100 Days of Python, Projects 51-53 #100DaysofCode

Ramen Junkie 0 Comments

Here we are now with a few more automated bot tasks.  It’s been a fun series of lessons, though I enjoyed using Beautiful Soup more then Selenium.  Selenium runs into too many anti-bot measures on the web to be truly effective.  I mean, it’s definitely a useful too, but in my experience, it’s not reliable enough.  BS seems to be much more effective, though it can’t really interact with pages.

In the long run, I think I am more just irritated by “clever bull shit” on web pages that makes both pieces of software a pain to work with. Take Instagram, none of the classes or ids are anything but jumbled characters.  The code feels like it was written by a machine, and it probably was.

Also, this round is a bit shorter than before because the course is veering off into a new direction with Flask Apps, so it seemed appropriate to wrap things up on the Automation Section of the Projects.

Day 51 – Twitter Speed Complainer

This project is great, because this is something I have tried to run from other people’s code but it never seems to actually work.  Now, I just have my own code to run.

EZ Mode.

It will need something with a desktop to run it on, but I have a while Windows PC for running random shit and a mess of Raspberry Pis.  I don’t even care about the complaining part, in fact, I would rather not, I just want to track Internet speed.  I may even change this ti push to a spread sheet or database or something later.

But for now, it Tweets.

So, the Speed test part was easy, though I used SpeedOf.me instead of SpeedTest.net, because SpeedTest.net supposedly will give dodgy numbers by partnering with ISPs and putting servers in ISP data centers.  I just prefer SpeedOf.me mostly, it’s cleaner.

The Twitter part was tricky… ish…  So, a common problem I keep running into with Selenium, is it thinks my Bot Programs, are Bots.  

I’m so offended for my Bots, accusing them of being Bots.   They run into captchas and email verifications and just flat out fail to log in or load half the time. It makes sense, captchas and email verifying exist, 100% to stop people from abusing things like, Selenium. Fortunately, Twitter Bots is one thing I do have a fair amount if experience with. I wrote one ages ago that just tweeted uptime of the server.  I wrote one script that would pull lines from a text file and tweet them out at an interval.  I have another Python based bot that tweets images.  What do these Bots do differently?  They are 100% Bots, running with the proper Twitter Bot Based AI, and labeled as such.  

So, since Selenium was being a pain to deal with using Twitter, I pulled out my Image Posting Bot code and scavenged out the pieces I needed, which was about 4 or 5 lines of code.  It uses a Python Library called Tweepy.  In order to use Tweepy, you have to use the Twitter Developer console to get API Keys, which I already had.

Day 52 – Instagram Follower

Another almost useful project. For this project, you open up Instagram and log in, then it opens an account of your choosing, and follows, anyone following that account.

Now, while I have a love/hate relationship with Instagram, I am not super interested in cluttering up my feed with thousands of accounts.  So, while I did complete the task, I set it up to ONLY follow the first 10 accounts.   I also added a check to make sure I wasn’t already following said account.

I may revisit this again later with other, more useful ways to interact with IG.  Maybe instead of following random people from another account, it auto follows back.  Or maybe it goes through “suggested” and looks for keywords in a person’s profile and follows them.

Day 53 – Zillow Data Aggregator Capstone

The final project for this section combines Selenium and Beautiful Soup to aggregate real estate listings from Zillow into a Google Spreadsheet doc.  I quite liked this one actually, it’s straight forward and relatively harmless.  I did run into an issue where it started thinking I was a Bot, but by that point, I knew I could successfully scrape what I needed from Zillow, so I commented out the Zillow call and replaced it with a file load using an HTML file snapshot of the Zillow page.

This was very easy to slide in as a fix because I was already pulling the source code using Selenium into a variable, then passing that variable to Beautiful Soup.  It was simply a matter of passing the file read instead.

Scraping the data itself was a bit tricky, Zillow seems to do some funny dynamic loading so my number of listings and addresses and prices didn’t always match.  To solve this, I added a line that just uses whichever value is the smallest.  They seem to capture in order, but eventually, some fell off, so if I got 8 prices and 10 addresses, I just took the first 8 of each.

Another issue I came across, the URLs for each listing, don’t always have a full URL.  Sometimes you had to add “https://www.zillow.com” to the front.  It wasn’t a hard fix,

if “zillow” not in link:

link = “https://www.zillow.com”+link

There was also an issue with the links because each link shows up twice using the scrape I was using.  A quick search gave a clever solution to remove duplicates.  It’s essentially:  list = dictionary converted to list(list converted to dictionary).  A Dictionary can’t have duplicate keys, so those get discarded converting the list to a dictionary, and then that result just gets flatted back out into a dictionary.

Lastly was the form entry itself.  The Data Entry uses a method I’ve used before for entering data to Google remotely, with Google Forms.  Essentially, Selenium fills out and submits the form over and over for each result.  I had a bit of issue here because the input boxes uses funny tags and are hard to target directly.  Then my XPATHs were not working properly.  I fixed this by adding two things, one, I had Selenium open the browser maximized, to make sure everything loaded.  Second, I added more sleep() delays here and there, to make sure things loaded all the way.  

One thing I have found working with Selenium, you can never have too many sleep()s.  The web can be a slow place.

« Previous Page — Next Page »

You Missed

Opinion/Editorial/Life

Thoughts on Twitter, Musk, and Alternatives…

Journal

Eclipse Adventure

Other Photos

Full Solar Eclipse, Anderson, IN (2024)

Opinion/Editorial/Life

On Artifact App Shutting Down