Reddit

Code Project: Automated List From Reddit Comments

Python Logo

This is one of those quick and kind of dirty projects I’ve been meaning to do for a while. Basically, I wanted a script that would scrape all of the top level comments from a Reddit post and push them out to a list. Most commonly, to use on /r/AskReddit style threads like, well, for this example, “What is a song from the 90s that young people should listen to.”

Basically, threads that ask for useful opinions on list. Sometimes it’s lists of websites or something. Often it’s music. The script here is made for music but could be adjusted for any thread. Here is the script, I’ll touch on it a bit in more detail after.

## Create an APP for Secrets here:
## https://www.reddit.com/prefs/apps

import praw

## Thread to scrape goes here, replace the one below
url = "https://www.reddit.com/r/Music/comments/10c4ki0/name_one_90s_song_kids_born_after_2000_should_add/"

## Fill in API Information here
reddit = praw.Reddit(
    client_id="",
    client_secret= "",
    user_agent= "script by u/", # Your Username, not really required though
    redirect_uri= "http://localhost:8080",
)


submission = reddit.submission(url=url)
submission.comments.replace_more(limit=0)
submission.comment_limit = 1

for x in submission.comments:
    with open("output.txt", mode="a", encoding="UTF-8") as file:
        if "-" in x.body:
            file.write(str(x.body)+"\n")
            # print(x.body)

The script uses praw, Python Reddit API Wrapper. A Library made for use in Python and the Reddit API. It requires free keys which can be gotten here: https://www.reddit.com/prefs/apps. Just create an app, the Client ID is a jumble of letters under the name, the secret is labeled. User Agent can be whatever really, but it’s meant to be informative.

The thread URL also needs filled in.

The script then pulls the thread data and pulls the top level comments.

I’m interested in text file lists mostly, though for the sake of music based lists, if I used Spotify, I might combine it with the Spotify Playlist maker from my 100 Days of Python course. Like I said before though, this script is made for pulling music suggestions, with this but of code:

        if "-" in x.body:
            file.write(str(x.body)+"\n")
            # print(x.body)

It’s simple, but if the comment contains a dash, as in “Taylor Swift – Shake it Off” or “ACDC – Back in Black”, it writes it to the file. Otherwise it discards it. There is a chance it means discarding some submissions, but this isn’t precision work so I’m OK with that to filter out the chaff. If I were looking for URLs or something, I might look for “http” in the comment. I could also eliminate the “if” statement and just have it write all the comments to a file.

Reddit Rate-Limiting VPN Users

Yesterday I started getting these annoying pop ups on Reddit.  “You’re doing that too often, please wait X minutes”.  This most often happens with new accounts that are low in Karma.  It’s a sort of trust trap that prevents spam.  My account is almost 10 years old.  I have almost 500 thousand Karma.  I don’t meet any of the new account nonsense.  This block, is very suspicious.

I’ve seen it before though.  It seems individual subs can enable something like this for accounts that post infrequently to that particular sub.  But this block seemed to span across many subs, so it was clearly site wide.   I assumed at first Reddit was blocking 3rd party apps in an effort to push people to use the shitty official app.  So I installed the official app.  And good fucking God does it suck.   The problem persisted though, so I uninstalled the app.  I also tried just using the web interface in Firefox mobile.  No go.

So I decided to try on another account.  Not only did this still happen on the other account, but the block from the first account, applied to other accounts.  So clearly the block is IP based.

Now, on mobile, when not on my home network, I use a VPN basically full time.  Why would fill a blog post on it’s own but all we need for this is to know that I use it all the time, I’ve done this for something like 5 years.  It’s not a cheap freebie VPN, I use a paid service that’s trustworthy (I am the customer, not some other advertisers).  I also use Reddit basically daily, a lot.

I disabled the VPN for a bit, the problem went away.

I re-enabled the VPN, the problem came back.

I did some searching on Reddit and Twitter and found a lot of other people having similar issues.  Reddit is rate-limiting VPN users.  It’s super annoying and super bull shit.  There are plenty of valid reasons to use a VPN, like I said, that’s a whole different blog post.  Blanked throttling VPN users is not a good solution.  I would dump using Reddit before dumping using my VPN.

Hopefully it’s just a temporary glitch that gets fixed, but for now it’s incredibly annoying.

Identity, Privacy, Anonymity

identity I had a short conversation recently over on Reddit that got me thinking a bit about the idea of online identity and, by extension, the ideas behind privacy, and anonymity online.  Privacy is a hot button issue in general lately and there has also been a lot of people causing some fuss over the idea of anonymity online. 

It’s ridiculously easy to be anonymous online.  Ok, let’s rephrase that, it’s ridiculously easy to be mostly anonymous online.  You want to be mostly anonymous, it’s trivial to make "fake" email accounts or identities.  You can even be pseudo anonymous by using a pseudonym.  If you were doing something malicious, it wouldn’t be hard to track you down from a simple pseudonym, especially if some large corporation or government wanted to track you.  Chances are you’re pulling cookies around in your browsing, and you’re connection will have a unique, logged IP address complete with time stamps etc. 

Being actually anonymous is trickier but still pretty trivial, spoofed IPs, TOR browsing, using open WiFi access points, especially public ones, in areas where there are no cameras, etc.  I’m not really here to discuss true anonymity online though, more the idea of pseudo anonymity.  This is the sort of anonymity that many more casual users of the internet greatly dislike.  It certainly has it’s good sides and it’s bad sides.

The complaints often come because of "Trolls" who use the anonymity granted by the internet as a means to be rude or mean.  The problem is that the term troll is often greatly misused or misappropriated.  I once wrote a pretty long essay back on usenet about what a troll is but the short version comes down to a few things.  Trolls and straight bullies are not the same.  Trolls and straight assholes are not the same thing.  Being an actual troll does require some effort, just going and telling someone they are "a stupid fag" on an anonymous board doesn’t make you a troll, it mostly just makes you an idiot.  The real point of trolling someone is to speak contradictory to what is being presented, not necessarily to prove an alternate point of view but to disprove or discredit the original view being presented.  There is a point when trolling turns into idiocy and harassment.

It’s simple, people don’t like being disagreed with or having their viewpoint challenged.  If that person can’t actually defend their viewpoint, they may get called out on it, and they call the person calling them out a troll, a "coward" hiding behind anonymity. 

"You wouldn’t say that to my face in person, why do you do it online?"

This is a tricky question on many levels and isn’t really an exact parallel.  If you put masks on everyone involved to make them "faceless", put them in a room, and had the originator read their originating comment out loud, would people still say "mean things"?  What if just the trolls had masks?  Also, a lot of people do say dumb asshole comments in face to face situations.

However, yes, there is something freeing and liberating about anonymity or even pseudo anonymity.  I’m sure there is some actual psychology behind this concept.  It’s basically the same concept of "dancing naked when home alone".  People act differently when they think no one is watching.  It’s human nature.  We pick our noses, we scratch out but cracks, we dance naked, we make rude comments online.  The main difference is that picking your nose doesn’t really hurt anyone else.  Does a rude comment really hurt anyone when everyone is anonymous and everyone has the option to make rude comments? 

"You wouldn’t say that to my face in person?"  Maybe they would.  But what if you could punch back.

So let’s take a site like 4chan (the website not the "1337 hax0r duud"), where everyone is anonymous.  Yeah, it can fall into a cesspool of filth but it also can lead to a lot of good discussion.  Much of the worst is confined to /b/, and 4chan is much more than just /b/.  Even discussion of places like /v/ (Video Games) and /toy/ (Toys) can be more interesting when people feel more free to dislike what they dislike and speak their minds.  You also don’t have to deal with people trying to be some sort of crazy internet celebrity in their area of interest.  Identity is frowned upon in general, so you just get pure discussion.  No one trying to be pretentious about who they are and what they want people to think of them as, just pure discussion.

Does it lead to arguments and shit slinging?  Of course it does.  Does it lead to idiotic arguments that make no sense?  Yep.  It also leads to acceptance.  Acceptance of ideas, because maybe you actually lose an argument, but because you aren’t saddled with the pride of your identity, you are free to accept defeat, even if it just means quitting the argument in disgust.

Then there is the idea of Psudo anonymity.  Your online handle or username if you will.  You still end up with some level of reputation but it’s one step removed from your private life and personal identity.  Its also pretty easy to manage multiple online online "personas".  Chances are if you are managing multiple identities on any one website that website could easily cross connect them to each other but for basic outward facing use, it can serve a purpose.

Take Reddit.  While similar in nature to 4chan, since it’s full of user generated content that lives and dies by how much support it gets, Reddit has an identity system.  Reddit also has an archive, everything on 4chan drops off eventually sometimes in minutes, sometimes in days, but it eventually dies.  Reddit has an archive, and an identity and everything you post is easily attributed to you.  More importantly, Reddit has the "Karma system" where users can up and downvote good and bad posts.  Granted that a lot of people know that Karma is "useless fake internet points" and there are even people who try to get negative karma instead of positive karma, but it does help by giving a tangible indicator to "how good" a person is.  It’s not perfect of course, some people may be good at things that aren’t relevant.  Someone who posts to Gonewild and has 4000 karma as a result isn’t necessarily going to mean anything when it comes to political discussion. 

Hence, "meaningless fake internet points."

Then there is Facebook.  Facebook is where you connect to friends and family.  People you will know for long periods of time, possibly your entire life.  Many of these people will know you better than you know yourself.  They will know when you’re being fake and call you out when you’re being an idiot.  These are most likely the people you want to know and want to be judged by.  There is also a lot of push for having a "real identity" on Facebook.

Facebook is a place where people go to "be real." 

All of these places have elements of each other, and it’s a very tiny sampling of the endless array of websites on the internet.  Each exemplifies a major component of online identity.  A board like  4chan is all about being anonymous, but you can choose to fill in that name field, and there is even a system in place to keep your identity verified.  Reddit gives you a name and points, encouraging you to behave for the most part, but it’s not required and can one can easily start over if there is a major screw up.  Sites like Facebook, want you to be "the real you" but really, nothing is stopping anyone from making multiple Facebook profiles and fake identities.  Hell I have a Facebook Page for my cat that I rarely post to.

The point is, the closer you get to your true identity, the less publicly open you tend to be, at least about your real feelings.  There is a fear of being judged or shamed.  A fear of upsetting the lifetime friends and family we have.  Anonymity still has a place though.  It’s like a confessional, or the comment drop box.  It’s a way to voice opinion without feat of retaliation.  There are often many reasons to fear retaliation.  Assholishness and trolling aside, opinions are often formed that are negative towards people with power.  People with power often have very strong methods of retaliation at their disposal.  There needs to be a means for people to speak out against real injustices.  The side effect is that sometimes you end up with "trolls" and assholes.

Some people just need to accept that sometimes a difference of opinion is a good thing.