QBNers for QBN
- Started
- Last post
- 11 Responses
- palimpsest
Word is bond.
Panopticon imgur search: https://bit.ly/3VNoKgq
This tool retrieves the user's imgur posts to compliment skinny_puppy's awesome haasie tool.
https://www.qbn.com/reply/409086…
https://haasie.com
Search for a user, select a year and get the post. I don't know what the entry limit is for haasie.com so go easy.I created a new thread in the hope that other users will post projects that compliment the QBNverse.
For the people.
- sted0
TypeError: dataset is undefined
RuntimeError: dataset is undefined- You have to define them. Choose a dataset and type a username.palimpsest
- https://i.imgur.com/…palimpsest
- I did, and it wasn't working... but it does now. thankssted
- wow i had no idea that observablehq shows the code while you're writing it :Dsted
- Observablehq is a gift from god.palimpsest
- the God that also gave us d3js:
Mike Bostock
also showed the world what data visualisation can be working for the nyt.uan - Amen.palimpsest
- Definitely going to play with observablehq now too.skinny_puppy
- uan8
thanks palimpsest! Nice Work!
- grafician-4
Nice work!
But, how do we switch the links in context in all QBN posts?
Also, an alternative to Imgur exists?
- skinny_puppy1
This so is awesome! Thank you!
The biggest upload I've seen to haasie,com was around 150,000 links. Or they just uploaded repeatedly very quickly.
But I am reasonably confident that around 2000 links a pop should be fine. Unfortunately I have not had a large enough dataset to test with.
Planning some improvements in that domain soon which will allow for more.
- 2000, impressive!
Don't search for PhanLo, it'll break your computer. I need to add pagination for the preview.palimpsest - Cool, if it doesn't break my computer might be a good test dataset.skinny_puppy
- 2000, impressive!
- skinny_puppy2
@palimpsest two changes if you can:
1. Remove the href=" prefix.
2. Don't trim .jpeg to .jpeI can add support for both of those (or people can just fix their own exports), but I'll only get to it tomorrow night.
- Yes, sir!
I'll take care of it today.palimpsest - Anyone know why imgur made the change from .jpg to jpeg extensions a while back? They still all resolve, but still...Nairn
- I only know, it 'ignores' the extension. you can change url to .png .gif .jpg .jpeg and it will load fine. only on download you get the real format.uan
- Thanks!skinny_puppy
- Removed the href and fixed the jpeg issue. At first I only had jpg on my regex then thought I should add png and gif so changed it to w{3}palimpsest
- not counting on the pesky .jpeg
I also did a quick fix to only preview the last 100 posts (or the PhanLo safeguard).palimpsest - PhanLo 2022:
2429 posts with imgur images and a total of
3271 imgur images.palimpsest - YAASSSS.PhanLo
- Hahaha fucking hell, Phan!Continuity
- Awesome! This is how we stress test Haasie.skinny_puppy
- Thanks @palimpsest much appreciatedskinny_puppy
- 2020 has been scraped and updated.
https://i.haasie.com…palimpsest
- Yes, sir!
- Continuity2
2021 was light year for me. On 19 images sent to imgur!
- only*Continuity
- That is light.
Heads up: 2020 only has data starting from 2020-11-25.
I'll scrape the whole year today or tomorrow.palimpsest
- skinny_puppy13
Stress tested with PhanLo 2022. Uploaded 3182 links in one go. They were processed in about 5 min. This was the last image of 2022 (on the list):
- Passed the PhanLo test.
Yeah!palimpsest - Damn, noice!Continuity
- Also just did PhanLo 2021, 3271 images!skinny_puppy
- Thumbs up! that image was used on a tshirt, finally saw some of them at the weekend.PhanLo
- Very cool.skinny_puppy
- hell yeah!YakuZoku
- Passed the PhanLo test.
- skinny_puppy3
Quick update. There are 2 parts to Haasie at the moment: 1. Ingesting links. 2. Downloading the files for the links from Imgur and storing them.
Good news: I've improved the link ingestion. Haasie breezed through 58K links in a test.
Bad news: Haasie has been blocked from Imgur from downloading more files. So all the files downloaded recently have been 0 bytes.
I've taken Haasie down for the moment, to give Imgur a breather. Hopefully in an hour or two I can start it up again.
The files that were uploaded at 0 bytes are easy to identify and I can reprocess them and download them properly when Imgur lets me again.
Will aim to build a more elegant solution for throttling soon.
- I blame PhanLo.
<3palimpsest - Haha!skinny_puppy
- sorry :-(PhanLo
- Are you storing all these images on your servers?drgs
- @drgs to get technical they are eventually stored on an S3 equivalent called Spaces (from DigitalOcean). And then cached by a CDN called Fastly.skinny_puppy
- To be less technical, effectively stored on servers I am running. Setup specifically for this purpose.skinny_puppy
- Couldn't you upload them back to imgur, but this time under your account, and your server only translates old codes to the new onesdrgs
- Imgur pics launderingdrgs
- Interesting idea, that could definitely be done as a last solution. sted suggested something similar. The first challenge remains to backup the files.skinny_puppy
- TBH though, I am having a lot of fun atm. I guess if stops being fun, I can look for a longer term archive solution - I like your suggestion for that.skinny_puppy
- lol @ launderingskinny_puppy
- I blame PhanLo.
- skinny_puppy0
Wow. I have tried creating a number of new servers. Thought I could proxy through them to continue the downloads. But they are all blocked. Both on DigitalOcean and Linode.
Even my trusty old server with a few blogs on it that has never even seen an Imgur link is blocked. It's like all these IP addresses have just been blanket blocked.
- Interestingly, ever since I plugged in my images links into Haasie on Thursday (or Friday), not a single imgur image shows up for me when I'm on my home WiFi.Continuity
- It's all broken.Continuity
- I've seen similar comments online. I guess Imgur is just blocking swathes of IP addresses.skinny_puppy
- Interestingly, everything's working again since the last couple of hours. I don't know WTF imgur are doing.Continuity
- My servers are not that blessed. But my local is working. I'll download what I can here and see how to get it into Haasie afterwards.skinny_puppy
- skinny_puppy1
@palimpsest could you send me a dump of all the links? I've been downloading them 1 user at a time but it is slow going. That way I can just package them all into 1 upload and let it chug away in the background on my local.
- Yes, I'll take care of that.palimpsest
- Done:
https://observablehq…palimpsest - Magic thank you!skinny_puppy
- skinny_puppy8
Been catching up on other work, but here is a quick update.
Haasie is now closed for uploading Imgur links, but you can upload images and videos directly. Nothing bigger than 200MB atm. Copy n Paste works too:
I'm still devising a plan to move all the images I managed to download from palimpsest's links over to the live site. Download took a few days due to loadshedding.
- you da man!YakuZoku
- Thanks @YakuZoku!skinny_puppy
- i can scrape all imgur links on the site and provide referring urls if that helps? might take a long time tho as the site is biiigtrooperbill
- ^ That's what palimpsest has been doing.Continuity