QBNers for QBN

Out of context: Reply #8

  • Started
  • Last post
  • 14 Responses
  • skinny_puppy3

    Quick update. There are 2 parts to Haasie at the moment: 1. Ingesting links. 2. Downloading the files for the links from Imgur and storing them.

    Good news: I've improved the link ingestion. Haasie breezed through 58K links in a test.

    Bad news: Haasie has been blocked from Imgur from downloading more files. So all the files downloaded recently have been 0 bytes.

    I've taken Haasie down for the moment, to give Imgur a breather. Hopefully in an hour or two I can start it up again.

    The files that were uploaded at 0 bytes are easy to identify and I can reprocess them and download them properly when Imgur lets me again.

    Will aim to build a more elegant solution for throttling soon.

    skinny_puppy 3Permalink
    UpvoteDownvote
    Flag
    • I blame PhanLo.
      <3      palimpsest
    • Haha!skinny_puppy
    • sorry :-(PhanLo
    • Are you storing all these images on your servers?drgs
    • @drgs to get technical they are eventually stored on an S3 equivalent called Spaces (from DigitalOcean). And then cached by a CDN called Fastly.skinny_puppy
    • To be less technical, effectively stored on servers I am running. Setup specifically for this purpose.skinny_puppy
    • Couldn't you upload them back to imgur, but this time under your account, and your server only translates old codes to the new onesdrgs
    • Imgur pics launderingdrgs
    • Interesting idea, that could definitely be done as a last solution. sted suggested something similar. The first challenge remains to backup the files.skinny_puppy
    • TBH though, I am having a lot of fun atm. I guess if stops being fun, I can look for a longer term archive solution - I like your suggestion for that.skinny_puppy
    • lol @ launderingskinny_puppy
    Show [[ numHiddenNotes ]] more notesAdd Note
    SaveCancel

View thread