You do the best you can with what you know

This is a lightly re-edited post that I originally shared with my NYU colleagues. I’m posting here because I thought the process of making do with the technology one has and understands might resonate with other self-taught technologists. 

I thought I’d share a crazy thing that I did simply because I couldn’t figure out any other way to do it (and because I lacked the knowledge and skills to do it any other way).

I was getting ready to go to DHSI in 2013 where I was signed up for a weeklong text analysis course. I wanted to assemble a corpus to work with. So I decided to download the complete run of a journal since it was all freely available for the taking. I wrote to the editors asking if they could send me the pdf files, but they never answered. So I hacked around over the course of a week or so to concoct a semi-automated way to download the files. Here is a detailed description of the rather convoluted way that I did it. This process has many steps, requires lots of operator intervention (i.e., me), and takes lots of time. So much time that as my departure date for DHSI neared I started to panic, so I set up as many computers as I could get my hands on in an attempt to speed up the work through “parallel processing.”

Here’s a picture of the setup at my apartment just days before the start of DHSI. All of the computers are downloading issues of the journal.

Parallel processing the low-tech way

Once I got to DHSI (where there are typically a good number of highly technically proficient people — of which I am NOT) I was talking about my crazy parallel processing setup and of course someone said something like “oh, why didn’t you just use [something something tool / technique / I know not what]? That would have been so much easier.” Well, yes, it would indeed have been so much easier but I didn’t have the skills or the knowledge to do it that way. So I did the best I could with the tools I knew and I at least learned something in the process. And it worked (albeit slowly). I got the corpus.

So I guess if there’s any moral at all to this story, it’s that you do the best you can, think creatively about the tools you know, and push them as far as they can go. Sometimes it works. Sometimes it doesn’t.

(FYI – I still haven’t made any progress on that text analysis project.)