How I built my earthin24 twitter bot

At work my team have recently taken over a golang app. To learn a new language I like to build something that keeps my interest and by the nature of exploring something of interest you tend to cover way more surface area of a language this way.

Check out the code on github.

Space is awesome

If you follow me on twitter you may notice I retweet a lot of space stuff so building something space related was inevitable. Way back something came up in my twitter feed, glittering.blue, which is an awesome site that has a video of earth over 24hrs as seen from the incredible japanese himawari8 satellite. This satellite is in geostationary orbit and gives us a nice fixed view of Australia – where I’m from.

Live background image of earth

Initially I wrote a go cli tool to download the latest image of earth available from himawari8 and set it as my background, I was heavily inspired by the code from Charlie Loyd’s python script to download the images found on glittering.blue.

Why not a video every day

To extend the idea I thought why not generate a video everyday and post it to twitter, I did a search and found that a decent twitter name was still available, @earthin24. If you follow you’ll get a gorgeous video once a day.

Things I learned

My background is in front-end development so Go was new territory for me here’s some things I learned along the way while creating this bot.

goroutines and WaitGroup

Initially I had the for loop requesting each frame for the video synchronously which took quite a while as it would fetch 144 images, luckily go is really good at concurrency so I explored the sync package to help track goroutines and only exit once all images have downloaded. Using concurrency would allow me to create 144 goroutines very quickly and concurrently download all of them over a much shorter time period.

Starting a goroutine is as simple as prefixing a function with the go keyword.

for i, _ := range frames {
    go getImage(framePath(first), i)
}

However the process won’t wait for all goroutines to finish before exiting so we need to use sync.WaitGroup

func main() {
     var wg sync.WaitGroup

     for i, _ := range frames {
          wg.Add(1)
          go getImage(framePath(first), i, &wg)
     }

     wg.Wait()
}

For each goroutine we increment the WaitGroup counter using Add and at the end of the block we use wg.Wait() which blocks from exiting until the counter has reached zero.

You’ll also notice I pass a pointer reference to WaitGroup &wg to the getImage method so I can call wg.Done() outside of the current function scope. For more on pointers see the asterisk and the ampersand – a golang tale.

func getImage(url string, name int, wg *sync.WaitGroup, date time.Time) {
     defer wg.Done()
     //...
}

wg.Done() decrements the counter once it’s done. The defer keyword means to delay the function call until the end of the function. It would be the same as just calling wg.Done() at the end of the function but it is idiomatic Go to do this at the beginning using defer.

Automated hands free artisanal space tweets

So far I’ve been running this for a few months and it has posted daily without me needing to think about it. It’s pretty straight forward but there’s a few more learning’s when I set up the cronjob.

bash date

On MacOS you can type date to get the current date in your terminal, when my cronjob runs my bash script it needs to get today-1 in the following format YYYYMMDD:

date -v-1d +%Y%m%d

-v making it very easy by being able to subtract a day from the current date using -1d and then using the +%Y%m%d for the date output_format so a date could come out like so 20170819.

The only problem was when I tried to run this on my Linux based server the date command failed! Seems MacOS uses its own date version different from Linux, to subtract a day we need to use the --date="1 day ago" argument.

date --date="1 day ago" +%Y%m%d

ffmpeg

The go app downloads all the frames ready to be passed into ffmpeg to actually generate the mp4 video that’s posted to twitter & instagram. Since some of the frames are dropped due to the satellite not processing the image at the time the sequential file naming of the files can sometimes skip some numbers throwing off ffmpeg. If you try and pass a sequentially ordered file name pattern to it, ffmpeg will error out when looking for a file name that may be missing.

To get around this we use -pattern_type glob argument for ffmpeg and then just use a simple *.png wildcard.

Instead of ffmpeg -i %4d.png we need to do ffmpeg -pattern_type glob -i '*.png' to cater for the potential non-sequential frames.

Amazing

It continues to blow my mind that I can get images from a satellite and actually do this, the Japanese Space Ageny (JAXA) is absolutely incredible for releasing this to the public.

Bonus

I also extended this to instagram @earthintwentyfour so if you don’t like twitter follow me there.

🌏 12pm 25th August – 12pm 26th August 2017

A post shared by Earth (@earthintwentyfour) on