Monday, March 20, 2017

User Agent Strings : Rules of Thumb


Usage Stats via User Agent string

All incoming web traffic to a web server contains a "user agent", which is a string value that clients include with their web requests. The user agent string contains information about the device hardware, the device's OS, the browser type, and the browser's version. Thus, if you have access to a web server's traffic logs, you have a great way to gather statistics about your clients and the devices they use.

Unfortunately, user agent strings are not standardized. There is some consistency within a given device manufacturer/browser type, but otherwise sometimes one or more of these pieces of information are missing or are ambiguous.
When interpreting user-agent strings you need to do a little bit of investigating to reverse-lookup these pieces of information.

Recently I went through this exercise, and I developed a few rules of thumb.

Analysis Caveats

  • If 80% accuracy is all you're looking for, then these rules of thumb will serve you well. 
    • These rules encompass major PCs, smart phones, and tablets. These rules do not cover miscellaneous devices such as iPods, smart TVs, and smart watches.
    • Understand the business decision you're trying to make. Is 80% acceptable?
    • These rules represent the device market in 2017. New devices are coming out constantly with potentially new combinations of user agent strings. These rules may become dated in a couple years. Be prepared to adjust the rules of thumb accordingly.
    • Some of these rules are self explanatory, some are quirky.
  • If you're looking for 100% accuracy, then beware:
    • Not easy to automate. There will some online tools that attempt to do this; I found them all to be inadequate, and I kept falling back to manual analysis.
    • Don't waste your time analyzing every unique string. Set yourself a cutoff value; aim for the big fish. For example: during my research I deemed any string with less than 100 hits per week not worth my time.
    • Any client can spoof their user agent string (e.g. bots). There is a non-trivial margin of error baked into this kind of statistics gathering.
    • Duplicate strings. Manufacturers may be too lazy (or purposefully) use the same non-unique strings for multiple devices. The resolution of your results are have some coarseness baked into them. Don't try to precisely measure the imprecise.

User Agent String : Rules of Thumb

Ignore Junk

  • The front of the user-agent string is always "Mozilla". Completely ignore it. All user-agent strings from all devices and all browsers will contain "Mozilla". This is because of historical reasons.
  • The middle part of the user-agent string is always "AppleWebKit ... like Gecko". All user-agent strings from all devices and all browsers will contain it. Completely ignore it. Again, historical reasons. 


Interpret Browser
Generally, browser info is located near the end of the string.

  • Chrome
    • Chrome on android devices will contain both strings "Chrome" and "Safari". Completely ignore the "Safari" part of the string.
    • Chrome on an iOS device will contain the string "CriOS".
    • Chrome's version is directly after the "Chrome" string
  • Safari
    • Safari on an iOS device will only make reference to "Safari" (i.e. it will make no reference to "Chrome")
    • Safari's version is directly after the "Version" string
  • Firefox
    • Contains the string "Firefox"
    • Firefox's version is directly after the "Firefox" string
  • IE
    • Will contain the string "MSIE" or "Trident"


Interpreting Device OS
Generally, device info is near the front of the string
  • Android
    • Android will contain the string "Linux; Android", or sometimes just "Android"
    • Android version is directly part of the string. Easy.
  • iOS
    • iOS will contain the string "iPhone" and "iPad"
    • iOS version is directly part of the string. Easy.

  • Windows
    • A windows OS device with "Windows NT 10.0" corresponds to a Windows 10 operating system
    • A windows OS device with "Windows NT 6.2" corresponds to a Windows 8 operating system
    • A windows OS device with "Windows NT 6.1" corresponds to a Windows 7 operating system
  • OSX
    • Will contain the string "Macintosh"


Interpreting Device Hardware
Generally, device info is near the front of the string

  • Android
    • Most android devices include a unique 5-10 digit hardware string. Google the string to see which device name it corresponds to.
    • Android smart phone will include the string "mobile safari" (even if chrome was used)
    • Android tablets will include the string "safari" (i.e. the word "mobile" will be omitted).
  • iOS
    • iPads will contain the string "iPad"
    • iPhones will contain the string "iPhone"
    • iOS devices are otherwise ambiguous and do not contain specific hardware strings. Usually you can only narrow it down to a range of devices (e.g. between iPhone 5 and iPhone 7)
      • The only hints we get are from the iOS version and the 5-6 digit string for the iOS build version of iOS. Do reverse lookups on iOS device/version charts. Sometimes the build version is only used by 1 or 2 devices.
  • OSX
    • Assumed to be a laptop or desktop
  • Windows
    • Assumed to be a laptop or desktop, unless:
    • A windows phone will contain the string "windows phone"
    • A Microsoft surface tablet will contain the string "touch"

External References used during research
http://www.useragentstring.com/index.php
https://developer.chrome.com/multidevice/user-agent
https://msdn.microsoft.com/en-us/library/ms537503(v=vs.85).aspx
http://www.enterpriseios.com/wiki/Complete_List_of_iOS_User_Agent_Strings (out dated!)
https://en.wikipedia.org/wiki/IOS_version_history
https://ipsw.me/9.3.5

https://ipsw.me/10.0.2
https://ipsw.me/10.1
https://ipsw.me/10.1.1
https://ipsw.me/10.2
https://ipsw.me/10.2.1

Sunday, September 25, 2016

How to setup a Linksys WNDR3700v3 as a repeater.


Long house - Short Wifi

Living in a typical new orleans 'shotgun-layout' house, extending wifi to the back of the of the house has been an on going issue. Unfortunately the cable hook up is at the front of the house, and short of running 65' of coax, we needed something to boost wifi to the back of the house.

Our first solution involved an off the shelf wifi "extender", which we placed halfway between the front and back of the house, but our coverage remains spotty. Some of my peers recommended ditching the extended and using a router to boost wifi as a "repeater". I had an extra router layout around and, with a little bit of dark sorcery, I was able to setup my older router into a capable wifi booster. Specifically, I was able to successfully install DD-WRT firmware onto my router, and utilize its repeater capabilities.

(pic of both units)

Linksys WNDR3700v3

This article will be specific to the Linksys WNDR3700v3. I have written some steps a bit generically to potentially help those with a different routers; but unfortunately the steps involved vary so much depending on the specific router model and even down to which chip set the manufacture used. Results may vary.

What you will need:
-A Linksys WNDR3700v3 (this will be your secondary router)
-Another router (i.e. your host router) with internet.
-A laptop/desktop with an ethernet port
-An ethernet cable
-2 hours of your time
-Some patience (fail 3 times, succeed the fouth time)
-(optional) A second device with wifi capabilities. This will be handy to have a phone etc. to browse the net with for instructions while your laptop/desktop is tied up installing firmware on the router.
And lastly, at risk of sounding incompetent:
-the ability to sniff out when these instructions deviate from your situation. (In the end I needed mix-and-match 5 different walk-thoughs before I cobble together instructions for my hardware). This by no means a comprehensive guide; good luck :)

What you don't need:
-Physical access to the host router.
That's right, if all goes well, the host router will be none-the-wiser that a repeater has been setup. All setup is done with the secondary router. You could potentially setup a repeater off of your neighbor's wifi (so long as you know their wifi password) ;)

Terminology

When dealing with routers I find that terminology is the biggest barrier. I'll make a small attempt to de-obfuscate. Please read this section or you will regret it.

"Repeater" vs. "Extender"
Manufactures are inconsistent or use misleading branding. All in all the two terms are interchangable. Either way, the concept is the same. A 'secondary router' will connect to the 'host router' via wifi. Devices connecting to either the host or secondary router via wifi will have internet access.
For this article I'll be using these terms in the following manner
* Extender - an off the shelf device intended solely for boosting signal. I won't talk about extenders in this article.
* Repeater - a normal router which has been configured to boost signal.

"Repeater Mode" vs. "Repeater Bridge Mode"
The folks over at DD-WFT provide the following definitions:
* Repeater Mode - This mode will extend wifi internet access to both routers, but the networks created by each router will exist on a separate subnet. The distinction here is that devices connected to the host router cannot communicate directly to devices on the secondary router. Effectively, this only impact you if you are LAN gaming or file sharing; otherwise this mode probably suits your needs. This is mode is easier to setup, and this is the mode I opted to use.
* Repeater Bridge Mode - This mode accomplishes signal boost, and both routers will exist on the same subnet (i.e. devices connected to different routers will behave as if all devices were connected to the same router). This mode is a bit more difficult to setup, and is not strictly covered in this article.
* Access Point - This is the default configuration for routers. This mode does no do any repeating.

While Repeater Bridge Mode is an ideal solution, it requires that your host router and secondary router have matching chip sets, and that both routers have supported DD-WRT firmware. In my instance, only my secondary router had supported DD-WRT firmware, so I fell back to using Repeater Mode. At the end of the day it accomplished my primary goal (getting better wifi coverage for my roommates); no biggie.


"Client" and "Client Bridge"
Without digging too deep into these modes, they are intended for people who want to run an ethernet cable between their routers, or for people who want to restrict client to connect to their routers via an ethernet cable (i.e. clients cannot connect to their router via wifi!). These setups only make sense for an office environment that has particular security restrictions. Most likely you'll want to stear clear of these modes. Just remember that "Client Bridge" is not the same as "Repeater Bridge".

Firmware from DD-WRT

DD-WRT is a community of people that make free opensource firmware, and is often cited as the best solution for most people (citation needed). Unfortunately their wikis are a bit rough around the edges, and sometimes even constradictory. It appears community support has wanned over the years. Nonetheless I had good luck with DD-WRT, at the very least will point you in the right direction.

If DD-WRT fails for you, you'll need to confide in an alternative source ( https://en.wikipedia.org/wiki/List_of_router_firmware_projects ).

As with any firmware shinanigans there a chance you can brick your router. How big a chance? I don't know. But you weren't using that old router for anything anyways, right? What do you have to lose! (Even if you do brick your router, there are large sections of the DD-WRT wikis dedicated to recovering a bricker router). You can do it!

(create image: brick with wifi antennas)

and without futher ado:

How to setup a Linksys WNDR3700v3 as a repeater.

The firmwawre that Linksys WNDR3700v3 ships with does not support any repeater modes. Especially for router that are 5-10 years old, there's a good change your router does not support a repeater mode.

1) Give your roomates a heads up that your messing around with the routers.
If all goes well, you won't even touch the host router, and they can go on watching netflix without interruption. You will only be touching the secondary router. Nonetheless hope for the best, set expectations for the worst.

2) Download your secondary router's manual.
Google it. Read it. Confirm that it doesn't already ship with a repeater mode. This may sound like an obvious step, I only include it beware of terminology soup. (They might call it "bridge mode" or something stupid).

3) Log into your secondary router's web interface.
Any device connected to a router can alter the router's configuration, so long as you know the address, username, and password. Unless you've explictly setup new values, username and password will still be the manufactures default. All routers have a default password. For Linksys WNDR3700 the default info is:
address: 192.168.1.1
username: admin
password: password
Just open your web browser (e.g. chrome, safari, etc.) and type the address as the URL.

4) Update manufacture's firmware.
It's possible the manufactures has an updated firmware with what you need. Updating the manufactures firmware should be a one-click operation. With in the router's web interface, find a button to check for updates. Please double check and you may be able to skip this whole thing.
As of the date of this post the most recent Linksys firmware (v1.0.0.38_1.0.3.1) "Genie" does not support repeater modes.

5) Verify DD-WRT has a supported firmware for your router.
http://dd-wrt.com/site/support/router-database
Pay attention to the exact 'version' of your device. I.e. a WNDR3700v2 is way different than a WNDR3700v3.
Of the 3 routers I own, only 1 was supported. If the router you picked to be your secondary router is not supported, but your host router is supported. Swap them out, and proceed with the other router as your secondary. Or maybe trade routers with your friends.

6) From the DD-WRT website, download the associated .chk file. Normally installing firmware is a 2 step process: install the .chk (aka 'the flash' or 'trailing build'), then install the .bin. (The .chk is a base firmware, and the .bin is a more advanced and updated firmware). I found that the .chk contained both 'Repeater Mode' and 'Repeater Bridge Mode', thus I skipped all instructions to install the .bin. If you need your router's USB/external harddrive support, you'll probably need to install the .bin.

Ok, you're finally ready to start messing with the firmware. Get excited!

7) Connect your laptop to the router via ethernet cable.
-During the course of these instructions, the router's wifi will be severed a few times during power cycles and reconfigurations. Connecting via an ethernet cable is *probably* mandatory; otherwise you risk firmware not being applied correctly. I hope you don't own a mac book air.

(create an image of a mac book air dongle with $999)

(optional) And just to be safe turn off your wifi adapter. This will ensure that all your browser's communications are going over the ethernet cable. This will reduce confusion and false positives in later steps. I know I've screamed Eureka a couple of times, only to realize I was connected to the wrong router.

8) Give your laptop a static IP on the same subnet as the secondary router.
At the moment your secondary router is probably on the subnet 192.168.1.X; thus set a manual static (i.e. non-changing) IP such as 192.168.1.8.

(OSX pic)

9) Perform the voodoo that is the 30-30-30 Reset http://www.dd-wrt.com/wiki/index.php/Hard_reset_or_30/30/30 on the router. I had a stopwatch handy, otherwise counting to 30 gets really old really fast.

10) Log into your router's web interface (e.g. 192.168.1.1).
If you have the latest firmware (v1.0.0.38_1.0.3.1) "Genie", you will be propted to answer one of three choices. This is good; this is an indication that the 30/30/30 reset was successful. At any point during the remaining instructions if you mess up, you can alway re-do the 30/30/30 and re-start from this step.

11) Install the .chk
Select the choice “I wanna do it myself”; click OK for the "... are you sure?" dialogs.
Click “Administration” button (on left of the screen) > click ‘router update’
Upload the .chk
It will warn you about installing an old version. Click OK.

(pic; blur sensitive information)

12) More voodoo
The loading bar will take about a minute to finish. After it has finished, wait an additional 5 minutes. Then perform another 30-30-30 reset.

13) Log back into the secondary router's web interface (e.g. 192.168.1.1)
You should see a new "dd-wrt.com" interface.
These remaining instructions assume the following version is installed:
DD-WRT v24 SP2

14) Specify the host router's details.
Moment of truth.
Go to Wireless > Basic Settings
Here you should see 2 sections for Wireless Physical Interfaces, and 1 empty section for "Virtual Interfaces"

Pick one of the physician interfaces to use. The physical interfaces run on different frequencies; pick the freq that your devices most commonly connect to (probably the 2.4 GHz). You're welcome to try and apply these setting to both physical interfaces; I've only tried applying the settings to one interface.

For "Wireless mode" select "Repeater"
-If you do not see a value for Repeater, uh oh. You may need to install that .bin afterall; or your router might have a funky chip set. You may need to experiment with installing other versions of the DD-WRT firmware. Good luck.

For "Wireless Network Mode" select the same value as your host router. 90% odds are this value should be "Mixed". Otherwise, if you have web interace access to the host router, double check.

For "Wireless Network Name (SSID)" set to the SSID of the host network.

Leave "Sensitivity Range" as is.
Leave "Network Configuration" as "Bridged".

(pic; blur my mac address)

15) Add a virtual interface.
Click "Add" button
Populate "Virtual Network Name (SSID)". This will be the name of the wifi network that devices will connect to. This SSID cannot be the same as the host router's SSID.
Leave the other attributes as is.

At the bottom of the page, click the save button. (Do not click the Apply button yet.)

16) Specify passwords
Under the Wireless > Wireless Security tab
For the interface you selected in step 14:
For "Security Mode" select "WPA2 Personal"
For "WPA Shared Key" input the password for the Host router
Both the host router and the secondary router should have the same wifi password, and also the same security mode. This guide assumes the Host router has WPA Personal; otherwise, if you have access to the host router's web interface, log in and double check.
Repeat this for the virtual interface.

At the bottom of the page, click the save button. (Do not click the Apply button yet.)

17) Set the secondary routers on a different subnet.
By default, the virtual interface will be on the same subnet as the host router; ensure the virtual interface is on a different subnet.
Go to Setup > Basic Setup > Network Setup
For local IP address change the subnet (e.g. 192.168.2.1)

18) Ease dynamic packet filtering firewall restrictions.
We'll remove some normal firewall restrictions, (as was recommended by one of the guides). If you know enough about firewalls, you may choose to leave these on. Otherwise:
Go to Security tab
- Uncheck "Block Anonymous LAN Requests"
- Leave "Filter Multicast" checked
- Uncheck "Filter WAN NAT Redirection"
- Uncheck "Filter IDENT (Port 113)"
Then select "SPI firewall" disabled. (It's important to do this last)

At the bottom of the page, click save button.
Now click Apply.

19) Verify secondary router has internet
If you have another device handy, connect to the secondary router's wifi, and verify you have internet connect.
If you are having issues, there are some additional tips at this link: https://www.dd-wrt.com/wiki/index.php/Wlan_Repeater

20) Clean up; get your laptop back to normal
* Re-enable your laptop's wifi adapter
* Re-set your laptop's IP to DHCP

21) Enjoy!

So, at the end of the day I replaced my out of the box extender with a router configured to be repeater. Was this time well spent? Did I actually see any performance improvements? Initial signs say yes. Though I don't have any before/after wifi heat maps to back it up, I'll be able to report back with my measurements in 'number of roommate complains per week'. Or, if push comes to shove, I can leap frog even further, and boost the secondary router's range with the new collecting dust extender.

Guides I referenced:
https://www.dd-wrt.com/wiki/index.php/Netgear_WNDR3700
http://www.dd-wrt.com/wiki/index.php/Netgear_WNR3500L
https://www.dd-wrt.com/wiki/index.php/Wlan_Repeater
https://www.flashrouters.com/blog/2011/10/19/what-is-the-difference-between-client-bridge-wireless-repeater-modes-in-dd-wrt/
https://www.dd-wrt.com/wiki/index.php/Repeating_Mode_Comparisons
http://kb.netgear.com/app/answers/detail/a_id/23856/~/how-do-i-update-my-router%E2%80%99s-firmware-using-the-desktop-genie-app%3F?cid=wmt_netgear_organic


Sunday, May 8, 2016

Tripod & Pickup Shots

Tight framing on computer monitors

This week I started filming pickup shots for the computer monitors for the movie. During filming I had intentionally left all prop laptops with blank screens (because I hadn't had time before the shoot to rendered the fake data on the laptop screens). After a bit of GIMP-fu, I'm ready to finally replace the placeholder images in my timeline with actual footage.

My rendering of 'atomic collisions'
During the sciencey segments of the movie the sequence of edits will alternate between:
  • Back of laptop facing the camera (can't see screen) while the characters reacts to something on the laptop.
  • Cut to pickup shot of laptop screen, in a tight enough shot that the characters can't be seen.
Without access to my original actors or the original shooting locations I knew my shots would need to be tight to keep continuity alive. But at the same time I didn't want my pickup shots of monitors the be suspiciously close either. My original plan was to fill the frame 75% with the monitor and only the out-of-focus edges of the frame would be used to 'feather in' the scene. My goal was to re-stage the laptop in an environment that ball-park resembled the original set (e.g. place the laptop on a similar colored desk, use a similar colored wall in the background, etc.).

When project planning for this I had allotted quite a bit of time to 'reverse engineering' some of the sets in addition to trying to reproduce the same natural lighting. I had even gone as far as wearing similar clothes to the original actors, since my hand and arms would be visible on the keyboard. But when it came time to film the pickup shot I quickly realized a full-framed shot of the screen would be sufficiently believable. No hands in shot, no room details in the shot, no wasted time reverse engineering the set.

No two hands shake alike

The principle photography was shot completely hand held, and mostly by my hands. Though I was extremely conscious to avoid shaky hands during filming, most of the footage has a noticeable shake to it. In my opinion though this isn't a show stopper, since the omnipresence of the shaking throughout the movie quickly loses its distracting nature and settles into more of a stylistic signature. Thus, as it turns out, consistency in the shaking appears to be more important.

This realization hit home during filming of the pickup shots. I recruited my roommates to hold the camera while I drove the computers. Later, after I dropped the footage into the timeline, I felt something off. It took me a while to realize that each person shook the camera in slightly different ways (a bit more side-to-side, a bit more shaky in bursts, more shoulder shaky than wrist shaky, etc.).


Reading shaky text

As it turns out, my script relies pretty heavily on some exposition via text that appears on laptops and phone. Subversion, secrets, and other expositiony bits that I preferred not to be doled out through dialog. Additionally when the characters are experimenting with the machine, the information on the screens will be used to visually re-enforce the success or failure of their experiments. All in all, I am asking the audience to do quite a bit of reading during these pickup shots.
Shaky cam in all other shots are forgivable; but in the shots with text, it flirts with the 'reading while in a car' kind of sickness.

While a moment ago I touted the benefits of maintaining consistent shaking in all shots, these will have to be an exception. At risk of being captious, it is for this reason that I will have to re-shoot the pickup shots.

How to convert a rock band mic stand into a camera tripod

To truly get shake-free shots, I'll need a tripod. Instead of buying a tripod I very successfully converted an old mic stand (the last remaining rock band periphery that I still own). Using random parts that I had laying around, it took about 10 minutes to convert.

Stuff you'll need:
  • Philips screw driver
  • A bolt that fits the camera's tripod mount
  • A small L bracket with a large enough holes on both sides.
  • A medium sized spacer (or a few washers)
Procedure:
  1. The curved plastic part at the top is removable. Unscrew the bolt and nut that holds it in place. Keep the nut and bolt. Set aside the plastic piece. 
  2. Practice fitting the L bracket with the spacer in the groove where the plastic part used to be. Alternatively, if you don't have a right size spacer, you could cut off the bottom part of the original plastic piece and use it as the spacer.
  3. Practice fitting the tripod bolt. We want to make sure the tripod bolt doesn't have too much play (which would make the camera susceptible to rotation. You could use a nut to accomplish this, but I found that simply the head of the tripod bolt fitting snuggly against the top of the stand was sufficient.
  4. Screw it together: Place the tripod bolt in L bracket facing upward. Place the other half of the L bolt and the spacer in the groove. Use the original nut and bolt to tighten it all together.
  5. Spin the camera onto the tripod bolt.
  6. Loosen the bolt just a tad to allow the camera to tilt up and down as needed
Original plastic mic piece on left.
L bracket in place
 where the mic piece used to be.

Quick and easy.

Wednesday, April 20, 2016

Sprint 2 - Retrospective


Work-Life-Music Balance

Completion of Sprint 2 took longer than expected. Partial to blame is a recent expansion of duties at work (i.e. putting out fires with the latest release) which has resulted in long hours at the office. Also partial to blame: my past few weekend were dominated with attending music events. Umphrey's McGee, Chick Corea and Bela Fleck, the New Orleans Opera: Puccini's Tosca, and several bouts of karaoke with friends (ohhhhhh, sweet child of miiiiiine) :).

For Goodness Sake; Document your Deploy Procedures!

Part of my increased duties at work includes overseeing a production deploy. At the moment our production deploy is a small feat, requiring half a dozen people during late evening hours (when there is least impact to our users). It has been very educational. I've had to embrace making go/abort judgement calls, making risk assessment, exercise kurt communication during crisis events, and improving procedures to minimize any 'developer gets hit by a bus' scenarios (i.e. better documentation).

On the movie making front, at the end of each sprint I cut an updated version of the film. Naturally I couldn't help but apply some of my new-found deploy discipline. I've learned that even the most seemingly obvious steps are exposed for all their complexities (and kludges) once written out. Even a procedure so seemingly easy as 'backup footage' requires a couple hours to properly document. As annoying as it might be, it's always an extremely valuable exercise. Pedantic and fool-proof.
Take a gander at the result;

Synchronicity Move Release Requirements:

  • Access to project JIRA (https://synchronicitymovie.atlassian.net)
  • Access to the production computer
  • Access to the backup drive
  • Full procedure should take about 45min
Synchronicity Movie Release Procedure:

  1. Compile list of scenes touched during this sprint 
    • In JIRA, reference the scene labels of all the tickets closed during sprint.
  2. Export each scene in the list
    • In Premiere, for each scene, select the corresponding sequence in the project panel, right click, the select "Export Media". The Export Settings dialog will open; leave all the settings as is, then click "Queue".
    • After all sequence have been queue, switch to the Adobe Media Exporter application and click green arrow button at the top.
    • This operation will take about 15 minutes.
    • Close both Premiere and Media Encoder when the operation is finished
  3. Move the exported clips into their own folder
    • Go to the output directory for the film clips (C:/User/.../Adobe/Premier Pro/9.0)
    • Create new folder called Sprint_##. (e.g. "Sprint_02")
    • Move to newly exported clips into the Sprint ## folder
  4. Backup files
    • Retrieve the back drive from its secure location
    • Plug in the USB external harddrive.
    • WARNING: Shortly after plugging in the drive Windows will prompt with a warning dialog suggesting that the drive is corrupted. DO NOT CLICK THE OPTION TO SCAN AND FIX. (This will ruin the drive). Instead click the option to continue without scanning.
    • The drive will appear in explorer as "Seagate Backup Plus Drive"
    • Copy the Sprint_## folder to the external drive
    • Copy the Adobe Premiere project file ("C:\Users\spike\Documents\Adobe\Premiere Pro\9.0\Synchronicity_sc19_sc20")
    • Note: Ignore the fact that the file is named "_sc19_sc20". In reality this file contains work for all scenes. 
    • Change the name of the copy to include the current date as a suffix (e.g. Synchronicity_sc19_sc20_2016-04-20)
    • Copy the premier project file from the previous step to the external drive.
    • Wait for the copy to finish; should take a couple minutes
    • Eject the drive (read http://windows.microsoft.com/en-us/windows7/safely-remove-devices-from-your-computer)
    • Unplug drive from USB
    • Place the USB drive back to its secure location
Misc Release Notes:

  • There is no need to re-export every scene for every release. Only the scenes that have been touched.
  • There is no need to re-backup the original raw footage; the raw footage is already backed up. Only the premeire project file is changed.
  • The only purpose of the backup drive is ONLY for backing up movie files. 
  • Because of the confusing error message windows give NEVER lend the drive to anyone.

Evaluating Risk

One of the benefits of documenting your release procedure is it helps expose areas of risk. I'm talking about 'sink the company' levels of risk. Naturally I had similar insights after documenting my movie release procedures:

Current Sources of Risk:

Sound effects are not part of the back up procedure
Risk: Medium
Impact: Medium

  • This release procedure needs to be updated to include new sound effects and new raw footage added during the sprint.
  • At the moment my file organization for sound effects is attrocious (some files are in Downloads folder, some are on desktop, etc.)
  • I'll need to select a common place for sound effects to live, and update the file dependencies in the premier project
  • This represents some technical debt that will only get worst with each release.
  • Worst case scenario: About a week's worth of would would be lost.
  • Action Items: Ticket SM-159

Power outage during back up
Risk: Low
Impact: Very High

  • The production machine does not have a battery backup. 
  • Worst case scenario, a power outage occurs when copying files to the backup drive and both copies of the files are destroyed (about a 5 minutes window).
  • Raw footage is also backedup on another laptop; but we would lose all the premiere files. 
  • Worst case scenario: Over a year of work will be lost.
  • In New Orleans, expect about 5 power outages per year. Almost always in conjunction with heavy weather; thus do not perform backup during heavy weather.
  • Action Items: Get a UPS power supply for the production machine; Ticket SM-160

All sources of files are colocated
Risk: Very Low
Impact: Very Low

  • In the event of a natural disaster, all sources of files could be destroyed.
  • The backup drive is stored in a fire proof safe; but it is not water proof.
  • The backup drive is small and portable. The backup drive is already stored as part of my 'go bag'. It would only take a few seconds to evacuate the building with the backup drive.
  • Action Items: None

Looks like I'll be shopping for a UPS power supply pretty soon...

Sunday, April 3, 2016

Inspiration Corner

Creative Juices

Decor and aesthetics are usually at the bottom of my priority list. My work space at home usually consists of bare walls and filing cabinets. But after a recent conversation with my sister, (someone with in a degree in art/design), she explained the utility of decorating a workspace. Namely to surround yourself with images that inspire you. While I already have the left brain discipline to keep myself progressing on projects, a figured could use a little feng shui to keep my right brain piqued as well.

She also recommended maintaining separate areas for separate idea (e.g. she's had success with color coding work/life/play items). To this end I immediately identified with a need to compartmentalize ideas for my current movie and ideas for future movies.

So my goal was to create a space that was constantly visible, easy to pin up pictures, with separated areas for separate ideas. The goal here isn't to replace my existing organizational methods (I'll still maintain my dozens of google docs), but instead to keep the creative juices flowing with reminders of why I started a project in the first place.

A little bit of spring cleaning later, I had cleared a space in my room. Instead of picking a large space on just one wall, I opted for a corner space. Something I believe with help to encourage literally surrounding myself with ideas.

My space before decor.


Cork Board and Big Pad of Paper

Instead of one giant cork board (which wouldn't facility idea separation) I bought individual 12''x12'' cork tiles with adhesive strips. When envisioning the space I gravitated towards a spaced grid design, something that harkens to the chemistry of a periodic table or the physics of an elementary particle table (you know, nerdy stuff :) ). And instead of a pure grid I went a little tetrisy with it, which left gaps for me to hang up pictures/paintings.

While shopping I also grabbed a giant pad of paper; the kind you'd usually find in a classroom. This idea was from a piece of advice I had received last summer at dragoncon. I attended a panel in which leading board game designers took questions from the audience, and one response is particular stuck with me.
Q: What computer tools should I learn to use to help be design? Like maybe photoshop?
A: Honestly, I don't touch the computer until way later. Get a big pad of paper, so you can draw large diagrams quickly. And if you don't like it, just rip off the page, throw it away, and keep designing. You'll have hundreds of pages. Don't be afraid to sketch.

And for the final garnish, instead of using pins I'll be using novelty toothpicks with stickers from flags around the world. (to represent my ever increasing interest in geography and history).

Functional decor!

Tuesday, March 22, 2016

Sprint 1 Retrospective

My inaugural sprint. Though I put in more time than originally anticipated, some items didn't get finished.

Things that went well:
Re-organizing audio timelines
* The initial overhead of re-organizing will definitely speed up future timeline editing stories
* I discovered in premiere that I could label audio tracks; a simple enough feature that spurred me to start grouping audio clips according to their purpose. (Example: "Orig audio, VO character 1, Foley, BGM")
* Before the re-organization most of my sequences had audio crowded within 2 or 3 audio tracks; which could make simple edit laborious (rectifying audio clip clashed on the timeline)
* After the re-organization clip are separated along as many as 12 tracks. Less clashing when playing around clips on the timeline, and it also made it easier to eyeball the timeline and grok track coverage.

Things that didn't go well:
20% roll over
* One task took double time to complete the estimated.
* This was due to scope creep. Original time estimate was otherwise spot on.
* My goal was to only clean edits in the second half of the scene; but after digging into it I realized a strong dependency on pending work from the first half of the scene.

Things to improve on:
* I need to add 30min to account for track re-organization (only for the first audio mixing story in each sequence)
* Do cleaning on the first half of the scene first. It's less likely that the start of the scene will be dependent on pending edits on the second half of the scene.

Tuesday, March 15, 2016

Using JIRA for Movie Making

Tackling the unknown known.

Tomorrow marks the 3 year anniversary since I started filming my movie Synchronicity. (see post from 3 years ago). While the filming only lasted 7 days, the footage has sat in editing for over 2 years. It's been an epic hobby, one that I tend to in occasional bursts, but starting this week my leisurely pace is about to ramp up into full blown efficiency.

Part of what makes a project drag on is the difficulty of quantifying unknowns. While many of the tools involved with edit are things I've never done before, the overall project is divided into sections (number of scenes, number of lines of dialog, number of shots with special effects, etc.). The reality being, once the initial filming was completed, there weren't that many unknowns left in the project. All this to say, I have no excuse not to pin a deadline on the calendar and get my butt into gear.

Of all the tools I embraced for film making, I never thought JIRA would be one of them. JIRA is a time-tracking tool designed for companies making software, to facilitate the industry standard practice of 'agile software development'. It's a tool I've been using every day at work for the past 3 years, and I've come to find it as valuable for business collaboration as Word or Excel. And, as my engineering role has become more and more focused on project management, I eat, sleep, and breath JIRA.

Sprint 1. In general, work is grouped into JIRA 'epics' according to the tools/expertise involved.


I can already hear you laughing. Why would a project so rooted in creative flow use a tool as cold and dry as JIRA? The longer I stared at footage in Adobe Premiere watching replay after replay, I realized a scary amount of commonality between movie cuts and software releases.

My plan is to regularly cut fully length versions of the movie until the release date; v0.1, v0.2, etc. until the big day! The movie premiere! Synchronicity 1.0! Where as the similarity brakes down when you consider that software has the wiggle room to patch future releases; Movie 1.0 will remain the same for all time. (Though there is the occasional director's cut: Blade Runner 5.0... but I digress)...


It's never gonna be perfect

How many software releases have been 100% feature complete? ZERO. A thousand good ideas reduces to a hundred good ideas and further down to a dozen core features (without which the product would be dead on arrival). Imagine software designers and architects as equivalent to a movie's director or visionary. These people are talented and creative enough that given 5 years they could make the most beautiful masterpiece ever.

Nobody's got 5 years. We ain't all James Cameron. Like it or not this titanic is gonna have a little rust on it. I could spend an eternity meticulously crafting every frame, but at some point someone's gotta see it. It's learning to compromise. No. It's being forced to compromise. Because without that force it's just "one more fix here", "ooo, and another small fix here" and "maybe if we rearrange this is could turn out better"...

Imagine your movie producer: The film is scheduled to hit theaters! We need to see progress on the film! Does it work as an overall story? Until then, ignore the bells and whistles!

James Mangold, the director of 3:10 to Yuma (2007) had planned to build an entire wild-west village for his flick. As they ran up against their budget, set building had to cease, and some houses were left exposed as little more than a collection of 2x4s. But at the end of it all he managed to dazzle the audience with a several minute chase sequence from one end of the village to the other. He achieved all the emotion and excitement needed for the climax of the film, despite Russel Crow and Christian Bale running through half built sets. Two extra houses wouldn't have made it a better experience.

Imagine your software CEO: We gotta ship it! We need to do regular releases! Do the core use-cases work? Until then, ignore the bells and whistles!

Bill Gates, the CEO of yadda, yadda, yadda... insert your own inspiration reference to how DOS 1.0 shipped as a piece of crap. :P

Reality is a B****

Before putting my remaining editing work into JIRA, I made an intuitive guess that I had 8 weeks of work left. Afterwords I spent several days cataloging all my desired fixes into JIRA (everything from single frame tweaks to major plot point reworks). Using my existing editing experience I was able to breakdown and justify each task into detail. 

Example details for a ticket. A specific set of high level requirements, a proposed implementation, and an itemized list of time.
When the dust settled I had created 135 tickets and 30 weeks worth of work (with my measure of 12 hours per week). This is the reality check. I'm dreaming too big. Gotta get back to the core.

The next step will be the hardest. To pick a calendar date sometime this summer to premiere my movie. A date that will likely end up 10 weeks earlier than my proposed work. I'll commit to it by organizing the premiere event with all that my friends involved in the film. I'll be forced to drop a 1/3 of my originally planned edits, all with a workable final produce in mind. Priorities will trickle down quickly.

Another great exercise in intuition vs. measuring. Keep an eye out for the burndown charts. :)