The Rough and Tumble of Tech

Sébastien Heins is an actor, writer, producer and director based in Toronto.
Torien Cafferata is an immersive theatre artist based in Saskatoon.

Instigated by Cohort, these artists undertook an email exchange about immersive theatre, technology and the cultural sector. We edited, but not much. Here’s the first of four parts of their extended conversation.

On Sat, Aug 8, 2020 at 6:41 PM Torien Cafferata wrote:

Hi Sebastien!

My apologies, our Fringe-but-actually-Not-Fringe Fest just ended, as did our fest run of Play Fail Win (PFW). There is as good a place to start as any! PFW is an interactive first-person play experiment in which a Zoom room of players voice-commands The Guy (a live performer) in an outdoor sandbox adventure game all about embracing embarrassment and satirizing rewarded incompetence. Our biggest challenges:

a) giving the performer access to the player HUD [heads-up display] made thru Open Broadcast System while also being able to climb trees and roll down hills, and

b) how to navigate the public social character-building / dialogue without it becoming too awkward or unwieldy, without the use of NPCs [non-player characters]. Definitely things that can be resolved with more resources as we grow the project and test out better AR gear; super curious about Oculus VR headsets, haptic input from player to performer or vice versa (gloves, vests, etc.), allowing players to take turns being each other’s real-world avatar.

I’ve found myself feeling way more at home with this project, despite the rough-and-tumble tech, because it speaks more to my gamer logic and game studies brain, even though my training is in theatre acting. What sort of challenges have you experienced specifically in the most interactive and digital of your works?

Thanks again for doing this :)


On Sat, Sep 12 at 3:32 PM Sébastien Heins wrote:

Dear Torien,
Thank you very much for the trip through your journey thus far making immersive, auditory, tech-powered experiences! I really enjoyed your videos and project breakdowns — your Overhear looked beautiful and was so life changing for its audiences. Congratulations on all of the well-deserved success, and for completing your fest run of Play Fail Win. It sounds like we share a lot of interests and curiosities with our projects.

Our project, The Itinerary is a live action video game that lets you control a stranger’s actions on a very important day of their life. Using your smartphone, you help them explore exciting memories, their imagination, and an emerging harsh reality sitting just below the surface. Groups of six players at a time jump into a story about the Caribbean Windrush generation, living the dream of “young, gifted and Black,” and a lifetime of joy and tragedy a hundred years in the making.

The Itinerary: Playtest, which we did in residency at The Kick + Push Festival in Kingston, Ontario, was a version that allowed us to test a number of theories, including tech infrastructure, media connectivity, story containers, and provided all sorts of opportunities for problem solving. The visuals for the meat of the game (the main character doing activities in a room reminiscent of the Hotline Bling music video meets a cliffside cell) were streamed over Zoom. Using multiple cameras in two rooms we set up a Mission Control room and the Sound Stage. For COVID purposes, we designed the running of the show around the goal that no two team members should be in the same room at the same time.

Mission Control was for our Stage Manager, the excellent Alice Ferreyra, who used Isadora, as set up by Melissa Joakim, our amazing Production Designer, to cue lighting shifts denoting times of day, weather, the radio starting to play, sound effects to signify an audience’s choice had been made, as well as video overlays which served as transitions between major story points.

We also used Switcher Studio as our video mixer, researched by the brilliant Jacob (he’s so smart, eh?), which pulled multiple live camera feeds from Cohort iPods set up in the Sound Stage to pick up the performer as well as in front of the mini games that comprised our climax, “The Dream” (think Mario Party and The Sims meets Big Fish).

For sound and team communication, we relied on a capricious mixture of a wireless Lav mic on the performer (who also had Cohort AirPods to keep them in connected to the SM and the production team if need be) and two Discord audio channels, one set up for the production team, and one for the performer, plus the Zoom audio feed which served the audience/players. We’re going to have the opportunity to give the show its world premiere in Q1 of 2021, online and potentially in-person, in accordance with the most up to date COVID guidelines.

The Itinerary: Playtest at the Kick + Push Festival was an opportunity to test some major questions:

  1. Can we develop a reliable, web-based app which allows the audience to effectively control the action?

  2. How will they know if it’s their turn?

  3. How will latency affect the audience’s enjoyment of the show?

  4. How can we mask latency?

  5. How do we improve the internet speed in a building with a cap on internet infrastructure? (Answer: We ended up getting a Bell Mobile LTE SIM, which we routed through Cohort’s TurboHub)

Best of all, we got to have real audience members trying out the game at home, and learned from their home-gaming habits, like what happens to the web-based app if someone gets a text message, etc.

Right now, we’re having a great time working with Cohort to develop free tools for live performance makers who are also interested in audience-to-performer control, overlaying mo-cap video effects, and low latency side-channel feedback loops.


Torien & Sébastien’s exchange continues in The Rough and Tumble of Tech - Part 2.