So, thanks everyone for being here.
I'm excited to talk about
"Virtual Environments
for Communicating Changing Forests."
Before we get into that,
I figure I should introduce myself.
I'm Mike Mahoney.
I'm currently a third-year PhD candidate
at SUNY-ESF,
and I work with Colin Beier
who spoke this morning,
looking at ways that we can use
new tools and techniques to help people
better think about and understand
complex large scale systems.
Things like forests,
which are processes that happen
over landscape scales,
and over temporal resolutions,
that don't really fit with the way
that humans normally think about things.
A lot of the processes that we need to care about
when we're making management decisions,
are things that are really difficult
to actually go and watch happen in real time.
And so,
I've been thinking a lot recently
about ways that we can use technology
to try and make these systems
fit better into human brains.
And specifically, how we can use visualization
and these virtual environments
as tools to do that.
Which brings up the question,
what is a virtual environment?
And as Aiden sort of was talking about this morning,
the virtual environment is a digital world.
It's sort of thing that you might have
in a video game,
where it's a realistic space
that you can walk around in,
that you can explore,
and that you can interact with.
And the sort of virtual environments
that I'm personally interested in,
are the ones that are attempting
to represent realistic places.
So this for instance, is a screenshot
from Aiden's virtual environment
depicting a forest in Berkshires,
out in Western Massachusetts.
And when you're doing this sort of thing,
or if you're taking actual data
and representing a real world place,
virtual environments are just a new form
of interactive data visualization.
And they're a type of data visualization
that makes the system that you are visualizing
extremely explicit for your viewers,
for the consumer of the visualization.
Which means that these are really interesting tools,
because they can help to reduce
the cognitive load that's associated
with normal visualization methods.
So for instance,
choose an example from sea level rise;
the way that we normally communicate things
like sea level rise
are maps that look like this.
This is a map of expected sea level rise
in Boston, Massachusetts,
over the course of the next century.
Those light blue polygons are the areas
that we're expecting to be impacted.
And for that piece of information,
this is an effective way to communicate that message.
You can look at this
and see all of the areas in the city
that will be impacted.
But once you want to move
beyond that piece of information,
this sort of visualization asks you as the viewer
to do a lot of work,
because to turn these light blue polygons
into what impact actually means
for your context,
for your situation,
and to make decisions about
what that information means for you
that requires that you come to the table
and have an understanding
of what sea level rise looks like,
and what it actually is going to do
to the areas you care about.
Which, most of the time works great
when we're using these sorts of maps,
but sometimes you might have
an audience member who says,
"Oh, I should buy property right next to it,
-so I have a seafront view." -(students chuckling)
You're relying a lot on the context
that your viewer brings
to the visualization with them.
If instead we make this much more explicit
using virtual environments,
if we render somewhere that someone cares about,
if we render their house,
and the expected sea level rise at that point,
we can get rid of a lot of that
implicit information that we're expecting
the viewer to bring to the visualization.
We can make the system much more explicit,
and we can make it so that
there's a lot less work required
for the viewer to get the actual intended message
of what we're talking about.
Now,
there's obviously some trade-offs of doing this;
you can't look at this screenshot
and tell all of the areas in Boston
that will be impacted by sea level rise,
we're trading breadth for depth here.
But in combination
with other visualization methods,
virtual environments can be a really useful way
to communicate with people
who maybe don't share the same
mental model of a system, that you do.
And to that end,
people have been using virtual environments
for a ton of communication education projects.
Aiden had a paper last year
where he was using virtual environments
to talk about how competition
shapes, stands, structure,
and composition, over time.
By taking people and putting them
in a virtual environment,
and letting them skip through time,
you can actually have people see
competition between individual trees,
and notice how more successful competitors
wind up moving into the canopy
while their neighbors wind up
getting suppressed and dying off.
You can take this
normally extremely abstract process
of forest competition,
and make it something that's much more
intuitively understandable.
Similarly, groups are using this for education
about how forest composition will change
as an impact of climate change,
into the future,
and show people how that process
might play out moving forward.
And there's also projects
that Aiden's involved with
using this to communicate things
like slash walls,
and other forms of deer exposures,
as ways to control what regenerates,
following management activities.
Now the other use case
for virtual environments right now,
that I personally am really excited about,
is that you can also use
these two informed decision making processes,
because if you're trying to, for instance,
cite a wind turbine,
you can render the environment
that you're putting wind turbines in,
and show your stakeholders
multiple different locations
that you might place these objects.
And you can then get their expressed preferences
and use that to inform the decision you actually make.
If you're deciding on forest management activities
you can show the expected outcomes
after management is done,
and 10, 50 years into the future
what will regenerate
as a result of the activity.
It's a way to get more informed decision making
from people who don't have the same concept
of the system that you might be working with.
It's a really useful way to inform decisions.
And so, right now virtual environments
are a really exciting area,
it's a really vibrant area of research.
I'm really excited to see it going forward.
But,
there's a challenge with how
we actually use virtual environments
in practice right now.
And the first part of this
is that as you might guess,
it is really difficult to make a video game
out of your data. (laughs)
These things are hard to build.
The sort of tools that you need to use right now
in order to actually construct a virtual environment
aren't things that are usually taught
in an ecology degree,
they're 3D modeling softwares,
they're video game engines,
they're things that aren't part
of normal scientific workflows,
which means that this is a huge
technological barrier,
to actually using virtual environments.
But also,
because these are tools that are usually made
for making video games,
making movies, making art,
they encourage a way of working
that is much more artisan
than a lot of our other visualization methodologies.
The idea is that you are purposefully
crafting an experience for the viewer,
for the person playing your video game,
looking at your 3D models,
which means, that right now
I'm gonna steal an analogy
from a colleague of mine.
Virtual environments in practice
are kind of like wood carvings,
where you can spend as much time as you want
making them absolutely beautiful,
making them extremely faithful
to the area that you want to represent,
and making it so that they are
the perfect representation.
And then if you get new data for the area
that you need to visualize,
or you need to do a new location,
or a new type of data,
you have to go and get another log.
There's not a way to adapt that work
that you've already done
to apply it to new spaces, or new data.
And so that's a real problem
with the re-usability of virtual environments,
and practice right now.
But it also has implications
for reproducing and replicating research
that's being done,
or decision making processes
that are using virtual environments.
Because if you don't write down
every decision that you make
while you are sculpting a virtual environment,
then no one will be able to reproduce
the same environment as you.
That information will just be lost,
especially in the case of implicit decisions
of deciding to not do something,
while producing a virtual environment.
That's a lot of information
that's really difficult to capture.
And as far as replicating results
and processes that use virtual environments,
well, if you get new data,
you can go through the same process
and build a new virtual environment,
but then if you can't replicate someone's results
it might be the case that the results don't replicate,
and it might be the case
that you just made a worse environment than them.
Because these things are made entirely sculpted
rather than any form of automation,
it's really difficult to tease those things apart.
And so, that's what I've been working on recently,
is working on tools to make it
so that we can make these in a way
that fits better with standard scientific workflows,
in a way that enables more reuse, reproducibility,
and replicability of virtual environments,
and makes it so that they're more useful
for real world projects,
and the sort of data that you actually might have.
And to that end,
I've been working on a set of packages
for the R programming language
that enable you to create virtual environments
through code,
through something that you can then archive
alongside your data,
and send to people who want to know
what decisions you've made
while making these video games of your data.
I'm not gonna go too deep into the details here,
but to give a brief overview of the system,
unifir is a package
which connects R to the video game engine.
It lets you write R code that looks like R code,
but when you execute,
it creates physically rendered objects
inside of the game engine.
On top of that,
I've got these other packages,
terrainr, and forthetrees,
which take data that you might actually have.
So terrainr for instance,
takes like elevation rasters, and so on,
and turns it into physically rendered terrain
inside the game engine,
while forthetrees is going to take data
that you might have about a forest,
either summary statistics of trees per acre
and so on, or actual STEM maps,
and turn that into digitized forest renderings.
And the idea is that those packages
make it a little bit easier for the user
and for scientists and researchers, and so on,
to actually produce these visualizations.
They don't require you to think about
the game engine.
Using these packages in concert,
you're able to write R code, that again,
just looks like R code,
it's not anything wild,
we're not doing anything silly here.
And you take data that you can read into R,
you run it through these systems,
and you produce these virtual environments
in the game engine relatively automatically.
And then you can take that script
and you can archive it,
it has every decision that was involved
in producing this,
and it can be run again and again
-as many times as you need to, -(audience member coughs)
which means that both people
can reproduce your current environment,
but also you can reuse the work
that you put in making this complicated thing,
and apply it to new locations or new data.
If you have multiple measurements
at the same site,
if you're trying to show people change over time,
you can use that same script
and just change the inputs to it,
and make multiple environments all at once.
And so for instance,
I've got access to this data
from up in the Adirondacks,
where we went out and...
Well, we,
I was not involved even a little,
(audience chuckling)
but they went out and measured
in 1985, and 2000, and 2009, the same plot.
And the big story here
is that between '85 and 2009,
beech bark disease start really impacting the stand.
And so these green circles,
that are very washed out I think,
are the beech trees in the stand,
and you can kind of see between '85 and 2000
some of those circles start disappearing,
you start getting more holes in these maps,
as the older beech start dying off.
But then,
really between 2000 and 2009,
the story is, that as those older beech
start dying off
they start setting up the beech whips,
they start blocking off any other regeneration
that's on the ground,
and they start filling in any gap in the canopy
with this advanced regeneration of beech.
And normally this is the way that we visualize
that type of information;
we take the data, we run it through,
in this case it's GD plot,
and we make a top down STEM map of that.
But now using the pipeline that we've got
for creating virtual environments from data,
you can do the same thing
inside of a video game engine instead.
And this is an interesting way
to visualize the sort of information, because again,
it makes it more explicit
what's actually changing in the stand.
And so you can see on the right side of this,
that is not the pointer,
you can see on the right side of this over here
some of these light green trees
which are the beech models that I was using
for this environment,
between this '85 snapshot
and the 2000 snapshot,
start to be removed.
And you can see the older larger trees
being removed,
as the progression of the stand.
And then again, between this 2000 snapshot,
and 2009,
you can see more and more of those canopy trees
being removed, and starting to get replaced
by a lot of smaller regeneration down here.
And now this rendering is the same sort of thing
that you might get out of something
like forest vegetation simulator,
like it's pretty simple and top down.
But the really cool thing
about using this system
to produce these visualizations,
is that you can then drop in
as a player character,
and actually walk through the environment
and see what this change means on a ground level.
You have more ways of actually investigating
what this data looks like, in real life.
And so between the 1985 interval,
between this and 2009,
you can watch how these sight lines
wind up getting closed off
by more and more advanced regeneration,
as these beech whips start to emerge
and start taking up all of the light
that was left in the stand.
So, that's where we're at
with this toolkit right now,
we're at a place that you can make
these visualizations for any data that you might have
that you can get into R,
and you can archive your script
so that other people can reproduce
and replicate your work.
Going forward, I'm trying to make it easier
so that you can use these tools
to actually visualize the areas
that you might care about in other forms of data.
But the really cool thing
is that all of these components are openly,
freely available online now,
so if you have something that I haven't predicted,
if you have a need
that I have not developed tools for,
you can go out and change my scripts
so that it works better for the data
that you actually have,
so you can visualize the things you care about.
So, I wanna thank you again for being here today,
if you're interested,
you can find me online everywhere
@mikemahoney218
and these slides are available off my website.
Thanks.
(audience clapping)
I have no idea if there's time for questions.
[Host] Yep, yep, we have questions for you.
Yeah?
[Audience] I like your idea of using this
to help inform decision making.
So without getting into specifics,
how could we,
let's say we want to reduce the basal area
on a five acre parcel, by 50%.
You could theoretically take the data
and graphically depict
what that basal area reduction would be,
from you know, 140 to 70,
and then use that to inform the landowner
that this is what your forest may look like.
Exactly.
And part of the initial thinking
with this sort of system,
is to take the outputs from things,
like if you're doing community simulation
using the SORTIE model,
that's spitting out,
"Oh we're expecting that the forest
20 years after we do that reduction,
will look like this."
The idea is that this system
should be able to speak to
that sort of standard data that you might have.
And because it's coming out, as,
say you're doing
if you've got summary statistics
that are, "I'm reducing the basal area,
and I'm expecting to have this many trees."
We can regenerate a STEM map
and run it through the system.
If you have something
that is more explicit than that,
if you've got like individual trees in a CSV,
you can go through and delete rows
and visualize what will this look like
after I remove these trees.
Yeah?
[Audience] Where do you get the data
for the...
You know, like the characteristics of the bark
and kind of all the detail behind the data?
Yeah, so if you look at...
I'm going all the way back.
So an interesting thing with these setups
is that you have some virtual environments
that look like this,
this is one that Aiden
spent a lot of time like manually creating.
And then there's some that look like this,
which is more automated
so that it can show differences between trees
as they are competing.
So this is more automated
so that the trees are interacting with each other
as they grow.
And then we've got
doo, doo, doo,
these that I was showing;
you can see obviously,
very different levels of realism between them,
because we're pulling right now
from like purchased 3D models,
that someone else went through
and created these trees,
which is why
I think I've got two versions
of this particular beech.
Like they're throughout here,
but the same tree is repeated multiple times.
That's my current approach
is what got off the shelf assets that we're using,
and so for things like,
this is the stand impacted by beech bark disease,
none of these trees have beech bark disease,
and that's a real limitation with this approach.
There's ways to use other tooling
to make it a little more aware.
So like for instance here,
which tree is being planted
is a little bit determined
by how big it is.
So, which 3D model is used,
is determined by how big
the tree to represent, is.
And so you can do things like that
if you have a flag for beech mark disease.
You can have models that have that texture
and models that don't,
but that is still an artistic process
to actually create those models.
Yeah?
[Audience] I'm really naive on this subject,
so I apologize this (indistinct).
Can you add critters?
Yeah.
[Audience] So,
I tell you a concern I have,
especially about this municipal forest
going on in Santa Monica,
management plans with 'em,
and some folks try to do
the right thing scientifically,
and they look messy,
and all of a sudden a couple people go bonkers.
But I would think most of the people would be happy
if they could see a lizard,
a salamander, or a frog showing up
because it was a messy forest,
versus a clean park, like.
So, I unfortunately don't have the video,
but Aiden, on his website has one,
that's talking about
how management can improve
carbon sequestration over time.
And in that video part of it, is,
they come through, do the cut,
and then there's deer browse,
and like that's visualized in the scene
'cause you're able to add pretty much arbitrary.
So each tree here
is just a 3D model made up of other pieces.
And so, I can just point it to,
to instead of the folder that has a tree,
I point it at the folder that has a deer,
and the same tool works the same way.
Yeah?
[Audience] So, is there going to be
a paywall associated with this?
Like you have our program set up,
and all the data's available
and we can like tweak and modify,
but if you wanted something specific
or a specific scenario,
you would've to purchase
the actual 3D model from someone generating it.
Or can you-
Yeah, so,
something that is in the long term vision,
but I'm putting no dates to,
is better, open access,
of actual models.
So right now,
there's some amount of like tree models
that are available freely;
they're not the good ones,
(coughs) but so...
So all of the tooling to actually produce these
that I'm working on,
all about open source that is already released,
the Unity game engine
that these are actually rendered in
is a commercial product that you can use for free,
once you start selling the outputs from it,
you can no longer use it for free.
And then the assets in here
are, in this one, purchased.
So there's places in this system
that are obvious next targets
for how do we make these more open, as well,
that you can archive the models that you used
alongside the script,
and release that publicly.
[Host] That's all the time we have for this talk, so.
[Audience] Let's give a round.
(audience clapping)