Welcome to Tools and Techniques.
I'm Xana Wolf.
I'm a web database developer at the FEMC,
and I'm giving the first talk.
I actually recorded it.
I didn't need any more stress today.
(audience laughing)
(Xana laughing)
I'm gonna take it off
because it's a little bit long,
but if you have questions
that we don't get to at the end,
I'm around for lunch
and the poster session.
So feel free to (indistinct).
Welcome to a Tour
of the Northeastern Forest Inventory Network
or NEFIN.
Many of you have probably heard
of this project as it has been
in the works for a few years now.
But if not, what is NEFIN?
Essentially NEFIN is a searchable database for CFI data
from a variety of sources.
So as you can see,
we looked at a number
of programs with studies on Northeast
forests going back over 60 years.
Some of you may know
about the precursor to NEFIN,
an FEMC project from a few years ago
called the CFI Program Comparison Tool,
which provides details
on the methodologies of various CFI studies.
So if I clicked here under volume,
biomass and carbon next to mass CFI,
you can see information about
what mass CFI is collecting in that area.
And here's a view with all
of the metadata from mass CFI,
plot density and layout.
Things like expansion factors,
codes for things like crown class
thresholds and standards
like minimum DVH and also things
like sapling and seedling
attributes and forest health indicators.
So once we had collected all of this information
and analyzed it for NEFIN,
we were able to come up
with a set of standard fields.
We came up with about 60 of them
and we correlate them to certain fields
in a programs data set.
Very important things like DBH and height,
which you can see belong to a tree category.
So each tree can have a DBH and a height.
One of the things we are also able
to offer programs
is the option to fuzz their plot locations.
So a plot latitude and longitude aren't
available to the public.
This is a program level definition
and you can see that these fields
are marked private,
so they won't show up on the public website.
Then we have a few more tree metrics,
species, sample year, ID,
and then we have forest health impacts,
which signifies an issue in forest health.
So I'm gonna pull up a list of them
from our database so you
can see how these are done.
So for example, you can see FEMC
forest health monitoring has bigger
dieback, defoliation and discoloration,
and a vigor that's in three, four, five
or eight will identify a forest health impact
or a forest health concern.
A dieback of greater than 55
or defoliation of greater than zero would
also indicate a forest health impact.
We go back to our list of metrics.
We can see more things like plot sample year
and plot ID that are plot categorized
a sample year for saplings.
And then we have a bunch
of program level definitions.
And like I said, we have about 60 of these fields.
So here is a little visualization
of the NEFIN processing workflow.
So you can see these programs
on the left have differing data.
They have different IDs,
they have a different field name
for their species
and different codes for their species.
They could have different units
that they're measuring things in.
So the data is fed into the NEFIN
pre-processor where we have scripts
to match the program's fields with our standard
fields and translate codes
like crown class or species codes.
We save a copy of the original uploaded data files,
code list, et cetera as a snapshot.
And then we process the data
into our own standardized data
tables with our standardized projected
coordinate system units and codes.
And we keep a record of how
to match each program's codes to our codes.
The data can be loaded, again,
if the original data needs to be updated
because of QC or if we have additional years
monitored as time goes on,
we can just put that new additional data
into the program and then at the end,
the users front end users can access
our website to search our standardized data
or download copies of the original data.
All right, so now let's look
at how we actually load data onto the website.
So NEFIN is not currently public facing yet.
We're very close,
but we're just wrapping up with our own QC
and we haven't moved everything
to its permanent home yet, but soon,
and I'm sure we'll announce
that through our FEMC social media and newsletter.
So definitely subscribe
to those if you don't already.
So for loading data,
the program manager would have
to log in to get to this page
and this is where they would be taken
to begin uploading their data.
You can see at the top the steps
they'll have to go through.
You'll upload files, match fields, check code lists,
quality checks, ancillary file uploads,
and then review and submit.
So in this example,
this is a MEER test case
that just means that the files
we're uploading will be in the format
that the main ecological
reserves program provided to us.
Right now we have to pre-set
up the structure for each
program prior
to having their data loaded.
So we know what format to expect.
We're planning to make
a generic template in the future so
that a new program could format
their data to fit our structure
and we wouldn't have to do
as much setup to bring in a new program.
But for now, we need to know which columns
in their data contain are
required fields such as plot
or tree IDs and sample years.
So first we'll want to enter the years
that our data contains that we're uploading.
So I know that I have 2002,
2004 and 2010 in this data set.
And then I'm gonna confirm
that the data set field names
that I uploaded prior still
match these fields.
In the future
we are gonna move this around a little bit,
so this step won't be necessary,
but for now I'm just gonna confirm
that that's all the same.
And then we're taking to the step two,
upload our actual files.
So you can see here we give you the name
of the last file that was
uploaded so that you can go
and check that your
field names are the same or you can match that
these names are the same when uploading.
So I already selected these.
We're gonna upload and pre-process.
Now that the files are pre-processed,
we have to match fields.
You can see our NEFIN
standard fields are listed on the left.
And then there's a dropdown here
with all of the data field names.
So these are all of the fields
that were in the data.
So now if we had changed the name
of our DBH field and there
was no longer a DBH field in the data,
this would be blank
and it would have a red border around it
'cause it's required
and you wouldn't be able to go to the next
step until you chose a field that matched DBH.
So you can also set units for your field.
You get a little visualization
of the sample data there,
and then you can put in notes
or a description of your field.
So these are all tree fields.
Under this tree setting
you can see one of these forests
health impacts fields like we talked about before,
damage type in the mirror data
if it's greater than zero,
that would indicate a forest health impact.
And then we can look at our plot fields,
our sapling fields,
our seedling fields,
and then program information here.
So you can update your expansion factors,
your coordinate systems,
make sure that everything looks good.
And we're gonna say that that's good.
And go to the next page.
On this page we are going
to match any codes.
So what this is telling us is
that there are no new codes in
our sapling data,
our seedling data for size class for species.
Oh, but look,
we have a new code in our tree data.
So I went in and I added this underscore test
to one of the,
I guess that's paper birch.
So it came up as a new code.
So I'm gonna enter paper birch test
as my code meaning,
and then I'm gonna match it to paper birch.
And you can see I can search by common name
or scientific name.
And now I've matched those codes
so we can go on to the next step.
Okay, now this is quality control checks.
So we've made decisions about what we want
to allow into our standardized database.
So here you can see we have a warning,
non-numeric value,
empty blank found in field 13 height
first occurrence found
in line 12 and occurs 16 times.
So warnings are just warnings
and you can continue on
and the data will be included
in our database for better or worse.
The next thing you can see
is an error out of range value
zero centimeters found
in field five DBH range is five inches,
12.7 centimeters to 300 centimeters.
First occurrence found line eight
and it occurs two times.
So row with errors will be removed
before entry into our database.
You may fix the errors and return
to step one and upload
files to upload revised files
or click next to accept removal.
So what this is saying is that (indistinct)
has defined a minimum tree DBH
of five inches or 12.7 centimeters and we found a
DBH of zero centimeters,
which does not meet that criteria.
It's not within five and 300 centimeters.
So we are gonna say that
that does not meet the definition
of a tree as defined by the program.
And so we're gonna remove those rows.
Just to switch back
to the PowerPoint for a minute,
these are our NEFIN standard ranges.
So if a program does not
provide ranges for tree DBH,
we're gonna say that the DBH
has to be between 10
centimeters and 300 centimeters.
Sapling DBH has to be between 2.54 centimeters
to 10 centimeters and tree height
has to be from zero centimeters
to 5,000 centimeters.
But if a program has a different definition,
we will honor those.
So I am gonna go ahead and accept
that those rows with zero
DBH will be removed.
Next, we're allowed
to upload any additional files,
field protocols, information about your code lists,
et cetera that you wanna include in this package.
So I am going to include
the monitoring plan from 2003
and then I can add more files
or remove this file.
I'm just gonna upload that one.
And then finally we can review and submit.
So I will give this the date.
(keyboard clicks)
And we'll say years 2002, 2004 and 2010 loaded.
And I'll put my name.
Now we can review what our input files are.
Oh, and look, we have a warning.
So the years we've specified
for this version overlap with
existing data already uploaded into NEFIN,
so I guess the year 2002 was already in there.
So what'll happen is if we proceed
with clicking submit version,
the existing data associated
with these years will be
removed and replaced with the data
that we are about to upload.
The previous version of data
will still be preserved in the version package,
that snapshot that was taken
at the last upload.
But the standardized data
that you can search in the front
end of NEFIN is gonna be updated
to what we are uploading now.
And I'm gonna say that that's okay.
And then we can review our standard metrics,
our program definitions, all of these plot trees,
sapling metrics,
our new codes that are going in
and our ancillary files that we upload.
So I'm gonna submit that version.
So we can check our upload
status and once it's finished
we'll be allowed to upload
another version if we want.
So that's how a program will
upload data into the system.
So let's now check out what it's like
to search the data on the front end.
So again, this isn't public yet, but stay tuned.
So the homepage just gives
a few summary statistics like
how many plots, trees, and species we have.
And these numbers will update
every time data is loaded.
And then we have a little chart below
that for top 10 species by basal area,
which is just a whole lot of sugar maple.
So the bulk of the utility of this site
is in the get data page.
Here you can search through
our various entities, programs, plots, trees,
saplings and seedlings using
different criteria on the left.
And these criteria will change
depending on what entity you're looking at.
I'll expand this for now.
So I can search by program, by state,
or by time range as far
as the programs are concerned.
So say we want to search for everything in Vermont
that gives us five programs.
And then let's say I only want from 2020 to 2022,
that narrows it down to four programs.
So if I wanted to download my data,
everything from those four programs,
I could either go to this download tab
or this link where it
says download results
and it takes us to this download tab
where I can download a package
that has all of the original
data, that snapshot that we uploaded,
all of the standardized data
that we were just searching
for, and then all of those related files.
And then this option is
to include the original data in the
standardized data file called a dock field.
What we've done is we've taken
this original data and if you
have a tree observation, we've stuck a JSON object,
basically a a string of all that original data
into one field in our standardized data.
So if you wanted,
you could only download the standardized
data but include that original data
in the doc field so that you would get both.
Then you also have the option
to just download these files.
So these are grayed out these packages
because these programs have chosen
to fuzz their plot data.
So you're not actually allowed
to download the original data from them.
You would have to contact us
or contact the program to see
if you could access that original data.
But FEMC, we're not fuzzing our plot.
So you can actually download the package.
So this would be our original data package.
And then you also have these scripts alongside.
So these are our pre-processing scripts.
There's one for each project.
And then allfuncs just has common functions
that are used by other scripts.
So that would be downloaded as well.
So I'm gonna wait to download
until we refine our search a
little bit more so we're
not downloading quite as much stuff.
So let us look at the plot table.
You'll see when we change tabs,
our search criteria remains.
So we're still searching Vermont year 2020.
And then this actually changed
to 2020 because I know we
don't have any plot observation data past 2020.
So it updated to the the max year
that's in the database right
now, which happens to be 2020.
So if we look at this little tool tip here,
it says click link to see original data
if program allows or program info.
So I know that NAMP fuzzes their plots.
So if we click on this,
we're getting all this information about the program,
we have program details,
we have a link to the actual project,
we get to see all of our
program definitions here and then
we see the files
that we can download from this program.
However, I know that FEMC does not fuzz spot,
so I'm just gonna search for FEMC for a minute.
And now since FEMC allows you
to see your original data,
when we click on that we get
this original plot observation data.
So this is a JSON of everything that was in the
original data for this plot observation.
So I did think it was strange
that we have a field called
year that gives an entire date,
not just a year,
but I went back and looked and that is how it is.
So this is actually correct.
So now if we move on to trees,
we can see that there are quite
a few more filters in the
trees that we can filter by.
And you can also see
that our search parameters change.
So now we're searching for tree states,
Vermont and tree year min instead of just the plots.
So let's look at some of these filters.
We can do species,
we can do tree status, DVH height, crown class,
and then presence, absence of forest health indicator.
So I'm going to search for some species,
let's search for spruce.
So now this is now filtered
to a bunch of different spruce
and I'm gonna select them all
and search for them all.
So now in our search parameters up here,
you can see that we have a list
of names and TSMs that we're
searching in our database
so you know exactly what's being searched.
And this is similar where
you'll get the original tree
observation data here.
So now let's look at saplings.
You can see our search parameters changed.
So we're searching saplings
for those species states in here.
And then the same thing with seedlings.
So let's download our results.
I'm gonna download an entire package
so we can see that that looks like.
Okay, I'm gonna download our search results
and I'll take a look at what we got here.
Okay, so in our search results dated zip,
you can see we have three CSV files, programs,
plots and seedlings.
Now if we were on the sapling
or tree tab when we downloaded,
we would have a tab sapling or tree CSV.
If we were on the program or plots tab
and we did not have a
tree sapling or seedling filter set,
then we would get all three tree sapling
and seedling CSVs.
But for now, let's take a look
at what we have in our CSVs.
So this is the program.csv.
It just has our one program
that we had returned in our
search results, the FBMC forest health monitoring.
Let's look at plots.
So this is all of our FHM plots
that came back in our search results.
And you can see we have
this doc field, raise (indistinct)
with all of the original data.
So this is the original
data observation that we uploaded.
And then this row is actually
our standardized data.
So these (indistinct) longs,
if they had been in a different
coordinate system or if there
was a DBH that had been calculated in inches
and we store in centimeters,
they would be converted.
And then we can look at the seedling file.
Same thing, we have our dock field
with our original data and then we
have our row with our standardized data.
So let's take a look at this folder.
So each of the programs returned in our search,
will have a folder
that has packages, scripts,
and then there would be another
folder here for files if we
have ancillary files that we uploaded.
So the scripts are those same two scripts,
the pre-processing script
and then the included functions
with allfuncs.
And then packages,
we get that snapshot data package
of what was uploaded.
So in the pre-processor folder
we get those scripts
once again in the program data folder though,
we get these data sets.
So these are the original files
that we chose to upload from our program.
Metadata gives us a metadata
EML with more information, oh,
I guess that's not gonna show up on our video,
but it's a metadata EML,
take my word for it
and you'll be able to
get a description of what metadata is being downloaded.
And then in our raw files,
these are our own standardized data files.
So if you took those files
in the data set folder
and you ran those scripts that we included on these,
you would come out with our raw files.
Okay, so back to the website for one last thing,
this inventory programs page,
you can get to all of your program level information.
So if I just wanted
to download data from FEMC Forest Health
Monitoring and that's it,
I could come here and there's
a link right here to download
standardized program data and files
and get all of my program definitions here.
I can get a history of the packages
that were downloaded and the notes
and then I can also get my files,
packages and scripts to download separately.
And then this little data summary
section is just an example
of what can be done with our data.
Sorin put together an R shiny app
and you can make visualizations like plot maps
or tree year sample, year histograms,
things like that.
I'm not gonna go into too much detail,
but you can look through this when the site is live.
So thank you all for your interest in NEFIN.
(audience applauding)