Lecturer: Hi, and welcome to module one
of Advanced GIS, module one will be delivered in two parts.
In the first part, I'll discuss metadata, what it is, how
to write it, and where
to find specific information describing the various
aspects of a dataset.
The second part,
we'll explore data creation modification procedures.
In the introduction to GIS course, we created, calculated
and updated attribute information modifying the structure
and contents of the attribute table.
In this module, you'll learn to create new vector datasets
and modify their contents.
Let's get started with a quick refresher on metadata.
Metadata is data documentation or data about data.
Ideally, metadata describes the data according
to an established standard.
This makes it easier for analysts outside your agency
or organization to use the metadata both
to locate data resources
and ensure it is used appropriately in an analysis.
Following an established standard
and identifying the standard in the metadata itself
alerts the astute data sleuth to the anticipated content
and approach to documenting the data.
State, local and federal agencies are required
to provide metadata for the geospatial data they produce.
They're not all equally adept at doing so.
The closer to home you get, the more likely you're going
to encounter metadata challenges.
Large agencies and organizations are likely to have a team
of geospatial technicians while a local government
or NGO is more likely to have a small handful
of employees each wearing multiple hats.
Why do we have metadata?
Metadatasets the standard for the way
that data is developed.
For example, when establishing an attribute schema
for a dataset, you might specify a domain range
to limit the values that can be entered
for the attribute.
Suppose you want to include
an attribute representing test scores in a dataset.
There are no extra credit questions on the exam.
This means that scores can only range from 0 to 100.
Specifying this constraint in the domain range eliminates
the possibility of entering a score that is not possible
to achieve either less than 0 or greater than 100.
Metadata can also facilitate data search
and retrieval via keywords
and tags associated with the dataset.
These could represent DatasetThemes
like LAMB cover developing entity like the state of Vermont
or the USGS or even specific attribution within a dataset.
Lastly, metadata guides the appropriate use of data
by providing an abstract describing the dataset
in general terms, the specific purpose
for which the data was developed
and any known use constraints
that may render the data unsuitable
for a particular analysis.
Suppose you want to perform an analysis
of Allstate Parks in Vermont.
The VCGI server is down for maintenance,
but you need to complete your analysis today.
The only dataset you can find related to state parks is
by a guy at UVM named Brian.
His data does not represent all the parks
in the state, only the ones he's visited.
Would this data suffice?
Likely not, and it's better to learn that
before investing the time, download the data
and run your analysis.
This brings me to one of the real takeaways of metadata.
Create metadata so the data developers don't have
to communicate individually
with respective data users regarding dataset details
and so that data users can be certain
that the data they located is suitable
for their intended purpose.
The better your metadata,
the less likely you'll be responding to help requests
to enable others to use the data you produced.
In the US, there's an entity known
as the Federal Geographic Data Committee.
They developed the content standard
for geospatial metadata back in 1998 and have adjusted
and upgraded this standard over time.
The primary purpose of the metadata standard is
to provide a common set of terminology
and definitions for the documentation of geospatial data
or really any data.
All federal agencies are required to use this standard.
Most state agencies also meet the standards as well,
although sometimes when dealing with historic data,
it's a little hit or miss.
I tend to think of metadata in this way.
The closer you are to home, meaning local government
or NGO, the more likely you are
to encounter additional challenges with metadata
or perhaps no metadata at all.
In that case, it's best to reach out
to the data developer to get questions answered
before you use that data for any analysis.
Generally speaking, metadata can be grouped
into the following six categories.
Let's take a look at the details of each.
The data identification section
includes all elements you might expect.
General information like a dataset description, the agency
or organization that created or curated the data, the name
of the person within the organization
that performed the data development or analysis procedure,
and even the name of the contact person within the
organization responsible for data distribution.
There's also information about access
and use constraints for a dataset.
This is particularly relevant if you're dealing
with some information that might be classified
or otherwise feature restricted access.
I don't have a security clearance, although
that's probably what I'd still tell you even if I did
and don't often work with sensitive data,
but early enough in my career that I was a summer intern
for the South Carolina Coastal Management Agency,
I digitized point locations
of civil war artifacts along the South Carolina coast
from existing paper maps, I was only able to access the data
because it was an agency priority for evaluating zoning
and development permit applications.
Use of the data was limited to senior personnel
and even I couldn't access it after the data was developed.
Lastly here, data currency describes the date of production
or last update for the dataset.
Unfortunately, this section does not provide insights
into Bitcoin mining.
The data quality section includes elements describing the
positional and attribute accuracy of the data,
important factors to communicate to the end user of the data
to ensure appropriate fitness for use.
You'll also find information about data completeness.
Has all the work been done
or is this a preliminary dataset that's been released
while the final data set undergoes review?
The lineage provides a history of the dataset identifying
where the original dataset was obtained,
an existing dataset from another organization
that you modified to meet your needs
or a raw dataset collection effort.
The processing steps section details
the data processing steps
that have been performed on the data itself.
Ideally, you could review the process steps
to reproduce the intermediate
and final datasets of an analysis.
In my opinion, the more detail here, the better.
Think of it this way, unless you work on a lot
of projects, it's
likely you'll remember the general details of a dataset.
Details you'd find in the overview section,
but trying to remember the specific roadmap
or geo processing workflow used to go from raw data
to final product may be beyond your mental capacity.
I can barely remember which ski runs I took yesterday.
The end result should also be easier for others
to interpret, which should minimize the need for you
to explain the data.
The spatial reference section provides a lot
of the same content that you would see
in the layer properties interface for a dataset.
This information can be found
in the horizontal coordinate system
and the geodetic model sections of the metadata.
For me, the most frequently accessed section
of metadata is the entity and attribute information.
This section provides information about the entity itself.
Is it a vector or raster dataset?
If it's a vector dataset, does it feature point line
or polygon geometry?
In theory, you should be able
to determine this yourself just by looking at the data,
but why waste the time dealing with a large dataset only
to find it's not what you need when
instead you can consult the documentation first.
Deciphering attributes is a whole different ballgame.
The attribute information includes a definition
of an attribute describing what the attribute represents,
the defining authority and any value restrictions.
Look at the examples on the right side of the slide.
The upper image represents the data definition
for block groups in the 2010 census.
The US Census Bureau is the defining authority
and the value of zero is assigned
when a block group is located on water,
think Great Lake states or houseboats
in the Puget Sound or one through nine,
otherwise, depending on the number
of block groups within a census tract.
The next level of the census geography hierarchy,
the lower image reveals the
for the total population attribute.
Note two things here.
There are no value constraints and no defining authority.
You could specify that the values must be numeric
with a minimum value of zero and no maximum value.
You might also choose to specify the US Census Bureau
as the defining authority,
but that should be apparent based
on the data resource itself in this case.
This information is also what allows the user
to translate from an attribute name represented
as an alphanumeric code to a specific concept,
thereby facilitating the interpretation of the data values.
The data distribution section provides contact
information for the person in charge of distributing data
for the agency that's producing it.
This is a little bit less crucial now
that most data can either be downloaded directly
from the web or linked to via a map server.
The need to contact someone directly is likely higher
for accessing historical archives.
Some of this data may never be digitized
or perhaps the process
of digitizing these historic records is slowed
by budget realities and time constraints.
Alternatively, you may find certain data distributed
in an ancient format like a CD ROM.
The data distribution section should provide you
with all you need to know about ordering data
and any associated fees.
Lastly, there's the meta-metadata
or the data about the metadata.
When was the metadata created or updated?
Who performed the procedure and what standard was used?
I mentioned the FGDC earlier
and their role in setting the standard for the US.
There are also other country standards
and an international standard as well.
Alerting the user to the standard under which the metadata
was developed makes it easier to perform search operations
or use an interpreter
to translate from one standard to another.
Let's look at a couple of examples of metadata.
We'll look at metadata for data that's being distributed
by the Vermont Center for Geographic Information.
Note that VCGI is both a data producer and a data provider.
For those of you that completed the introduction course,
we visited the Vermont Open Geo data portal to access data
and documentation in previous lab assignments.
Hopefully, this interface looks familiar to you.
If you have not yet visited VCGI's website,
you'll get your first chance
in the upcoming lab one assignment.
Let's first look in more detail
at the census dataset that we saw snippets of earlier.
Notice there are two different links
to metadata on this page.
Let's start with the one on the right hand side.
Clicking that link takes you to a new page featuring an
ISO 19139 metadata standard.
This version provides an excellent overview of the dataset
as well as keywords to enable data search and retrieval.
However, if you're interested in greater level
of detail about specific aspects of the data,
like geo processing steps that have been performed
or how to interpret attributes or their unit of measure,
you should click the link in the top left corner
of the overview tab.
This links to a metadata document that complies
with the FGDC standard
and is organized by the six categories I just outlined.
If you look at the data quality section,
you can find information about what,
if any accuracy assessment has been performed
to ensure logical consistency
and geographic accuracy in the dataset.
The top image describes the US Census Bureau's efforts
to ensure an accurate product for the 2010 census.
This is also where you look
for details regarding the lineage
and any data processing that has been undertaken.
Step-by-step process descriptions
for each geo processing operation
that was performed on the dataset gives you a complete
history of the evolution of that dataset.
In this example, we see the details of two process steps.
The first is extracting the Vermont data
from the national dataset,
and the second describes joining the attributes
from the summary file one, the 100% count of the population
to the various geographies produced in the step above.
My opinion on this is to be as detailed as possible
and only list a single processing step per entry.
I'm not sure I could recreate this dataset based
on what I've read here.
If you look at the entity
and attribute information section, you can obtain a lot
of detail about the attributes themselves.
Let's first look at the state attribute.
The definition states
that this attribute represents
the 2010 census state federal information
processing standards or FIPS code number indicating state
that value for Vermont is 50.
This means that the only value you'll find
for the state attribute in this dataset is 50.
If you were working with the national dataset,
you would expect to find the FIPS numbers
for each state in the dataset.
Looking towards the bottom here, we can see an attribute
with labeled P0030002.
This attribute definition states that this represents
that portion of the total population
that identifies as white alone.
If we look at the attribute table for the dataset,
we see code P0030002 and know that in 2010, according
to the US census, there were 596,292 people in the state
of Vermont that identified as white alone.
Okay, one last example.
This time, let's take a look at data representing electric
vehicle charging stations throughout Vermont.
There's the expected metadata link on the right side
of the page, but that won't provide us
with the answers we need
to interpret the attribute information.
Instead, click the external link
to the National Renewable Energy Lab provided
at the end of the dataset description.
That link brings you to a page that looks interesting,
but maybe not as you expected.
You're right, this is not the metadata standard we've seen
in previous examples.
Instead, this is a data dictionary that specifies how
to request and receive data from their server, what's known
as a get request, something we won't worry about for now,
since VCGI takes care of that for us.
It also specifies what data the user receives.
Let's look at the fuel station records fields since
that's the attribution attached to each
of the charging station sites.
This looks a bit more familiar.
There's a data type of string
for the fuel type code attribute,
and the domain of values has been specified with a two
to four digit code and an accompanying definition.
For example, BD represents biodiesel B20 and above,
so that's pretty much it for metadata.
It's not real fun to write and can be cumbersome to read.
That said, I hope you understand the critical role it plays
in both data development and use.
There's lots of interesting data available,
and while you could wait for it to trend on Twitter
or be featured in an interpretive dance
on TikTok, using search tools to discover data
and its associated metadata to understand it
on your own is probably a better path to geoprocessing fame.