---
title: "User Guide 1: The data and how to use them"
subtitle: "Package 'photobiologyFilters' `r packageVersion('photobiologyFilters')` "
author: "Pedro J. Aphalo"
date: "`r packageDate('photobiologyFilters')`"
output: 
  rmarkdown::html_vignette:
    toc: yes
vignette: >
  %\VignetteIndexEntry{User Guide 1: The data and how to use them}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, echo=FALSE}
knitr::opts_chunk$set(fig.width=8, fig.height=4)
```

## Introduction

This package, is a data only package, part of a suite, which has package 
'photobiology' at its core. Please visit (https://www.r4photobiology.info/) 
for additional information. For details on plotting spectra, please consult the 
documentation for package 'ggspectra', and for information on the calculation 
of summaries and maths operations between spectra, please, consult the 
documentation for package 'photobiology'.

```{r}
library(photobiology)
library(photobiologyWavebands)
library(photobiologyFilters)
library(ggspectra)
theme_set(theme_bw())
```

In this brief User Guide we describe how to access individual spectra or
subsets of spectra.

## The objects containing the data

The data are stored in objects of classes defined in package 'photobiology'
grouped into four collections of spectra. Functions and operators in the
packages that are part of the R for photobiology suite recognize these classes,
which allows automation of some computations and validation of the data values
returned by operations on the data.

Individual spectra are stored in clases are derived from tibbles, which in turn
is derived from data frames with additional metadata stored as attributes of the
objects. Being the objects derived from `data.frame` means that the data can be
used with native R functions and operators and with other R packages. As package
'photobiologyInOut' provides functions that facilitate the exchange of spectral
data with R packages 'colorSpec', 'hyperSpec' and 'pavo', the data in these
package can also be easily used with them.

The individual spectra are stored in collections as objects of classes derived
from R's list class.

`filters.mspct` contains spectral transmittance data for 
`r length(filters.mspct)` optical filters and other similar transparent and
semitransparent sheets and films.

```{r}
class(filters.mspct)
```

```{r}
class(filters.mspct[[1]])
```

`filters_UVIVIF.mspct` contains UV-A1-induced spectral fluorescence data for 
`r length(filters_UVIVIF.mspct)` optical filters.

```{r}
class(filters_UVIVIF.mspct)
```

```{r}
class(filters_UVIVIF.mspct[[1]])
```

`metals.mspct` contains spectral reflectance data for `r length(metals.mspct)`
metals.

```{r}
class(metals.mspct)
```

```{r}
class(metals.mspct[[1]])
```
`materials.mspct` contains spectral reflectance data for 
`r length(metals.mspct)` materials and surfaces, including different types
vegetation.

```{r}
class(materials.mspct)
```

```{r}
class(materials.mspct[[1]])
```
`refractive_index.mspct` contains spectral refractive index data for 
`r length(metals.mspct)` materials. These values allow the computation of 
reflectance at different angles of incidence.

```{r}
class(refractive_index.mspct)
```

```{r}
class(refractive_index.mspct[[1]])
```

Spectral reflectance and transmittance values are stored as fractions of one,
and wavelengths expressed in nanometres (nm).

```{r}
colnames(filters.mspct[[1]])
```
```{r}
colnames(metals.mspct[[1]])
```
```{r}
colnames(materials.mspct[[1]])
```
Refractive index values are:

```{r}
colnames(refractive_index.mspct[[1]])
```
## The transmittance and reflectance data

Spectral data are included both for optical filters, sold as such, and for
materials that either on purpose or by accident may be interposed in the "light"
path in photobiological experiments, including glass panes, plastic sheets and
films. It must be kept in mind that, 1) absorptance depends on the thickness of
a filter in addition to the properties of the material it is made off, and that
2) reflectance depends on the angle of incidence of the light beam. Spectral
transmittance can be expressed either as internal or total, depending on how
the effect reflections at surfaces are accounted for. The conversion between
total and internal transmittance requires reflectance to be known.

We provide reflectance estimates for several of the filters in this collection,
but we do not have such data available for all of them. For filters made of
ionic glass and coloured plastics, reflection is very weakly selective, and amounts to
about 9 to 10\% of radiation incident at an angle close to perpendicular to the
surface (when both interfaces with air are considered). Anti-reflection coating
(ARC) reduces reflections, and multi-coating (MC) even further. Such coatings
are not equally effective at all wavelengths and, consequently, their use can
modify the spectral properties of a filter. In contrast to absorptive filters
described above, dichroic or interference filters reflect the "rejected"
radiation. It is also possible to produce filters that have an absorbing glass
as substrate and a dichroic coating deposited onto one or both of its surfaces.
In the metadata filters are tagged as belonging to one of three
`attenuation.mode` types: `absorption`, `reflection` and `mixed`. Mixed includes
those filters where wavelength-selective attenuation is brought about both by
absorption and reflection, including scattering media and dichroic filters
deposited on absorptive ionic glass.

The spectral transmittance or absorbance data have been acquired with an
assortment of different instruments. Some data are measured by the authors with
spectrophotometers, others have been provided by filter manufacturers. Whenever
these metadata were available an approximate reflectance for normal incidence angle,
material thickness, and the mode of attenuation have been stored as attributes. In
the case of filters sold assembled in frames for use in photography and
industrial automation the thickness is rarely disclosed by suppliers, but in 
some cases we have been able to measure it.

For absortive filters with no coatings reflectance was
computed from the spectral refraction index if available, or copied from
manufacturer's specifications or for some standard materials based on published
sources. In the case of filters with anti-reflection coatings, in most cases the
estimated reflectance is based on the maximum total transmittance. Most of the data
are approximations given as a single value for the whole spectrum following
Schott's practice. (We aim to add in the future actual measured and/or 
computed spectral reflectance for some filters or clear glass to serve as 
examples).

The difference in resolution and slit function among instruments can give, for
the same filter, measured spectra with "apparent" peaks and valleys of slightly
different width, and slopes of slightly different steepness. This is an
inevitable artifact of spectral measurements, which has little effect on the
spectra from absorptive filters (with gradual slopes and bell-shaped peaks) but
can be more significant for dichroic (= interference) filters with very steep
cut-on and cut-off slopes.

Another important consideration is that some materials scatter transmitted and
reflected light, and consequently such materials can be accurately measured 
only with an integrating sphere. Data included here have been in many cases
measured without an integrating sphere; i.e., only by assessing the direct beam.
For non-scattering materials this causes only minor errors. For scattering
materials like polythene films, the data included have being measured with 
an integrating sphere.

Glass-filter properties vary to some extent among melt batches. Variation can
also be expected among batches of plastic filters. Furthermore, filters age
upon exposure to light and UV radiation, and in some cases even upon
exposure to air. Aging is not limited to plastic filters and can also affect
optical glass. Manufacturers occasionally may update the specifications
for filters while maintaining the product code.

All the spectral transmittance data in this package are stored in a single R object, a
collection of spectra of class `filter_mspct` with members of class
`filter_spct`. Individual or subsets of spectra can be retrieved by _name_. The
package includes also several `character` vectors of _names_, each one
containing names for filters of a given _color_, a given _type_ or from a given
_manufacturer_. The names of all these vectors are available in vector
`all_filter_selectors`. The _names_ used are in most cases the codes used by the
manufacturers for the given type with a code for the manufacturer of supplier
prepended. Any dashes in these codes have been replaced by underscores to 
convert them into syntactically valid R names.

```{r}
band_pass_filters
```

```{r}
schott_filters
```
Help pages give additional information.

```{r, eval=FALSE}
help(schott_filters)
```

In addition to these data for filters, the package contains spectral reflectance data for both metal surfaces and various man-made and natural surfaces. All spectral reflectance data in the package are for normally incident radiation and _total_ values, that is including both especular reflection and scattering. 

These collections are small, so we list the names of all members.

```{r}
names(metals.mspct)
```

```{r}
names(materials.mspct)
```
```{r}
names(refractive_index.mspct)
```

The collection of filter spectra is large, so we list only the first 20 names.

```{r}
head(names(filters.mspct), 20)
```

To obtain metadata in a tabular form, we can use function `spct_metadata()`. This
function is available in package 'photobiology' version 0.10.5 or later. (This
example is run only if a supported version is loaded.) The limit the number of
columns we select two of them by name.

```{r}
if (utils::packageVersion('photobiology') > "0.10.4.9001") {
  print(colnames(spct_metadata(filters.mspct[1:5])))
  spct_metadata(filters.mspct[1:5])[ , c("spct.idx", "what.measured")]
}
```

## Accessing individual spectra

The `filter_spct` member objects in `filters.mspct` and the `reflector_spct` objects contained in `metals.mspct`  and `materials.mspct` can
be accessed through their names or through a numeric index. As the numeric
indexes are likely to change with updates to the package, their use is
discouraged. Names should be used instead. The names are listed in the
documentation and be listed with method `names()`. All examples below use 
`filters.mspct` but operations on the other collections work in the same way.

We can use a character string with the name as an index to extract an individual spectrum.

```{r}
filters.mspct$Schott_UG11
```

```{r}
filters.mspct[["Schott_UG11"]]
```

Be aware that according to R's rules, using single square brackets will return
a `filter_mspct` object possibly of length one. This statement is not equivalent
to the one in the chunk immediately above.

```{r}
filters.mspct["Schott_UG11"]
```

Of course, with single square brackets it is possible to use a vector of member
names.

## Accessing subsets of spectra

We can subset the `filter_mspct` object by indexing with vectors of character
strings. The package provides several predefined ones, and users can easily
define their own, either as constants or through computation. Here we use
a vector defined by the package.

```{r}
filters.mspct[petri_dishes]
```

The vector `all_filter_selectors` contains the names of the different vectors of
names of members of `filters.mspct`.

```{r}
all_filter_selectors
```

In addition to the predefined vectors it is possible to compute numeric indexing vectors
using pattern matching with `grep()`. In this example we extract the member spectra
with names containing the string "UG".

```{r}
filters.mspct[grep("UG", names(filters.mspct))]
```

To generate the subset of names matching a pattern, we can also use `grep()`.

```{r}
grep("UG", names(filters.mspct), value = TRUE)
```

## Querying metadata

The spectra are saved in objects of class `"filter_spct"`, defined in package
'photobiology'. Specializations of several methods including `print()` and
`summary()` include a summary of the metadata in the header of the printout.
Two different definitions of transmittance exist, differing in how reflection
is treated: for "internal" transmittance, the divisor is the radiation entering
the material, and for "total" transmittance the incident radiation. For some
materials reflectance (`Rfr`) does not vary much with wavelength, and some
suppliers provide a constant value for it.

```{r}
filters.mspct$Schott_UG11
```

Metadata can also be queried with other methods. Please, see the documentation
for package 'photobiology' for the details.

```{r}
filter_properties(filters.mspct$Schott_UG11)
```

```{r}
what_measured(filters.mspct$Schott_UG11)
```

```{r}
how_measured(filters.mspct$Schott_UG11)
```

```{r}
is_normalized(filters.mspct$Schott_UG11)
```

```{r}
cat(comment(filters.mspct$Schott_UG11), "\n")
```

Of the different metadata items, the type of data is of great importance.
Transmittance (and absorptance) can be expressed in two different ways: 
1) taking incident radiation as reference, or 2) taking radiation entering
the material as reference (i.e. discounting reflection). 1) is referred as
_total_ transmittance while 2) is referred as _internal_ transmittance.
Attribute `Tfr.type" is used to store this information.

```{r}
getTfrType(filters.mspct$Schott_UG11)
```
## Conversions

When metatdata are available and the mode of attenuation is `"absorption"` it is
possible to compute the expected transmittance for a different thickness of the
material. In the example we compute transmittance for a thickness of 4 mm.

```{r}
convertThickness(filters.mspct$Schott_UG11, thickness = 4e-3)
```
We can also convert `"internal"` transmittance into `"total"` transmittance.

```{r}
convertTfrType(filters.mspct$Schott_UG11, Tfr.type = "total")
```
Conversion between transmittance, absorptance and absorbance is also possible.
In package 'photobiology' fractions of one are used to express transmittance,
reflectance and absorptance.

```{r}
any2Afr(filters.mspct$Schott_UG11, action = "replace")
```

For expressing absorbance, base-10 logarithms are used. In some fields, natural
logarithms are used instead. Expressing `A` on this base is not supported by
this package and any input absorbances must be first converted to log10 based
`A`.

```{r}
any2A(filters.mspct$Schott_UG11, action = "replace")
```


## Plotting

Spectra can be plotted in the same ways as other data stored in
data frames, using base R graphics, package 'lattice' or 'ggplot2'. However, another
package in our suite, 'ggspectra', built as an extension to 'ggplot2' makes 
plotting spectra extremely easy.

`autoplot()` methods use the metadata in the objects to set labels and decorations,
as well as automatically setting the mapping of the _x_ and _y_ aesthetics.

```{r}
autoplot(filters.mspct$MIDOPT_LP500)
```
```{r}
autoplot(filters.mspct$MIDOPT_LP500,
         annotations = list(c("-", "peaks"), c("+", "wls")))
```

```{r}
autoplot(filters.mspct$MIDOPT_TB550_660_850, 
         annotations = c("+", "title:none:none:what", "wls"), 
         w.band = VIS_bands(),
         range = c(500, 910),
         span = 11)
```

To graphically compare filters, we can pass a collection of spectral objects,
such as subset of `filters.mspct`.

```{r}
autoplot(filters.mspct[c("Schott_UG1", "Schott_UG11")], 
         range = c(200, 900),
         annotations = c("+", "boundaries"), 
         span = 11)
```

To graphichaly compare filter thicknesses we can pass a collection of spectral objects.

```{r}
thin_and_thick.mspct <- 
  filter_mspct(list("1 mm" = filters.mspct$Schott_UG11,
                    "3 mm" = convertThickness(filters.mspct$Schott_UG11,
                                              thickness = 3e-3)))
autoplot(thin_and_thick.mspct,
         range = c(200, 900),
         annotations = c("+", "boundaries"),
         span = 101)
```

To graphically assess filter stacks with air gaps.

```{r}
stack.spct <- filters.mspct$Haida_Clear_Night_NanoPro_1.6mm_52mm * filters.mspct$Firecrest_UVIR_Cut_0.96mm_52mm
                    
autoplot(stack.spct,
         range = c(NA, 1400),
         w.band = c(UV_bands(), IR_bands("CIE")),
         annotations = list(c("+", "boundaries"), c("-", "peaks")),
         span = 21) +
  geom_line(data = filters.mspct$Haida_Clear_Night_NanoPro_1.6mm_52mm, colour = "purple") +
  geom_vline(xintercept = c(589, 589.6), linetype = "dotted") # Na emission lines
```

Package 'ggspectra' also defines specializations of method `ggplot()` for 
spectra that automatically maps the $x$ and $y$ _aesthetics_.

```{r}
ggplot(filters.mspct$Firecrest_UVIR_Cut_0.96mm_52mm) +
  geom_line() +
  scale_x_wl_continuous() +
  scale_y_Tfr_total_continuous()
```

```{r}
autoplot(metals.mspct$gold, range = c(NA, 800))
```
```{r}
autoplot(materials.mspct$grass, range = c(NA, 800))
```

## Calculating summaries

Here we use waveband defintions from package 'photobiologyWavebands' and 
summary functions from package 'photobiology'.

```{r}
transmittance(filters.mspct$Firecrest_UVIR_Cut_0.96mm_52mm, UVA())
```
```{r}
absorbance(filters.mspct$Firecrest_UVIR_Cut_0.96mm_52mm, list(UVA(), NIR()))
```

```{r}
transmittance(filters.mspct[grep("UG", names(filters.mspct))], 
              list(UVB(), UVA()))
```

```{r}
reflectance(metals.mspct, w.band = VIS_bands())
```

## Using the data in other contexts

As `filter_mspct` is a class derived from `list`, and `filter_spct` is derived from
`tibble::tible` which is a mostly compatible reimplementation of `data.frame` the
data can be used very easily with any R function.

```{r}
head(as.data.frame(filters.mspct$Schott_UG11))
```

Of course `attach` and `with` also work as expected.
```{r}
attach(filters.mspct)
transmittance(Schott_UG11, UVA())
detach(filters.mspct)
```

```{r}
attach(filters.mspct)
with(Schott_UG11, range(w.length))
detach(filters.mspct)
```

```{r}
with(filters.mspct, transmittance(Schott_UG11, UVA()))
```