Declare missing values on one or more variables

jdeclare_missing() declares missing values on one or more variables. Declared missing values are specific data values – typically negative codes such as -99, Stata-style tagged markers such as .a, or on a text variable words such as "UNKNOWN" – that indicate why a value is missing (refused, don't know, not applicable, etc.) rather than simply that it is missing. Once declared, those cells are automatically excluded from analyses but remain visible in the data for diagnostic purposes (see jfreq()). The haven package and its documentation call these user-defined missing values.

The function operates in declarative mode: what a call mentions, it replaces; what it omits survives. Supplying codes replaces the column's discrete-code set; supplying range replaces the column's missing-value range; an existing range survives a codes-only call, and existing discrete codes survive a range-only call. A second call therefore replaces, not augments, whichever parts it names – matching SPSS's MISSING VALUES and Stata's mvdecode semantics, neither of which has an additive form. When prior declarations are dropped, a note lists them so the destructive aspect of the replacement is not silent.

Variables are given either as unquoted names (jdeclare_missing(df, MathScore, EnglishScore, codes = c(-99))) or as a character vector via vars = (the programmatic form, e.g. vars = offence_cols). A multi-variable call applies the same declaration to every named column, all-or-nothing: if any column fails a check, no column is changed. There is deliberately no whole-data-frame default – declaring a code frame-wide would flag it missing even on columns where that value is legitimate data. To declare on every column, pass vars = names(data) explicitly.

Usage

jdeclare_missing(
  data,
  ...,
  codes = NULL,
  labels = NULL,
  range = NULL,
  vars = NULL,
  convention = NULL,
  missing.notice = TRUE,
  modify = FALSE
)

Arguments

data

A data frame containing the variable(s).

...

The variable(s) to declare missing values on, as unquoted names (e.g. Income, or MathScore, EnglishScore). Use either ... or vars, not both. All arguments after the variables must be named.

codes

The values to declare as missing: numbers, or on a text variable the values themselves in quotes, such as codes = "UNKNOWN" (see the Text variables section). Accepts two forms:

Option A (separate codes and labels)

Unnamed numeric vector; labels supplied via the labels argument. E.g. codes = c(-99, -98), labels = "-99=Refused; -98=Don't know".

Option C (haven-style named vector)

Named numeric vector; names are the labels. E.g. codes = c(Refused = -99, `Don't know` = -98).

On a column that already carries Stata-style or SAS-style missing values, codes may name the markers directly, as quoted strings, e.g. codes = c(Refused = ".a") – the string ".a" names the .a marker (or .A on a SAS-style column: the letter case you type is accepted either way and canonicalized to the column's convention). A quoted marker is refused on a column with no tagged missing values to label (see the Missing-Values Convention section).

A quoted marker may name one that no case currently carries – useful for FORWARD-DECLARING a label before recoding values into it. The label attaches, nothing in the data changes, and the notification says the marker is not present in the data. Such a label survives a save to Stata format (.dta) and the load back, and jconvert(to = "spss") declares its code like any other marker's.

A quoted marker given with no label is a no-op on an already-tagged column: the cells are already missing, so the only act available is naming a marker, and the call reports that it changed nothing. This differs from the numeric form, where a bare code IS the declaration.

labels

Optional. A quoted string in the form "value=label; value=label" pairing labels with codes (Option A only). Must be NULL when codes is named (Option C). It may also stand alone, with no codes: the values named in the string are then the codes declared, so labels = "-99=Refused; -98=Don't know" declares and labels both codes in one argument (and labels = ".a=Refused" names a marker on a tagged column). A label may be written in quotes, with either mark, right after its equals sign, and a semicolon inside the quotes is then part of the label. When a range is in effect (supplied in this call, or already on the column), entries may also name values inside the range: those attach as value labels on the in-range values without becoming discrete declared codes (see the Missing-value ranges section). On a text variable the values are text, as in labels = "REF=Refused; UNKNOWN=Unknown": a value holding an equals sign or a semicolon goes in quotes, and blank names the blank cells, as in the maps of jencode.

range

Optional. A length-2 numeric vector declaring a missing-value RANGE (band), e.g. range = c(-99, -51) – every value from the first bound through the second is treated as missing. The SPSS parallel is MISSING VALUES X (-99 THRU -51). Bounds may be non-integer, and one bound may be infinite (c(-Inf, -51) is SPSS's LO THRU -51). A missing-value range can exist only under SPSS convention: combining range with the Stata or SAS convention – as a per-call argument or as the joptions() setting – or with quoted markers is refused, and the refusal teaches the two-step route (declare the range under SPSS convention, then jconvert() the column) when the range covers few enough values to convert. When no convention is selected anywhere, a range declaration stops and asks for convention = "spss" on the call. SPSS allows at most ONE discrete code alongside a range, and the check applies to the column's composed result (what this call supplies plus what already survives on the column), so a declaration that a .sav file could not hold is refused here rather than at save time. A text variable takes no range.

vars

Optional. A character vector of variable names, the programmatic alternative to unquoted names in ... (e.g. vars = c("Age", "Income"), or vars = offence_cols where offence_cols holds the names). Use either vars or ..., not both.

convention

Optional. One of "spss", "stata", or "sas" (any capitalization is accepted); sets the convention for this call on columns that carry no missing-value declaration yet. A column that already carries the other convention's missing values is not overridden: the call stops and offers the two remedies (drop the argument to follow the column's form, or jconvert() the column first). When NULL (the default), the convention is resolved from the column's existing missing-value declaration (if any), then from the missing.convention setting in joptions; when neither supplies one, the call stops with a guided error asking you to choose – the package never infers a convention for a fresh declaration. A range requires SPSS convention (see range). The SAS convention behaves as the Stata convention with uppercase markers: markers are stored and labeled as .A-.Z. A quoted marker's letter case is accepted either way under both tagged conventions; the case written to the column follows the resolved convention. A text variable's missing values are SPSS-style whatever the setting, so "stata" and "sas" are refused for one.

missing.notice

Logical. When TRUE (the default), the function prints a notification summarizing what was declared, plus a reminder of how to keep the result. Under SPSS convention the notification lists the variable's resulting declaration, not only what the call named: a range or code the variable keeps from an earlier call is marked "already declared", a code with no value label is marked "no label", and a code or range that no case holds is marked "not present in the data" (the mark a mistyped code gets, such as -77 for -99). In a call on several variables that last mark is given only when none of the listed variables holds the value. Set FALSE to suppress.

modify

Logical. When TRUE, the declaration is written back onto the data frame named in the call (or onto the juse() default when the data argument is omitted), so no assignment is needed – the recommended workflow, since a declaration lost to a forgotten assignment silently poisons later statistics. Requires the data frame to be given as a plain name. When FALSE (the default), the caller's data frame is untouched; assign the returned data frame to keep the declaration.

Value

The data frame, with the specified variable(s) updated to carry the declared missing values, returned invisibly. With the default modify = FALSE, the caller's data frame is unchanged until the result is assigned back. With modify = TRUE, the change is also written back onto the caller's data frame, and the returned copy can be ignored.

Missing-Values Convention

Under SPSS convention, codes are declared as numeric values via the column's na_values attribute (haven's representation of SPSS-style missing values). The data cells themselves are unchanged; only the metadata that flags certain values as missing is added.

Under Stata or SAS convention with tagged missing-value input (quoted markers such as ".a"), the function attaches value labels to the column's existing tagged missing-value markers. This requires the column to already carry tagged missing values – either tagged cells, or markers previously declared through value labels (a marker may be labeled before any cases carry it, so a declaration made early in data collection is complete for later data). The letter case you type is accepted either way and canonicalized to the resolved convention: lowercase markers under Stata convention, uppercase under SAS. Quoted markers against a column with no tagged missing values are refused identically under every convention source: on a plain column the error points at jrecode() (which creates tagged cells from numeric codes); on a column carrying SPSS-style declarations it points at jconvert().

Note that on a plain numeric column with plain numeric codes and no convention argument, the resolved convention decides the outcome: under SPSS convention the numbers stay in the cells and are flagged missing; under Stata or SAS convention the matching cells are converted to markers and the numbers leave the data. This is the one place joptions(missing.convention = ...) changes what happens to data (see the examples).

Under Stata or SAS convention with numeric input, the function converts matching cells to tagged missing-value markers (SAS convention writes the same letters uppercase). The mapping is ordering-based: codes sorted by absolute value descending, more-negative-first as tie-breaker, then assigned .a, .b, .c, .d in that order (.A, .B, ... under SAS convention). The assignment proceeds independently of the missing.convention.codes setting in joptions (which only governs the reverse Stata-to-SPSS direction).

The letters are taken from the start of the alphabet whatever the variable already holds, so a code is refused when its letter is one the variable already carries, in its cells or as a labeled value: converting -99 to .a on a variable whose .a cells mean "Skipped" would put two kinds of missing data on one marker. The message gives two jrecode() calls that run: one recodes the codes to markers the variable does not use, which keeps the kinds distinct; the other recodes them to the markers they would have taken, for kinds that mean the same thing.

Missing-value ranges

A range declares a whole band of values missing at once – the form commercial statistical software uses when a study's missing-value codes share a band (e.g. every code from -99 through -51). SPSS accepts at most three discrete missing values, OR a range, OR a range plus one discrete value; jdeclare_missing() enforces the same rule on the composed result of each call, so a declaration is refused at the moment it becomes illegal rather than when jsave() later refuses the file. A range-only call is complete in itself (jdeclare_missing(df, X, range = c(-99, -51))).

Values inside the band may carry value labels: with a range in effect, labels entries that match no discrete code but fall inside the band attach as ordinary value labels on those values. They do not become discrete declared codes – the band already covers them – but analysis output that breaks out in-range values can then show their meanings. Because a range replaces the column's existing range and existing discrete codes survive a range-only call, a column already carrying two or more discrete codes cannot take a range in the same declaration; the call is refused with the surviving codes named. When the new range no longer covers all of the old one, a note names the range as it was and counts the cases that are no longer missing.

Text variables

On a text variable the codes are the values themselves, in quotes: jdeclare_missing(df, Marital, codes = "UNKNOWN"), or with labels, codes = c(Refused = "REF") or labels = "REF=Refused". They are declared SPSS-style – the form SPSS's MISSING VALUES gives a string variable, and the only form a text value has – under every missing.convention setting and with none selected, so no convention needs choosing first. Under a Stata or SAS setting a note says so; convention = "stata" or "sas" and a range are refused for text. At most three values may be declared, the SPSS limit on a string variable.

A value matches the cells that hold it exactly as stored: letter case and outer spaces count, as in SPSS. After declaring, a note names any values that differ from a declared one only in case or outer spaces ("Unknown" beside "UNKNOWN"), with their counts, and gives the call that declares them too – naming the values already declared, since a call replaces a variable's declaration – or, past three values, points to jencode. codes = "" declares every blank cell missing – empty, or holding only spaces or tabs – as SPSS declares a blank string missing; otherwise jstats counts blank cells as valid values and never assumes they are missing.

Numbers given for a text variable are refused with the call to run, the numbers in quotes. A call naming text and numeric variables together declares numbers on the numeric ones or words on the text ones, and refuses the others with the call to run. To keep the values in an SPSS file, each must be 8 characters or fewer: jsave() refuses a longer one, which the file would cut short.

Mixed conventions and file export

A single data frame may carry columns with SPSS-style and columns with Stata-style missing values. In-memory analysis and display tolerate the mix without issue (each column renders in its native form). The constraint shows up at file-export time: .sav cannot represent Stata-style missing values; .dta cannot represent SPSS-style na_values declarations; .xpt can represent neither form. jsave() pre-flights the DF against the destination format and errors with a pointer to jconvert() when the mix is incompatible. The post-declaration mismatch notice emitted at the bottom of this function's output exists to alert you early if a declaration ends up out of step with the rest of its DF.

See also

jrecode, jconvert, joptions, jstats

Examples

# A fresh declaration needs a chosen missing-value convention; the
# package never infers one. Choose SPSS convention for these examples:
joptions(missing.convention = "spss")
#> Options Settings
#> Missing-value convention: SPSS-style
#> SPSS-style missing value codes: -99, -98, -97
#> Run joptions() to see all settings.
#> 

# clinic$MoodRating arrives "dirty": -99/-98 sit in the data as
# ordinary numbers (the state after a CSV or Excel import), so summary
# statistics are poisoned until the codes are declared missing.
df <- clinic
jdesc(df, MoodRating)        # mean dragged far down by -99/-98
#> Descriptive Statistics
#> 
#> 70 Cases in the 1 Variable Pool
#> 
#> Variable    Total  Non_missing  Min  Max   Mean     SD
#> ----------  -----  -----------  ---  ---  ------  ------
#> MoodRating    70        70      -99   9   -4.943  31.477
#> 

# SPSS form: declare -99 and -98 as missing, with labels. modify = TRUE
# writes the declaration back onto df in one step -- the recommended
# workflow.
jdeclare_missing(df, MoodRating,
             codes  = c(-99, -98),
             labels = "-99=Refused; -98=Don't know",
             modify = TRUE)
#> Declared SPSS-style missing values on MoodRating in df:
#>   -99 ["Refused"]
#>   -98 ["Don't know"]
#> 
#> To keep it across sessions, save the data frame:
#>   jsave(df, "df.rds")
jdesc(df, MoodRating)        # codes now excluded as missing
#> Descriptive Statistics
#> 
#> 70 Cases in the 1 Variable Pool
#> 
#> Variable    Total  Non_missing  Min  Max   Mean    SD
#> ----------  -----  -----------  ---  ---  -----  -----
#> MoodRating    70        63       1    9   5.460  1.702
#> 

# Equivalent without modify: assign the returned data frame back
df2 <- clinic
df2 <- jdeclare_missing(df2, MoodRating,
                    codes  = c(-99, -98),
                    labels = "-99=Refused; -98=Don't know")
#> Declared SPSS-style missing values on MoodRating:
#>   -99 ["Refused"]
#>   -98 ["Don't know"]
#> 
#> This call changes df2 only if you assign the result:
#>   df2 <- jdeclare_missing(df2, MoodRating, ...)
#> 
#> To change df2 directly, rerun with modify = TRUE:
#>   jdeclare_missing(df2, MoodRating, ..., modify = TRUE)

# Equivalent using named codes (one step instead of codes + labels)
df3 <- jdeclare_missing(clinic, MoodRating,
                    codes = c("Refused" = -99, "Don't know" = -98))
#> Declared SPSS-style missing values on MoodRating:
#>   -99 ["Refused"]
#>   -98 ["Don't know"]
#> 
#> This call changes clinic only if you assign the result:
#>   clinic <- jdeclare_missing(clinic, MoodRating, ...)
#> 
#> To change clinic directly, rerun with modify = TRUE:
#>   jdeclare_missing(clinic, MoodRating, ..., modify = TRUE)

# The data frame may also be given by name, or set once with juse()
df3 <- jdeclare_missing(data = clinic, MoodRating, codes = c(-99, -98))
#> Declared SPSS-style missing values on MoodRating:
#>   -99 (no label)
#>   -98 (no label)
#> 
#> This call changes clinic only if you assign the result:
#>   clinic <- jdeclare_missing(clinic, MoodRating, ...)
#> 
#> To change clinic directly, rerun with modify = TRUE:
#>   jdeclare_missing(clinic, MoodRating, ..., modify = TRUE)
juse(clinic)
#> Default data frame set to: clinic
df3 <- jdeclare_missing(MoodRating, codes = c(-99, -98))
#> Declared SPSS-style missing values on MoodRating:
#>   -99 (no label)
#>   -98 (no label)
#> 
#> This call changes clinic only if you assign the result:
#>   clinic <- jdeclare_missing(clinic, MoodRating, ...)
#> 
#> To change clinic directly, rerun with modify = TRUE:
#>   jdeclare_missing(clinic, MoodRating, ..., modify = TRUE)
juse(NULL)
#> Default data frame cleared.

# A missing-value RANGE: every value from -99 through -51 is missing.
# SPSS parallel: MISSING VALUES MoodRating (-99 THRU -51).
df4 <- jdeclare_missing(clinic, MoodRating, range = c(-99, -51))
#> Declared SPSS-style missing values on MoodRating:
#>   range -99 to -51
#> 
#> This call changes clinic only if you assign the result:
#>   clinic <- jdeclare_missing(clinic, MoodRating, ...)
#> 
#> To change clinic directly, rerun with modify = TRUE:
#>   jdeclare_missing(clinic, MoodRating, ..., modify = TRUE)

# Range plus labeled values inside the band, on several variables at
# once. The same declaration lands on every named column,
# all-or-nothing.
if (FALSE) { # \dontrun{
jdeclare_missing(mydata, vars = c("Theft", "Assault", "Burglary"),
             range  = c(-99, -51),
             labels = "-99=Refused; -61=Not applicable",
             modify = TRUE)
} # }

# Stata-style: label Stata-style missing-value cells. The jrecode() call
# turns the literal codes into tagged cells; jdeclare_missing() labels them
# by naming the markers as quoted strings.
df5 <- clinic
df5$Mood2 <- jrecode(df5, MoodRating,
                     map = "-99=.a; -98=.b; else=copy",
                     convention = "stata")
#> Note: This call changes df5 only if you assign the result:
#>   df5$<name> <- jrecode(...)
#> To check the recode landed correctly, compare jfreq() on the original and the
#> new column.
jdeclare_missing(df5, Mood2,
             codes = c("Refused" = ".a", "Don't know" = ".b"),
             modify = TRUE)
#> Named Stata-style missing values on Mood2 in df5:
#>   .a is now "Refused"
#>   .b is now "Don't know"
#> 
#> To keep it across sessions, save the data frame:
#>   jsave(df5, "df5.rds")
#> 
#> Note: Mood2 uses Stata-style missing values, but your missing.convention
#> setting is "spss".
#> To convert the data frame, run:
#>   jconvert(df5, to = "spss", modify = TRUE)
#> To keep Stata-style instead, change the setting:
#>   joptions(missing.convention = "stata")
#> 
#> Note: Mood2 uses Stata-style missing values, but other columns in df5
#> are SPSS-style.
#> To align Mood2 with the rest, run:
#>   jconvert(df5, vars = "Mood2", to = "spss", modify = TRUE)

# A text variable: the values themselves, in quotes, declared SPSS-style
# under every setting. codes = "" declares the blank cells.
df8 <- data.frame(Marital = c("Married", "UNKNOWN", "Single", "",
                              "UNKNOWN", "Married"))
df8 <- jdeclare_missing(df8, Marital, codes = c(Unknown = "UNKNOWN"))
#> Declared SPSS-style missing values on Marital:
#>   UNKNOWN ["Unknown"]
#> 
#> This call changes df8 only if you assign the result:
#>   df8 <- jdeclare_missing(df8, Marital, ...)
#> 
#> To change df8 directly, rerun with modify = TRUE:
#>   jdeclare_missing(df8, Marital, ..., modify = TRUE)
df8 <- jdeclare_missing(df8, Marital, codes = c("UNKNOWN", ""))
#> Declared SPSS-style missing values on Marital:
#>   <blank> (no label)
#>   UNKNOWN ["Unknown"]
#> 
#> This call changes df8 only if you assign the result:
#>   df8 <- jdeclare_missing(df8, Marital, ...)
#> 
#> To change df8 directly, rerun with modify = TRUE:
#>   jdeclare_missing(df8, Marital, ..., modify = TRUE)
jfreq(df8, Marital)
#> Frequencies
#> 
#> 6 Cases in the 1 Variable Pool
#> 
#> Marital
#> 
#>                      Freq  Total %  Valid %  Cum. %
#> -------------------  ----  -------  -------  ------
#> Valid
#> Married                2     33.33    66.67   66.67
#> Single                 1     16.67    33.33  100.00
#> Total valid            3     50.00   100.00
#> 
#> Missing
#> <blank> (no label)     1     16.67       --      --
#> UNKNOWN ["Unknown"]    2     33.33       --      --
#> Total missing          3     50.00
#> 
#> Total                  6    100.00
#> 

if (FALSE) { # \dontrun{
# The same neutral call -- plain numeric codes, no convention argument,
# plain column -- forks on joptions(missing.convention = ...):
joptions(missing.convention = "spss")
df6 <- jdeclare_missing(clinic, MoodRating, codes = c(-99))
# -99 stays in the cells, flagged as missing (SPSS-style declaration)

joptions(missing.convention = "stata")
df7 <- jdeclare_missing(clinic, MoodRating, codes = c(-99))
# -99 cells become the .a marker; the number -99 leaves the data
} # }