SCMData ¶

Raises

TypeError – dts contains int

scmdata.filters.day_match(data: List, days: Union[List[str], List[int], int, str]) → numpy.ndarray [source]¶

Match days in time columns for data filtering.

Parameters

data – Input data to perform filtering on
days – Days to match

Returns

Array where True indicates a match

Return type

scmdata.filters.find_depth(meta_col: pandas.core.series.Series, s: str, level: Union[int, str], separator: str = '|') → numpy.ndarray [source]¶

Find all values which match given depth from a filter keyword.

Parameters

meta_col – Column in which to find values which match the given depth
s – Filter keyword, from which level should be applied
level – Depth of value to match as defined by the number of separator in the value name. If an int, the depth is matched exactly. If a str, then the depth can be matched as either “X-“, for all levels up to level “X”, or “X+”, for all levels above level “X”.
separator – The string used to separate levels in s. Defaults to a pipe (“|”).

Returns

Array where True indicates a match

Return type

Raises

ValueError – If level cannot be understood

scmdata.filters.hour_match(data: List, hours: Union[List[int], int]) → numpy.ndarray [source]¶

Match hours in time columns for data filtering.

Parameters

data – Input data to perform filtering on
hours – Hours to match

Returns

Array where True indicates a match

Return type

scmdata.filters.is_in(vals: List, items: List) → numpy.ndarray [source]¶

Find elements of vals which are in items.

Parameters

vals – The list of values to check
items – The options used to determine whether each element of vals is in the desired subset or not

Returns

Array of the same length as vals where the element is True if the corresponding element of vals is in items and False otherwise

Return type

scmdata.filters.month_match(data: List, months: Union[List[str], List[int], int, str]) → numpy.ndarray [source]¶

Match months in time columns for data filtering.

Parameters

data – Input data to perform filtering on
months – Months to match

Returns

Array where True indicates a match

Return type

scmdata.filters.pattern_match(meta_col: pandas.core.series.Series, values: Union[Iterable[str], str], level: Optional[Union[str, int]] = None, regexp: bool = False, separator: str = '|') → numpy.ndarray [source]¶

Filter data by matching metadata columns to given patterns.

Parameters

meta_col – Column to perform filtering on
values – Values to match
level – Passed to find_depth(). For usage, see docstring of find_depth().
regexp – If True, match using regexp rather than our pseudo regexp syntax.
has_nan – If True, convert all nan values in meta_col to empty string before applying filters. This means that “” and “*” will match rows with numpy.nan. If False, the conversion is not applied and so a search in a string column which contains numpy.nan will result in a TypeError.
separator – String used to separate the hierarchy levels in values. Defaults to ‘|’

Returns

Array where True indicates a match

Return type

Raises

TypeError – Filtering is performed on a string metadata column which contains numpy.nan and has_nan is False

scmdata.filters.time_match(data: List, times: Union[List[str], List[int], int, str], conv_codes: List[str], strptime_attr: str, name: str) → numpy.ndarray [source]¶

Match times by applying conversion codes to filtering list.

Parameters

data – Input data to perform filtering on
times – Times to match
conv_codes – If times contains strings, conversion codes to try passing to time.strptime() to convert times to datetime.datetime
strptime_attr – If times contains strings, the datetime.datetime attribute to finalize the conversion of strings to integers
name – Name of the part of a datetime to extract, used to produce useful error messages.

Returns

Array where True indicates a match

Return type

Raises

ValueError – If input times cannot be converted understood or if input strings do not lead to increasing integers (i.e. “Nov-Feb” will not work, one must use [“Nov-Dec”, “Jan-Feb”] instead)

scmdata.filters.years_match(data: List, years: Union[List[int], numpy.ndarray, int]) → numpy.ndarray [source]¶

Match years in time columns for data filtering.

Parameters

data – Input data to perform filtering on
years – Years to match

Returns

Array where True indicates a match

Return type

Raises

TypeError – If years is not int or list of int

scmdata.groupby¶

Functionality for grouping and filtering ScmRun objects

class scmdata.groupby.RunGroupBy(run, groups)[source]¶

Bases: scmdata.groupby._GroupBy

GroupBy object specialized to grouping ScmRun objects

all(dim=None, axis=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying all along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply all.
axis (int or sequence of int, optional) – Axis(es) over which to apply all. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then all is calculated over axes.
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating all on this object’s data.

Returns

reduced – New RunGroupBy object with all applied to its data and the indicated dimension(s) removed.

Return type

any(dim=None, axis=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying any along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply any.
axis (int or sequence of int, optional) – Axis(es) over which to apply any. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then any is calculated over axes.
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating any on this object’s data.

Returns

reduced – New RunGroupBy object with any applied to its data and the indicated dimension(s) removed.

Return type

count(dim=None, axis=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying count along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply count.
axis (int or sequence of int, optional) – Axis(es) over which to apply count. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then count is calculated over axes.
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating count on this object’s data.

Returns

reduced – New RunGroupBy object with count applied to its data and the indicated dimension(s) removed.

Return type

map(func, *args, **kwargs)[source]¶

Apply a function to each group and append the results

func is called like func(ar, *args, **kwargs) for each ScmRun ar in this group. If the result of this function call is None, than it is excluded from the results.

The results are appended together using run_append(). The function can change the size of the input ScmRun as long as run_append() can be applied to all results.

Examples

>>> def write_csv(arr):
...     variable = arr.get_unique_meta("variable")
...     arr.to_csv("out-{}.csv".format(variable)
>>> df.groupby("variable").map(write_csv)

Parameters

func (function) – Callable to apply to each timeseries.
*args – Positional arguments passed to func.
**kwargs – Used to call func(ar, **kwargs) for each array ar.

Returns

applied – The result of splitting, applying and combining this array.

Return type

max(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying max along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply max.
axis (int or sequence of int, optional) – Axis(es) over which to apply max. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then max is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating max on this object’s data.

Returns

reduced – New RunGroupBy object with max applied to its data and the indicated dimension(s) removed.

Return type

mean(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying mean along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply mean.
axis (int or sequence of int, optional) – Axis(es) over which to apply mean. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then mean is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating mean on this object’s data.

Returns

reduced – New RunGroupBy object with mean applied to its data and the indicated dimension(s) removed.

Return type

median(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying median along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply median.
axis (int or sequence of int, optional) – Axis(es) over which to apply median. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then median is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating median on this object’s data.

Returns

reduced – New RunGroupBy object with median applied to its data and the indicated dimension(s) removed.

Return type

min(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying min along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply min.
axis (int or sequence of int, optional) – Axis(es) over which to apply min. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then min is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating min on this object’s data.

Returns

reduced – New RunGroupBy object with min applied to its data and the indicated dimension(s) removed.

Return type

prod(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying prod along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply prod.
axis (int or sequence of int, optional) – Axis(es) over which to apply prod. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then prod is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
min_count (int, default: None) – The required number of valid values to perform the operation. If fewer than min_count non-NA values are present the result will be NA. Only used if skipna is set to True or defaults to True for the array’s dtype. New in version 0.10.8: Added with the default being None. Changed in version 0.17.0: if specified on an integer array and skipna=True, the result will be a float array.
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating prod on this object’s data.

Returns

reduced – New RunGroupBy object with prod applied to its data and the indicated dimension(s) removed.

Return type

reduce(func, dim=None, axis=None, **kwargs)[source]¶

Reduce the items in this group by applying func along some dimension(s).

Parameters

func (function) – Function which can be called in the form func(x, axis=axis, **kwargs) to return the result of collapsing an np.ndarray over an integer valued axis.
dim (…, str or sequence of str, optional) – Not used in this implementation
axis (int or sequence of int, optional) – Axis(es) over which to apply func. Only one of the ‘dimension’ and ‘axis’ arguments can be supplied. If neither are supplied, then func is calculated over all dimension for each group item.
**kwargs (dict) – Additional keyword arguments passed on to func.

Returns

reduced – Array with summarized data and the indicated dimension(s) removed.

Return type

std(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying std along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply std.
axis (int or sequence of int, optional) – Axis(es) over which to apply std. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then std is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating std on this object’s data.

Returns

reduced – New RunGroupBy object with std applied to its data and the indicated dimension(s) removed.

Return type

sum(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying sum along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply sum.
axis (int or sequence of int, optional) – Axis(es) over which to apply sum. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then sum is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
min_count (int, default: None) – The required number of valid values to perform the operation. If fewer than min_count non-NA values are present the result will be NA. Only used if skipna is set to True or defaults to True for the array’s dtype. New in version 0.10.8: Added with the default being None. Changed in version 0.17.0: if specified on an integer array and skipna=True, the result will be a float array.
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating sum on this object’s data.

Returns

reduced – New RunGroupBy object with sum applied to its data and the indicated dimension(s) removed.

Return type

var(dim=None, axis=None, skipna=None, **kwargs)¶

Reduce this RunGroupBy’s data by applying var along some dimension(s).

Parameters

dim (str or sequence of str, optional) – Dimension(s) over which to apply var.
axis (int or sequence of int, optional) – Axis(es) over which to apply var. Only one of the ‘dim’ and ‘axis’ arguments can be supplied. If neither are supplied, then var is calculated over axes.
skipna (bool, optional) – If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).
keep_attrs (bool, optional) – If True, the attributes (attrs) will be copied from the original object to the new one. If False (default), the new object will be returned without attributes.
**kwargs (dict) – Additional keyword arguments passed on to the appropriate array function for calculating var on this object’s data.

Returns

reduced – New RunGroupBy object with var applied to its data and the indicated dimension(s) removed.

Return type

scmdata.run.ScmRun.from_nc()

scmdata.netcdf¶

NetCDF4 file operations

Reading and writing ScmRun to disk as binary

scmdata.netcdf.inject_nc_methods(cls)[source]¶

Add the to/from nc methods to a class

Parameters: cls – Class to add methods to

scmdata.netcdf.nc_to_run(cls, fname)[source]¶

Read a netCDF4 file from disk

Parameters: fname (str) – Filename to read

See also

scmdata.netcdf.run_to_nc(run, fname, dimensions=('region'), extras=(), **kwargs)[source]¶

Write timeseries to disk as a netCDF4 file

Each unique variable will be written as a variable within the netCDF file. Choosing the dimensions and extras such that there are as few empty (or nan) values as possible will lead to the best compression on disk.

Parameters

fname (str) – Path to write the file into
dimensions (iterable of str) – Dimensions to include in the netCDF file. The time dimension is always included (if not provided it will be the last dimension). An additional dimension (specifically a co-ordinate in xarray terms), “_id”, will be included if extras is provided and any of the metadata in extras is not uniquely defined by dimensions. “_id” maps the timeseries in each variable to their relevant metadata.
extras (iterable of str) – Metadata columns to write as variables in the netCDF file (specifically as “non-dimension co-ordinates” in xarray terms, see xarray terminology for more details). Where possible, these non-dimension co-ordinates will use dimension co-ordinates as their own co-ordinates. However, if the metadata in extras is not defined by a single dimension in dimensions, then the extras co-ordinates will have dimensions of “_id”. This “_id” co-ordinate maps the values in the extras co-ordinates to each timeseries in the serialised dataset. Where “_id” is required, an extra “_id” dimension will also be added to dimensions.
kwargs – Passed through to xarray.Dataset.to_netcdf()

See also

scmdata.run.ScmRun.to_nc()

scmdata.offsets¶

Allow stepping through time using xarray’s offset functionality

Provides similar functionality to https://pandas.pydata.org/pandas-docs/stable/user_gui de/timeseries.html#dateoffset-objects

scmdata.offsets.generate_range(start: cftime._cftime.datetime, end: cftime._cftime.datetime, offset: xarray.coding.cftime_offsets.BaseCFTimeOffset, date_cls: cftime._cftime.datetime = <class 'cftime._cftime.DatetimeGregorian'>) → Iterable[cftime._cftime.datetime][source]¶

Generate a range of datetime objects between start and end, using offset to determine the steps.

The range will extend both ends of the span to the next valid timestep, see examples.

Parameters

start (cftime.datetime) – Starting datetime from which to generate the range (noting roll backward mentioned above and illustrated in the examples).
end (cftime.datetime) – Last datetime from which to generate the range (noting roll forward mentioned above and illustrated in the examples).
offset – Offset object for determining the timesteps.
date_cls (cftime.datetime) – The time points will be returned as instances of date_cls

Yields

cftime.datetime – Next datetime in the range (the exact class is specified by date_cls)

Raises

ValueError – Offset does not result in increasing cftime.datetime’s

Examples

The range is extended at either end to the nearest timestep. In the example below, the first timestep is rolled back to 1st Jan 2001 whilst the last is extended to 1st Jan 2006.

>>> import datetime as dt
>>> from pprint import pprint
>>> from scmdata.offsets import to_offset, generate_range
>>> g = generate_range(
...     dt.datetime(2001, 4, 1),
...     dt.datetime(2005, 6, 3),
...     to_offset("AS"),
... )

>>> pprint([d for d in g])
[cftime.DatetimeGregorian(2001, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2002, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2003, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2004, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2005, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2006, 1, 1, 0, 0, 0, 0)]

In this example the first timestep is rolled back to 31st Dec 2000 whilst the last is extended to 31st Dec 2005.

>>> g = generate_range(
...     dt.datetime(2001, 4, 1),
...     dt.datetime(2005, 6, 3),
...     to_offset("A"),
... )
>>> pprint([d for d in g])
[cftime.DatetimeGregorian(2000, 12, 31, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2001, 12, 31, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2002, 12, 31, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2003, 12, 31, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2004, 12, 31, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2005, 12, 31, 0, 0, 0, 0)]

In this example the first timestep is already on the offset so stays there, the last timestep is to 1st Sep 2005.

>>> g = generate_range(
...     dt.datetime(2001, 4, 1),
...     dt.datetime(2005, 6, 3),
...     to_offset("QS"),
... )
>>> pprint([d for d in g])
[cftime.DatetimeGregorian(2001, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2001, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2001, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2002, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2002, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2002, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2002, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2003, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2003, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2003, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2003, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2004, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2004, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2004, 7, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2004, 10, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2005, 1, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2005, 4, 1, 0, 0, 0, 0),
 cftime.DatetimeGregorian(2005, 7, 1, 0, 0, 0, 0)]

scmdata.ops¶

Operations for ScmRun objects

These largely rely on Pint’s Pandas interface to handle unit conversions automatically

scmdata.ops.add(self, other, op_cols, **kwargs)[source]¶

Add values

Parameters

other (ScmRun) – ScmRun containing data to add
op_cols (dict of str: str) – Dictionary containing the columns to drop before adding as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the addition will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Sum of self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> total = fos.add(afolu, op_cols={"variable": "Emissions|CO2"})
>>> total.head()
time                                                   2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable      unit
idealised idealised World|NH Emissions|CO2 gigatC / a                  1.0                 13.0                 25.0
                    World|SH Emissions|CO2 gigatC / a                  5.0                 17.0                 29.0
>>>
>>> nh = start.filter(region="*NH")
>>> nh.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU  GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
>>>
>>> sh = start.filter(region="*SH")
>>> sh.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU  GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> world = nh.add(sh, op_cols={"region": "World"})
>>> world.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region variable             unit
idealised idealised World  Emissions|CO2|Fossil gigatC / a                  2.0                 14.0                 26.0
                           Emissions|CO2|AFOLU  gigatC / a                  4.0                 16.0                 28.0

scmdata.ops.delta_per_delta_time(self, out_var=None)[source]¶

Calculate change in timeseries values for each timestep, divided by the size of the timestep

The output is placed on the middle of each timestep and is one timestep shorter than the input.

Parameters: out_var (str) – If provided, the variable column of the output is set equal to out_var. Otherwise, the output variables are equal to the input variables, prefixed with “Delta ” .
Returns: scmdata.ScmRun containing the changes in values of self, normalised by the change in time
Return type: scmdata.ScmRun
Warns: UserWarning – The data contains nans. If this happens, the output data will also contain nans.

scmdata.ops.divide(self, other, op_cols, **kwargs)[source]¶

Divide values (self / other)

Parameters

other (ScmRun) – ScmRun containing data to divide
op_cols (dict of str: str) – Dictionary containing the columns to drop before dividing as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the division will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Quotient of self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> fos_afolu_ratio = fos.divide(
...     afolu, op_cols={"variable": "Emissions|CO2|Fossil : AFOLU"}
... )
>>> fos_afolu_ratio.head()
time                                                                     2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable                     unit
idealised idealised World|NH Emissions|CO2|Fossil : AFOLU dimensionless             0.000000             0.857143             0.923077
                    World|SH Emissions|CO2|Fossil : AFOLU dimensionless             0.666667             0.888889             0.933333

scmdata.ops.inject_ops_methods(cls)[source]¶

Inject the operation methods

Parameters: cls – Target class

scmdata.ops.integrate(self, out_var=None)[source]¶

Integrate with respect to time

Parameters: out_var (str) – If provided, the variable column of the output is set equal to out_var. Otherwise, the output variables are equal to the input variables, prefixed with “Cumulative ” .
Returns: scmdata.ScmRun containing the integral of self with respect to time
Return type: scmdata.ScmRun
Warns: UserWarning – The data being integrated contains nans. If this happens, the output data will also contain nans.

scmdata.ops.linear_regression(self)[source]¶

Calculate linear regression of each timeseries

Note

Times in seconds since 1970-01-01 are used as the x-axis for the regressions. Such values can be accessed with self.time_points.values.astype("datetime64[s]").astype("int"). This decision does not matter for the gradients, but is important for the intercept values.

Returns: list of dict[str – List of dictionaries. Each dictionary contains the metadata for the timeseries plus the gradient (with key "gradient") and intercept ( with key "intercept"). The gradient and intercept are stored as pint.Quantity.
Return type: Any]

scmdata.ops.linear_regression_gradient(self, unit=None)[source]¶

Calculate gradients of a linear regression of each timeseries

Parameters: unit (str) – Output unit for gradients. If not supplied, the gradients’ units will not be converted to a common unit.
Returns: self.meta plus a column with the value of the gradient for each timeseries. The "unit" column is updated to show the unit of the gradient.
Return type: pandas.DataFrame

scmdata.ops.linear_regression_intercept(self, unit=None)[source]¶

Calculate intercepts of a linear regression of each timeseries

Note

Times in seconds since 1970-01-01 are used as the x-axis for the regressions. Such values can be accessed with self.time_points.values.astype("datetime64[s]").astype("int"). This decision does not matter for the gradients, but is important for the intercept values.

Parameters: unit (str) – Output unit for gradients. If not supplied, the gradients’ units will not be converted to a common unit.
Returns: self.meta plus a column with the value of the gradient for each timeseries. The "unit" column is updated to show the unit of the gradient.
Return type: pandas.DataFrame

scmdata.ops.linear_regression_scmrun(self)[source]¶

Re-calculate the timeseries based on a linear regression

Returns: The timeseries, re-calculated based on a linear regression
Return type: scmdata.ScmRun

scmdata.ops.multiply(self, other, op_cols, **kwargs)[source]¶

Multiply values

Parameters

other (ScmRun) – ScmRun containing data to multiply
op_cols (dict of str: str) – Dictionary containing the columns to drop before multiplying as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the multiplication will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Product of self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> fos_times_afolu = fos.multiply(
...     afolu, op_cols={"variable": "Emissions|CO2|Fossil : AFOLU"}
... )
>>> fos_times_afolu.head()
time                                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable                     unit
idealised idealised World|NH Emissions|CO2|Fossil : AFOLU gigatC ** 2 / a ** 2                  0.0                 42.0                156.0
                    World|SH Emissions|CO2|Fossil : AFOLU gigatC ** 2 / a ** 2                  6.0                 72.0                210.0

scmdata.ops.prep_for_op(inp, op_cols, meta, ur=<openscm_units._unit_registry.ScmUnitRegistry object>)[source]¶

Prepare dataframe for operation

Parameters

inp (ScmRun) – ScmRun containing data to prepare
op_cols (dict of str: str) – Dictionary containing the columns to drop in order to prepare for the operation as the keys (the values are not used). For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then we will drop the “variable” column from the index.
ur (pint.UnitRegistry) – Pint unit registry to use for the operation

Returns

Timeseries to use for the operation. They are the transpose of the normal ScmRun.timeseries() output with the columns being Pint arrays (unless “unit” is in op_cols in which case no units are available to be used so the columns are standard numpy arrays). We do this so that we can use Pint’s Pandas interface to handle unit conversions automatically.

Return type

scmdata.ops.set_op_values(output, op_cols)[source]¶

Set operation values in output

Parameters

output (pandas.Dataframe) – Dataframe of which to update the values
op_cols (dict of str: str) – Dictionary containing the columns to update as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.

Returns

output with the relevant columns being set according to op_cols.

Return type

pandas.Dataframe

scmdata.ops.subtract(self, other, op_cols, **kwargs)[source]¶

Subtract values

Parameters

other (ScmRun) – ScmRun containing data to subtract
op_cols (dict of str: str) – Dictionary containing the columns to drop before subtracting as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the subtraction will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Difference between self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> fos_minus_afolu = fos.subtract(
...     afolu, op_cols={"variable": "Emissions|CO2|Fossil - AFOLU"}
... )
>>> fos_minus_afolu.head()
time                                                                  2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable                     unit
idealised idealised World|NH Emissions|CO2|Fossil - AFOLU gigatC / a                 -1.0                 -1.0                 -1.0
                    World|SH Emissions|CO2|Fossil - AFOLU gigatC / a                 -1.0                 -1.0                 -1.0
>>>
>>> nh_minus_sh = nh.subtract(sh, op_cols={"region": "World|NH - SH"})
>>> nh_minus_sh.head()
time                                                               2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region        variable             unit
idealised idealised World|NH - SH Emissions|CO2|Fossil gigatC / a                 -2.0                 -2.0                 -2.0
                                  Emissions|CO2|AFOLU  gigatC / a                 -2.0                 -2.0                 -2.0

scmdata.plotting¶

Plotting helpers for ScmRun

See the example notebook ‘plotting-with-seaborn.ipynb’ for usage examples

scmdata.plotting.inject_plotting_methods(cls)[source]¶

Inject the plotting methods

Parameters: cls – Target class

scmdata.plotting.lineplot(self, time_axis=None, **kwargs)[source]¶

Make a line plot via seaborn’s lineplot

If only a single unit is present, it will be used as the y-axis label. The axis object is returned so this can be changed by the user if desired.

Parameters

time_axis ({None, "year", "year-month", "days since 1970-01-01", "seconds since 1970-01-01"} # noqa: E501) –
Time axis to use for the plot.

If None, datetime.datetime objects will be used.

If "year", the year of each time point will be used.

If "year-month", the year plus (month - 0.5) / 12 will be used.

If "days since 1970-01-01", the number of days since 1st Jan 1970 will be used (calculated using the datetime module).

If "seconds since 1970-01-01", the number of seconds since 1st Jan 1970 will be used (calculated using the datetime module).
**kwargs – Keyword arguments to be passed to seaborn.lineplot. If none are passed, sensible defaults will be used.

Returns

Output of call to seaborn.lineplot

Return type

matplotlib.axes._subplots.AxesSubplot

scmdata.plotting.plumeplot(self, ax=None, quantiles_plumes=[((0.05, 0.95), 0.5), ((0.5), 1.0)], hue_var='scenario', hue_label='Scenario', palette=None, style_var='variable', style_label='Variable', dashes=None, linewidth=2, time_axis=None, pre_calculated=False, quantile_over=('ensemble_member'))[source]¶

Make a plume plot, showing plumes for custom quantiles

Parameters

ax (matplotlib.axes._subplots.AxesSubplot) – Axes on which to make the plot
quantiles_plumes (list[tuple[tuple, float]]) – Configuration to use when plotting quantiles. Each element is a tuple, the first element of which is itself a tuple and the second element of which is the alpha to use for the quantile. If the first element has length two, these two elements are the quantiles to plot and a plume will be made between these two quantiles. If the first element has length one, then a line will be plotted to represent this quantile.
hue_var (str) – The column of self.meta which should be used to distinguish different hues.
hue_label (str) – Label to use in the legend for hue_var.
palette (dict) – Dictionary defining the colour to use for different values of hue_var.
style_var (str) – The column of self.meta which should be used to distinguish different styles.
style_label (str) – Label to use in the legend for style_var.
dashes (dict) – Dictionary defining the style to use for different values of style_var.
linewidth (float) – Width of lines to use (for quantiles which are not to be shown as plumes)
time_axis (str) – Time axis to use for the plot (see timeseries())
pre_calculated (bool) – Are the quantiles pre-calculated? If no, the quantiles will be calculated within this function. Pre-calculating the quantiles using ScmRun.quantiles_over() can lead to faster plotting if multiple plots are to be made with the same quantiles.
quantile_over (str, tuple[str]) – Columns of self.meta over which the quantiles should be calculated. Only used if pre_calculated is False.

Returns

Axes on which the plot was made and the legend items we have made (in case the user wants to move the legend to a different position for example)

Return type

matplotlib.axes._subplots.AxesSubplot, list

Examples

>>> scmrun = ScmRun(
...     data=np.random.random((10, 3)).T,
...     columns={
...         "model": ["a_iam"],
...         "climate_model": ["a_model"] * 5 + ["a_model_2"] * 5,
...         "scenario": ["a_scenario"] * 5 + ["a_scenario_2"] * 5,
...         "ensemble_member": list(range(5)) + list(range(5)),
...         "region": ["World"],
...         "variable": ["Surface Air Temperature Change"],
...         "unit": ["K"],
...     },
...     index=[2005, 2010, 2015],
... )

Plot the plumes, calculated over the different ensemble members.

>>> scmrun.plumeplot(quantile_over="ensemble_member")

Pre-calculate the quantiles, then plot

>>> summary_stats = ScmRun(
...     scmrun.quantiles_over("ensemble_member", quantiles=quantiles)
... )
>>> summary_stats.plumeplot(pre_calculated=True)

Note

scmdata is not a plotting library so this function is provided as is, with little testing. In some ways, it is more intended as inspiration for other users than as a robust plotting tool.

scmdata.run¶

class scmdata.run.ScmRun(data: Any, index: Optional[Any] = None, columns: Optional[Union[Dict[str, list], Dict[str, str]]] = None, metadata: Optional[Dict[str, Union[str, int, float]]] = None, copy_data: bool = False, **kwargs: Any)[source]¶

Bases: scmdata.run.BaseScmRun

Data container for holding one or many time-series of SCM data.

__init__(data: Any, index: Optional[Any] = None, columns: Optional[Union[Dict[str, list], Dict[str, str]]] = None, metadata: Optional[Dict[str, Union[str, int, float]]] = None, copy_data: bool = False, **kwargs: Any)¶

Initialize the container with timeseries data.

Parameters

data (Union[ScmRun, IamDataFrame, pd.DataFrame, np.ndarray, str]) –
If a ScmRun object is provided, then a new ScmRun is created with a copy of the values and metadata from :obj: data.

A pandas.DataFrame with IAMC-format data columns (the result from ScmRun.timeseries()) can be provided without any additional columns and index information.

If a numpy array of timeseries data is provided, columns and index must also be specified. The shape of the numpy array should be (n_times, n_series) where n_times is the number of timesteps and n_series is the number of time series.

If a string is passed, data will be attempted to be read from file. Currently, reading from CSV, gzipped CSV and Excel formatted files is supported.
index (np.ndarray) –
If index is not None, then the index is used as the timesteps for run. All timeseries in the run use the same set of timesteps.

The values will be attempted to be converted to numpy.datetime[s] values. Possible input formats include :
- datetime.datetime
- int Start of year
- float Decimal year
- str Uses dateutil.parser(). Slow and should be avoided if possible
If index is None, than the time index will be obtained from the data if possible.

columns –

If None, ScmRun will attempt to infer the values from the source. Otherwise, use this dict to write the metadata for each timeseries in data. For each metadata key (e.g. “model”, “scenario”), an array of values (one per time series) is expected. Alternatively, providing a list of length 1 applies the same value to all timeseries in data. For example, if you had three timeseries from ‘rcp26’ for 3 different models ‘model’, ‘model2’ and ‘model3’, the column dict would look like either ‘col_1’ or ‘col_2’:

>>> col_1 = {
    "scenario": ["rcp26"],
    "model": ["model1", "model2", "model3"],
    "region": ["unspecified"],
    "variable": ["unspecified"],
    "unit": ["unspecified"]
}
>>> col_2 = {
    "scenario": ["rcp26", "rcp26", "rcp26"],
    "model": ["model1", "model2", "model3"],
    "region": ["unspecified"],
    "variable": ["unspecified"],
    "unit": ["unspecified"]
}
>>> assert pd.testing.assert_frame_equal(
    ScmRun(d, columns=col_1).meta,
    ScmRun(d, columns=col_2).meta
)

metadata –
Optional dictionary of metadata for instance as a whole.

This can be used to store information such as the longer-form information about a particular dataset, for example, dataset description or DOIs.

Defaults to an empty dict if no default metadata are provided.
copy_data (bool) –
If True, an explicit copy of data is performed.

Note

The copy can be very expensive on large timeseries and should only be needed in cases where the original data is manipulated.
**kwargs – Additional parameters passed to _read_file() to read files

Raises

ValueError –
- If you try to load from multiple files at once. If you wish to do this, please use scmdata.run.run_append() instead. * Not specifying index and columns if data is a numpy.ndarray
scmdata.errors.MissingRequiredColumn – If metadata for required_cols is not found
TypeError – Timeseries cannot be read from data

add(other, op_cols, **kwargs)¶

Add values

Parameters

other (ScmRun) – ScmRun containing data to add
op_cols (dict of str: str) – Dictionary containing the columns to drop before adding as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the addition will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Sum of self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> total = fos.add(afolu, op_cols={"variable": "Emissions|CO2"})
>>> total.head()
time                                                   2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable      unit
idealised idealised World|NH Emissions|CO2 gigatC / a                  1.0                 13.0                 25.0
                    World|SH Emissions|CO2 gigatC / a                  5.0                 17.0                 29.0
>>>
>>> nh = start.filter(region="*NH")
>>> nh.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU  GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
>>>
>>> sh = start.filter(region="*SH")
>>> sh.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU  GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> world = nh.add(sh, op_cols={"region": "World"})
>>> world.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region variable             unit
idealised idealised World  Emissions|CO2|Fossil gigatC / a                  2.0                 14.0                 26.0
                           Emissions|CO2|AFOLU  gigatC / a                  4.0                 16.0                 28.0

append(other, inplace: bool = False, duplicate_msg: Union[str, bool] = True, metadata: Optional[Dict[str, Union[str, int, float]]] = None, **kwargs: Any)¶

Append additional data to the current dataframe.

For details, see run_append().

Parameters

other – Data (in format which can be cast to ScmRun) to append
inplace – If True, append data in place and return None. Otherwise, return a new ScmRun instance with the appended data.
duplicate_msg – If True, raise a scmdata.errors.NonUniqueMetadataError error so the user can see the duplicate timeseries. If False, take the average and do not raise a warning or error. If "warn", raise a warning if duplicate data is detected.
metadata – If not None, override the metadata of the resulting ScmRun with metadata. Otherwise, the metadata for the runs are merged. In the case where there are duplicate metadata keys, the values from the first run are used.
**kwargs – Keywords to pass to ScmRun.__init__() when reading other

Returns

If not inplace, return a new ScmRun instance containing the result of the append.

Return type

Raises

NonUniqueMetadataError – If the appending results in timeseries with duplicate metadata and duplicate_msg is True

convert_unit(unit: str, context: Optional[str] = None, inplace: bool = False, **kwargs: Any)¶

Convert the units of a selection of timeseries.

Uses scmdata.units.UnitConverter to perform the conversion.

Parameters

unit – Unit to convert to. This must be recognised by UnitConverter.
context – Context to use for the conversion i.e. which metric to apply when performing CO2-equivalent calculations. If None, no metric will be applied and CO2-equivalent calculations will raise DimensionalityError.
inplace – If True, apply the conversion inplace and return None
**kwargs – Extra arguments which are passed to filter() to limit the timeseries which are attempted to be converted. Defaults to selecting the entire ScmRun, which will likely fail.

Returns

If inplace is not False, a new ScmRun instance with the converted units.

Return type

Notes

If context is not None, then the context used for the conversion will be checked against any existing metadata and, if the conversion is valid, stored in the output’s metadata.

Raises: ValueError – "unit_context" is already included in self’s meta_attributes() and it does not match context for the variables to be converted.

copy()¶

Return a copy.deepcopy() of self.

Also creates copies the underlying Timeseries data

Returns: copy.deepcopy() of self
Return type: ScmRun

data_hierarchy_separator = '|'¶

String used to define different levels in our data hierarchies.

By default we follow pyam and use “|”. In such a case, emissions of CO₂ for energy from coal would be “Emissions|CO2|Energy|Coal”.

Type: str

delta_per_delta_time(out_var=None)¶

Calculate change in timeseries values for each timestep, divided by the size of the timestep

The output is placed on the middle of each timestep and is one timestep shorter than the input.

Parameters: out_var (str) – If provided, the variable column of the output is set equal to out_var. Otherwise, the output variables are equal to the input variables, prefixed with “Delta ” .
Returns: scmdata.ScmRun containing the changes in values of self, normalised by the change in time
Return type: scmdata.ScmRun
Warns: UserWarning – The data contains nans. If this happens, the output data will also contain nans.

divide(other, op_cols, **kwargs)¶

Divide values (self / other)

Parameters

other (ScmRun) – ScmRun containing data to divide
op_cols (dict of str: str) – Dictionary containing the columns to drop before dividing as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the division will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Quotient of self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> fos_afolu_ratio = fos.divide(
...     afolu, op_cols={"variable": "Emissions|CO2|Fossil : AFOLU"}
... )
>>> fos_afolu_ratio.head()
time                                                                     2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable                     unit
idealised idealised World|NH Emissions|CO2|Fossil : AFOLU dimensionless             0.000000             0.857143             0.923077
                    World|SH Emissions|CO2|Fossil : AFOLU dimensionless             0.666667             0.888889             0.933333

drop_meta(columns: Union[list, str], inplace: Optional[bool] = False)¶

Drop meta columns out of the Run

Parameters

columns – The column or columns to drop
inplace – If True, do operation inplace and return None.

Raises

KeyError – If any of the columns do not exist in the meta DataFrame

property empty: bool¶

Indicate whether ScmRun is empty i.e. contains no data

Returns: If ScmRun is empty, return True, if not return False
Return type: bool

filter(keep: bool = True, inplace: bool = False, log_if_empty: bool = True, **kwargs: Any)¶

Return a filtered ScmRun (i.e., a subset of the data).

>>> df
<scmdata.ScmRun (timeseries: 3, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-01-01T00:00:00
Meta:
       model     scenario region             variable   unit climate_model
    0  a_iam   a_scenario  World       Primary Energy  EJ/yr       a_model
    1  a_iam   a_scenario  World  Primary Energy|Coal  EJ/yr       a_model
    2  a_iam  a_scenario2  World       Primary Energy  EJ/yr       a_model
    [3 rows x 7 columns]

>>> df.filter(scenario="a_scenario")
<scmdata.ScmRun (timeseries: 2, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-01-01T00:00:00
Meta:
       model     scenario region             variable   unit climate_model
    0  a_iam   a_scenario  World       Primary Energy  EJ/yr       a_model
    1  a_iam   a_scenario  World  Primary Energy|Coal  EJ/yr       a_model
    [2 rows x 7 columns]

>>> df.filter(scenario="a_scenario", keep=False)
<scmdata.ScmRun (timeseries: 1, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-01-01T00:00:00
Meta:
       model     scenario region             variable   unit climate_model
    2  a_iam  a_scenario2  World       Primary Energy  EJ/yr       a_model
    [1 rows x 7 columns]

>>> df.filter(level=1)
<scmdata.ScmRun (timeseries: 2, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-01-01T00:00:00
Meta:
       model     scenario region             variable   unit climate_model
    0  a_iam   a_scenario  World       Primary Energy  EJ/yr       a_model
    2  a_iam  a_scenario2  World       Primary Energy  EJ/yr       a_model
    [2 rows x 7 columns]

>>> df.filter(year=range(2000, 2011))
<scmdata.ScmRun (timeseries: 3, timepoints: 2)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2010-01-01T00:00:00
Meta:
       model     scenario region             variable   unit climate_model
    0  a_iam   a_scenario  World       Primary Energy  EJ/yr       a_model
    1  a_iam   a_scenario  World  Primary Energy|Coal  EJ/yr       a_model
    2  a_iam  a_scenario2  World       Primary Energy  EJ/yr       a_model
    [2 rows x 7 columns]

Parameters

keep – If True, keep all timeseries satisfying the filters, otherwise drop all the timeseries satisfying the filters
inplace – If True, do operation inplace and return None
log_if_empty – If True, log a warning level message if the result is empty.
**kwargs –
Argument names are keys with which to filter, values are used to do the filtering. Filtering can be done on:
- all metadata columns with strings, “*” can be used as a wildcard in search strings
- ’level’: the maximum “depth” of IAM variables (number of hierarchy levels, excluding the strings given in the ‘variable’ argument)
- ’time’: takes a datetime.datetime or list of datetime.datetime’s TODO: default to np.datetime64
- ’year’, ‘month’, ‘day’, hour’: takes an int or list of int’s (‘month’ and ‘day’ also accept str or list of str)
If regexp=True is included in kwargs then the pseudo-regexp syntax in pattern_match() is disabled.

Returns

If not inplace, return a new instance with the filtered data.

Return type

scmdata.run.ScmRun.from_nc()

classmethod from_nc(fname)¶

Read a netCDF4 file from disk

Parameters: fname (str) – Filename to read

See also

get_unique_meta(meta: str, no_duplicates: Optional[bool] = False) → Union[List[Any], Any]¶

Get unique values in a metadata column.

Parameters

meta – Column to retrieve metadata for
no_duplicates – Should I raise an error if there is more than one unique value in the metadata column?

Raises

ValueError – There is more than one unique value in the metadata column and no_duplicates is True.
KeyError – If a meta column does not exist in the run’s metadata

Returns

List of unique metadata values. If no_duplicates is True the metadata value will be returned (rather than a list).

Return type

[List[Any], Any]

groupby(*group)¶

Group the object by unique metadata

Enables iteration over groups of data. For example, to iterate over each scenario in the object

>>> for group in df.groupby("scenario"):
>>>    print(group)
<scmdata.ScmRun (timeseries: 2, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-01-01T00:00:00
Meta:
       model    scenario region             variable   unit climate_model
    0  a_iam  a_scenario  World       Primary Energy  EJ/yr       a_model
    1  a_iam  a_scenario  World  Primary Energy|Coal  EJ/yr       a_model
<scmdata.ScmRun (timeseries: 1, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-01-01T00:00:00
Meta:
       model     scenario region        variable   unit climate_model
    2  a_iam  a_scenario2  World  Primary Energy  EJ/yr       a_model

Parameters: group (str or list of str) – Columns to group by
Returns: See the documentation for RunGroupBy for more information
Return type: RunGroupBy

head(*args, **kwargs)¶

Return head of self.timeseries().

Parameters

*args – Passed to self.timeseries().head()
**kwargs – Passed to self.timeseries().head()

Returns

Tail of self.timeseries()

Return type

integrate(out_var=None)¶

Integrate with respect to time

Parameters: out_var (str) – If provided, the variable column of the output is set equal to out_var. Otherwise, the output variables are equal to the input variables, prefixed with “Cumulative ” .
Returns: scmdata.ScmRun containing the integral of self with respect to time
Return type: scmdata.ScmRun
Warns: UserWarning – The data being integrated contains nans. If this happens, the output data will also contain nans.

interpolate(target_times: Union[numpy.ndarray, List[Union[datetime.datetime, int]]], interpolation_type: str = 'linear', extrapolation_type: str = 'linear')¶

Interpolate the dataframe onto a new time frame.

Parameters

target_times – Time grid onto which to interpolate
interpolation_type (str) – Interpolation type. Options are ‘linear’
extrapolation_type (str or None) – Extrapolation type. Options are None, ‘linear’ or ‘constant’

Returns

A new ScmRun containing the data interpolated onto the target_times grid

Return type

line_plot(**kwargs)¶

Make a line plot via seaborn’s lineplot

Deprecated: use lineplot() instead

Parameters: **kwargs – Keyword arguments to be passed to seaborn.lineplot. If none are passed, sensible defaults will be used.
Returns: Output of call to seaborn.lineplot
Return type: matplotlib.axes._subplots.AxesSubplot

linear_regression()¶

Calculate linear regression of each timeseries

Note

Times in seconds since 1970-01-01 are used as the x-axis for the regressions. Such values can be accessed with self.time_points.values.astype("datetime64[s]").astype("int"). This decision does not matter for the gradients, but is important for the intercept values.

Returns: list of dict[str – List of dictionaries. Each dictionary contains the metadata for the timeseries plus the gradient (with key "gradient") and intercept ( with key "intercept"). The gradient and intercept are stored as pint.Quantity.
Return type: Any]

linear_regression_gradient(unit=None)¶

Calculate gradients of a linear regression of each timeseries

Parameters: unit (str) – Output unit for gradients. If not supplied, the gradients’ units will not be converted to a common unit.
Returns: self.meta plus a column with the value of the gradient for each timeseries. The "unit" column is updated to show the unit of the gradient.
Return type: pandas.DataFrame

linear_regression_intercept(unit=None)¶

Calculate intercepts of a linear regression of each timeseries

Note

Times in seconds since 1970-01-01 are used as the x-axis for the regressions. Such values can be accessed with self.time_points.values.astype("datetime64[s]").astype("int"). This decision does not matter for the gradients, but is important for the intercept values.

Parameters: unit (str) – Output unit for gradients. If not supplied, the gradients’ units will not be converted to a common unit.
Returns: self.meta plus a column with the value of the gradient for each timeseries. The "unit" column is updated to show the unit of the gradient.
Return type: pandas.DataFrame

linear_regression_scmrun()¶

Re-calculate the timeseries based on a linear regression

Returns: The timeseries, re-calculated based on a linear regression
Return type: scmdata.ScmRun

lineplot(time_axis=None, **kwargs)¶

Make a line plot via seaborn’s lineplot

If only a single unit is present, it will be used as the y-axis label. The axis object is returned so this can be changed by the user if desired.

Parameters

time_axis ({None, "year", "year-month", "days since 1970-01-01", "seconds since 1970-01-01"} # noqa: E501) –
Time axis to use for the plot.

If None, datetime.datetime objects will be used.

If "year", the year of each time point will be used.

If "year-month", the year plus (month - 0.5) / 12 will be used.

If "days since 1970-01-01", the number of days since 1st Jan 1970 will be used (calculated using the datetime module).

If "seconds since 1970-01-01", the number of seconds since 1st Jan 1970 will be used (calculated using the datetime module).
**kwargs – Keyword arguments to be passed to seaborn.lineplot. If none are passed, sensible defaults will be used.

Returns

Output of call to seaborn.lineplot

Return type

matplotlib.axes._subplots.AxesSubplot

long_data(time_axis=None)¶

Return data in long form, particularly useful for plotting with seaborn

Parameters

time_axis ({None, "year", "year-month", "days since 1970-01-01", "seconds since 1970-01-01"}) –

Time axis to use for the output’s columns.

If None, datetime.datetime objects will be used.

If "year", the year of each time point will be used.

If "year-month", the year plus (month - 0.5) / 12 will be used.

If "days since 1970-01-01", the number of days since 1st Jan 1970 will be used (calculated using the datetime module).

If "seconds since 1970-01-01", the number of seconds since 1st Jan 1970 will be used (calculated using the datetime module).

Returns

pandas.DataFrame containing the data in ‘long form’ (i.e. one observation per row).

Return type

property meta: pandas.core.frame.DataFrame¶: Metadata

property meta_attributes¶

Get a list of all meta keys

Returns: Sorted list of meta keys
Return type: list

multiply(other, op_cols, **kwargs)¶

Multiply values

Parameters

other (ScmRun) – ScmRun containing data to multiply
op_cols (dict of str: str) – Dictionary containing the columns to drop before multiplying as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the multiplication will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Product of self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> fos_times_afolu = fos.multiply(
...     afolu, op_cols={"variable": "Emissions|CO2|Fossil : AFOLU"}
... )
>>> fos_times_afolu.head()
time                                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable                     unit
idealised idealised World|NH Emissions|CO2|Fossil : AFOLU gigatC ** 2 / a ** 2                  0.0                 42.0                156.0
                    World|SH Emissions|CO2|Fossil : AFOLU gigatC ** 2 / a ** 2                  6.0                 72.0                210.0

plumeplot(ax=None, quantiles_plumes=[((0.05, 0.95), 0.5), ((0.5), 1.0)], hue_var='scenario', hue_label='Scenario', palette=None, style_var='variable', style_label='Variable', dashes=None, linewidth=2, time_axis=None, pre_calculated=False, quantile_over=('ensemble_member'))¶

Make a plume plot, showing plumes for custom quantiles

Parameters

ax (matplotlib.axes._subplots.AxesSubplot) – Axes on which to make the plot
quantiles_plumes (list[tuple[tuple, float]]) – Configuration to use when plotting quantiles. Each element is a tuple, the first element of which is itself a tuple and the second element of which is the alpha to use for the quantile. If the first element has length two, these two elements are the quantiles to plot and a plume will be made between these two quantiles. If the first element has length one, then a line will be plotted to represent this quantile.
hue_var (str) – The column of self.meta which should be used to distinguish different hues.
hue_label (str) – Label to use in the legend for hue_var.
palette (dict) – Dictionary defining the colour to use for different values of hue_var.
style_var (str) – The column of self.meta which should be used to distinguish different styles.
style_label (str) – Label to use in the legend for style_var.
dashes (dict) – Dictionary defining the style to use for different values of style_var.
linewidth (float) – Width of lines to use (for quantiles which are not to be shown as plumes)
time_axis (str) – Time axis to use for the plot (see timeseries())
pre_calculated (bool) – Are the quantiles pre-calculated? If no, the quantiles will be calculated within this function. Pre-calculating the quantiles using ScmRun.quantiles_over() can lead to faster plotting if multiple plots are to be made with the same quantiles.
quantile_over (str, tuple[str]) – Columns of self.meta over which the quantiles should be calculated. Only used if pre_calculated is False.

Returns

Axes on which the plot was made and the legend items we have made (in case the user wants to move the legend to a different position for example)

Return type

matplotlib.axes._subplots.AxesSubplot, list

Examples

>>> scmrun = ScmRun(
...     data=np.random.random((10, 3)).T,
...     columns={
...         "model": ["a_iam"],
...         "climate_model": ["a_model"] * 5 + ["a_model_2"] * 5,
...         "scenario": ["a_scenario"] * 5 + ["a_scenario_2"] * 5,
...         "ensemble_member": list(range(5)) + list(range(5)),
...         "region": ["World"],
...         "variable": ["Surface Air Temperature Change"],
...         "unit": ["K"],
...     },
...     index=[2005, 2010, 2015],
... )

Plot the plumes, calculated over the different ensemble members.

>>> scmrun.plumeplot(quantile_over="ensemble_member")

Pre-calculate the quantiles, then plot

>>> summary_stats = ScmRun(
...     scmrun.quantiles_over("ensemble_member", quantiles=quantiles)
... )
>>> summary_stats.plumeplot(pre_calculated=True)

Note

scmdata is not a plotting library so this function is provided as is, with little testing. In some ways, it is more intended as inspiration for other users than as a robust plotting tool.

process_over(cols: Union[str, List[str]], operation: Union[str, Callable[[pandas.core.frame.DataFrame], Union[pandas.core.frame.DataFrame, pandas.core.series.Series, float]]], na_override=- 1000000.0, **kwargs: Any) → pandas.core.frame.DataFrame¶

Process the data over the input columns.

Parameters

cols – Columns to perform the operation on. The timeseries will be grouped by all other columns in meta.
operation (str or func) –
The operation to perform.

If a string is provided, the equivalent pandas groupby function is used. Note that not all groupby functions are available as some do not make sense for this particular application. Additional information about the arguments for the pandas groupby functions can be found at <https://pandas.pydata.org/pan das-docs/stable/reference/groupby.html>`_.

If a function is provided, it will be applied to each group. The function must take a dataframe as its first argument and return a DataFrame, Series or scalar.

Note that quantile means the value of the data at a given point in the cumulative distribution of values at each point in the timeseries, for each timeseries once the groupby is applied. As a result, using q=0.5 is the same as taking the median and not the same as taking the mean/average.
na_override ([int, float]) –
Convert any nan value in the timeseries meta to this value during processsing. The meta values converted back to nan’s before the dataframe is returned. This should not need to be changed unless the existing metadata clashes with the default na_override value.

This functionality is disabled if na_override is None, but may result incorrect results if the timeseries meta includes any nan’s.
**kwargs – Keyword arguments to pass to the pandas operation

Returns

The quantiles of the timeseries, grouped by all columns in meta other than cols

Return type

Raises

ValueError – If the operation is not an allowed operation If the value of na_override clashes with any existing metadata

quantiles_over(cols: Union[str, List[str]], quantiles: Union[str, List[float]], **kwargs: Any) → pandas.core.frame.DataFrame¶

Calculate quantiles of the data over the input columns.

Parameters

cols – Columns to perform the operation on. The timeseries will be grouped by all other columns in meta.
quantiles – The quantiles to calculate. This should be a list of quantiles to calculate (quantile values between 0 and 1). quantiles can also include the strings “median” or “mean” if these values are to be calculated.
**kwargs – Passed to process_over().

Returns

The quantiles of the timeseries, grouped by all columns in meta other than cols. Each calculated quantile is given a label which is stored in the quantile column within the output index.

Return type

Raises

TypeError – operation is included in kwargs. The operation is inferred from quantiles.

reduce(func, dim=None, axis=None, **kwargs)¶

Apply a function along a given axis

This is to provide the GroupBy functionality in ScmRun.groupby() and is not generally called directly.

This implementation is very bare-bones - no reduction along the time time dimension is allowed and only the dim parameter is used.

Parameters

func (function) –
dim (str) – Ignored
axis (int) – The dimension along which the function is applied. The only valid value is 0 which corresponds to the along the time-series dimension.
kwargs – Other parameters passed to func

Returns

Return type

Raises

ValueError – If a dimension other than None is provided
NotImplementedError – If axis is anything other than 0

relative_to_ref_period_mean(append_str=None, **kwargs)¶

Return the timeseries relative to a given reference period mean.

The reference period mean is subtracted from all values in the input timeseries.

Parameters

append_str – Deprecated
**kwargs – Arguments to pass to filter() to determine the data to be included in the reference time period. See the docs of filter() for valid options.

Returns

New object containing the timeseries, adjusted to the reference period mean. The reference period year bounds are stored in the meta columns "reference_period_start_year" and "reference_period_end_year".

Return type

Raises

NotImplementedError – append_str is not None

required_cols = ('model', 'scenario', 'region', 'variable', 'unit')¶

Minimum metadata columns required by an ScmRun.

If an application requires a different set of required metadata, this can be specified by overriding required_cols on a custom class inheriting scmdata.run.BaseScmRun. Note that at a minimum, (“variable”, “unit”) columns are required.

resample(rule: str = 'AS', **kwargs: Any)¶

Resample the time index of the timeseries data onto a custom grid.

This helper function allows for values to be easily interpolated onto annual or monthly timesteps using the rules=’AS’ or ‘MS’ respectively. Internally, the interpolate function performs the regridding.

Parameters

rule – See the pandas user guide for a list of options. Note that Business-related offsets such as “BusinessDay” are not supported.
**kwargs – Other arguments to pass through to interpolate()

Returns

New ScmRun instance on a new time index

Return type

Examples

Resample a dataframe to annual values

>>> scm_df = ScmRun(
...     pd.Series([1, 2, 10], index=(2000, 2001, 2009)),
...     columns={
...         "model": ["a_iam"],
...         "scenario": ["a_scenario"],
...         "region": ["World"],
...         "variable": ["Primary Energy"],
...         "unit": ["EJ/y"],
...     }
... )
>>> scm_df.timeseries().T
model             a_iam
scenario     a_scenario
region            World
variable Primary Energy
unit               EJ/y
year
2000                  1
2010                 10

An annual timeseries can be the created by interpolating to the start of years using the rule ‘AS’.

>>> res = scm_df.resample('AS')
>>> res.timeseries().T
model                        a_iam
scenario                a_scenario
region                       World
variable            Primary Energy
unit                          EJ/y
time
2000-01-01 00:00:00       1.000000
2001-01-01 00:00:00       2.001825
2002-01-01 00:00:00       3.000912
2003-01-01 00:00:00       4.000000
2004-01-01 00:00:00       4.999088
2005-01-01 00:00:00       6.000912
2006-01-01 00:00:00       7.000000
2007-01-01 00:00:00       7.999088
2008-01-01 00:00:00       8.998175
2009-01-01 00:00:00      10.00000

>>> m_df = scm_df.resample('MS')
>>> m_df.timeseries().T
model                        a_iam
scenario                a_scenario
region                       World
variable            Primary Energy
unit                          EJ/y
time
2000-01-01 00:00:00       1.000000
2000-02-01 00:00:00       1.084854
2000-03-01 00:00:00       1.164234
2000-04-01 00:00:00       1.249088
2000-05-01 00:00:00       1.331204
2000-06-01 00:00:00       1.416058
2000-07-01 00:00:00       1.498175
2000-08-01 00:00:00       1.583029
2000-09-01 00:00:00       1.667883
                            ...
2008-05-01 00:00:00       9.329380
2008-06-01 00:00:00       9.414234
2008-07-01 00:00:00       9.496350
2008-08-01 00:00:00       9.581204
2008-09-01 00:00:00       9.666058
2008-10-01 00:00:00       9.748175
2008-11-01 00:00:00       9.833029
2008-12-01 00:00:00       9.915146
2009-01-01 00:00:00      10.000000
[109 rows x 1 columns]

Note that the values do not fall exactly on integer values as not all years are exactly the same length.

References

See the pandas documentation for resample <http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas. Series.resample.html> for more information about possible arguments.

property shape: tuple¶

Get the shape of the underlying data as (num_timeseries, num_timesteps)

Returns
Return type: tuple of int

subtract(other, op_cols, **kwargs)¶

Subtract values

Parameters

other (ScmRun) – ScmRun containing data to subtract
op_cols (dict of str: str) – Dictionary containing the columns to drop before subtracting as the keys and the value those columns should hold in the output as the values. For example, if we have op_cols={"variable": "Emissions|CO2 - Emissions|CO2|Fossil"} then the subtraction will be performed with an index that uses all columns except the “variable” column and the output will have a “variable” column with the value “Emissions|CO2 - Emissions|CO2|Fossil”.
**kwargs (any) – Passed to prep_for_op()

Returns

Difference between self and other, using op_cols to define the columns which should be dropped before the data is aligned and to define the value of these columns in the output.

Return type

Examples

>>> import numpy as np
>>> from scmdata import ScmRun
>>>
>>> IDX = [2010, 2020, 2030]
>>>
>>>
>>> start = ScmRun(
...     data=np.arange(18).reshape(3, 6),
...     index=IDX,
...     columns={
...         "variable": [
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Emissions|CO2|Fossil",
...             "Emissions|CO2|AFOLU",
...             "Cumulative Emissions|CO2",
...             "Surface Air Temperature Change",
...         ],
...         "unit": ["GtC / yr", "GtC / yr", "GtC / yr", "GtC / yr", "GtC", "K"],
...         "region": ["World|NH", "World|NH", "World|SH", "World|SH", "World", "World"],
...         "model": "idealised",
...         "scenario": "idealised",
...     },
... )
>>>
>>> start.head()
time                                                            2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable                 unit     region   model     scenario
Emissions|CO2|Fossil     GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
Emissions|CO2|AFOLU      GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
Emissions|CO2|Fossil     GtC / yr World|SH idealised idealised                  2.0                  8.0                 14.0
Emissions|CO2|AFOLU      GtC / yr World|SH idealised idealised                  3.0                  9.0                 15.0
Cumulative Emissions|CO2 GtC      World    idealised idealised                  4.0                 10.0                 16.0
>>> fos = start.filter(variable="*Fossil")
>>> fos.head()
time                                                        2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable             unit     region   model     scenario
Emissions|CO2|Fossil GtC / yr World|NH idealised idealised                  0.0                  6.0                 12.0
                              World|SH idealised idealised                  2.0                  8.0                 14.0
>>>
>>> afolu = start.filter(variable="*AFOLU")
>>> afolu.head()
time                                                       2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
variable            unit     region   model     scenario
Emissions|CO2|AFOLU GtC / yr World|NH idealised idealised                  1.0                  7.0                 13.0
                             World|SH idealised idealised                  3.0                  9.0                 15.0
>>>
>>> fos_minus_afolu = fos.subtract(
...     afolu, op_cols={"variable": "Emissions|CO2|Fossil - AFOLU"}
... )
>>> fos_minus_afolu.head()
time                                                                  2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region   variable                     unit
idealised idealised World|NH Emissions|CO2|Fossil - AFOLU gigatC / a                 -1.0                 -1.0                 -1.0
                    World|SH Emissions|CO2|Fossil - AFOLU gigatC / a                 -1.0                 -1.0                 -1.0
>>>
>>> nh_minus_sh = nh.subtract(sh, op_cols={"region": "World|NH - SH"})
>>> nh_minus_sh.head()
time                                                               2010-01-01 00:00:00  2020-01-01 00:00:00  2030-01-01 00:00:00
model     scenario  region        variable             unit
idealised idealised World|NH - SH Emissions|CO2|Fossil gigatC / a                 -2.0                 -2.0                 -2.0
                                  Emissions|CO2|AFOLU  gigatC / a                 -2.0                 -2.0                 -2.0

tail(*args: Any, **kwargs: Any) → pandas.core.frame.DataFrame¶

Return tail of self.timeseries().

Parameters

*args – Passed to self.timeseries().tail()
**kwargs – Passed to self.timeseries().tail()

Returns

Tail of self.timeseries()

Return type

time_mean(rule: str)¶

Take time mean of self

Note that this method will not copy the metadata attribute to the returned value.

Parameters

rule (["AC", "AS", "A"]) –

How to take the time mean. The names reflect the pandas user guide where they can, but only the options given above are supported. For clarity, if rule is 'AC', then the mean is an annual mean i.e. each time point in the result is the mean of all values for that particular year. If rule is 'AS', then the mean is an annual mean centred on the beginning of the year i.e. each time point in the result is the mean of all values from July 1st in the previous year to June 30 in the given year. If rule is 'A', then the mean is an annual mean centred on the end of the year i.e. each time point in the result is the mean of all values from July 1st of the given year to June 30 in the next year.

Returns

The time mean of self.

Return type

property time_points¶

Time points of the data

Returns
Return type: scmdata.time.TimePoints

timeseries(meta=None, check_duplicated=True, time_axis=None, drop_all_nan_times=False)¶

Return the data with metadata as a pandas.DataFrame.

Parameters

meta (list[str]) – The list of meta columns that will be included in the output’s MultiIndex. If None (default), then all metadata will be used.
check_duplicated (bool) – If True, an exception is raised if any of the timeseries have duplicated metadata
time_axis ({None, "year", "year-month", "days since 1970-01-01", "seconds since 1970-01-01"}) – See long_data() for a description of the options.
drop_all_nan_times (bool) – Should time points which contain only nan values be dropped? This operation is applied after any transforms introduced by the value of time_axis.

Returns

DataFrame with datetimes as columns and timeseries as rows. Metadata is in the index.

Return type

scmdata.run.ScmRun.to_nc()

Raises

NonUniqueMetadataError – If the metadata are not unique between timeseries and check_duplicated is True
NotImplementedError – The value of time_axis is not recognised
ValueError – The value of time_axis would result in columns which aren’t unique

to_csv(fname: str, **kwargs: Any) → None ¶

Write timeseries data to a csv file

Parameters: fname – Path to write the file into

to_iamdataframe() → None ¶

Convert to a LongDatetimeIamDataFrame instance.

LongDatetimeIamDataFrame is a subclass of pyam.IamDataFrame. We use LongDatetimeIamDataFrame to ensure all times can be handled, see docstring of LongDatetimeIamDataFrame for details.

Returns: LongDatetimeIamDataFrame instance containing the same data.
Return type: LongDatetimeIamDataFrame
Raises: ImportError – If pyam is not installed

to_nc(fname, dimensions=('region'), extras=(), **kwargs)¶

Write timeseries to disk as a netCDF4 file

Each unique variable will be written as a variable within the netCDF file. Choosing the dimensions and extras such that there are as few empty (or nan) values as possible will lead to the best compression on disk.

Parameters

fname (str) – Path to write the file into
dimensions (iterable of str) – Dimensions to include in the netCDF file. The time dimension is always included (if not provided it will be the last dimension). An additional dimension (specifically a co-ordinate in xarray terms), “_id”, will be included if extras is provided and any of the metadata in extras is not uniquely defined by dimensions. “_id” maps the timeseries in each variable to their relevant metadata.
extras (iterable of str) – Metadata columns to write as variables in the netCDF file (specifically as “non-dimension co-ordinates” in xarray terms, see xarray terminology for more details). Where possible, these non-dimension co-ordinates will use dimension co-ordinates as their own co-ordinates. However, if the metadata in extras is not defined by a single dimension in dimensions, then the extras co-ordinates will have dimensions of “_id”. This “_id” co-ordinate maps the values in the extras co-ordinates to each timeseries in the serialised dataset. Where “_id” is required, an extra “_id” dimension will also be added to dimensions.
kwargs – Passed through to xarray.Dataset.to_netcdf()

See also

to_xarray(dimensions=('region'), extras=(), unify_units=True)¶

Convert to a xarray.Dataset

Parameters

dimensions (iterable of str) –

Dimensions for each variable in the returned dataset. If an “_id” co-ordinate is
required (see extras documentation for when “_id” is required) and is not included in dimensions then it will be the last dimension (or second last dimension if “time” is also not included in dimensions). If “time” is not included in dimensions it will be the last dimension.
extras (iterable of str) –
Columns in self.meta from which to create “non-dimension co-ordinates” (see xarray terminology for more details). These non-dimension co-ordinates store extra information and can be mapped to each timeseries found in the data variables of the output xarray.Dataset. Where possible, these non-dimension co-ordinates will use dimension co-ordinates as their own co-ordinates. However, if the metadata in extras is not defined by a single dimension in dimensions, then the extras co-ordinates will have dimensions of “_id”. This “_id” co-ordinate maps the values in the extras co-ordinates to each timeseries in the serialised dataset. Where “_id” is required, an extra “_id” dimension will also be added to dimensions.
unify_units (bool) – If a given variable has multiple units, should we attempt to unify them?

Returns

Data in self, re-formatted as an xarray.Dataset

Return type

xarray.Dataset

Raises

ValueError – If a variable has multiple units and unify_units is False.
ValueError – If a variable has multiple units which are not able to be converted to a common unit because they have different base units.

property values: numpy.ndarray¶

Timeseries values without metadata

The values are returned such that each row is a different timeseries being a row and each column is a different time (although no time information is included as a plain numpy.ndarray is returned).

Returns: The array in the same shape as ScmRun.shape(), that is (num_timeseries, num_timesteps).
Return type: np.ndarray

scmdata.run.run_append(runs, inplace: bool = False, duplicate_msg: Union[str, bool] = True, metadata: Optional[Dict[str, Union[str, int, float]]] = None)[source]¶

Append together many objects.

When appending many objects, it may be more efficient to call this routine once with a list of ScmRun’s, than using ScmRun.append() multiple times.

If timeseries with duplicate metadata are found, the timeseries are appended and values falling on the same timestep are averaged if duplicate_msg is not “return”. If duplicate_msg is “return”, then the result will contain the duplicated timeseries for further inspection.

>>> res = base.append(other, duplicate_msg="return")
<scmdata.ScmRun (timeseries: 5, timepoints: 3)>
Time:
    Start: 2005-01-01T00:00:00
    End: 2015-06-12T00:00:00
Meta:
          scenario             variable  model climate_model region   unit
    0   a_scenario       Primary Energy  a_iam       a_model  World  EJ/yr
    1   a_scenario  Primary Energy|Coal  a_iam       a_model  World  EJ/yr
    2  a_scenario2       Primary Energy  a_iam       a_model  World  EJ/yr
    3  a_scenario3       Primary Energy  a_iam       a_model  World  EJ/yr
    4   a_scenario       Primary Energy  a_iam       a_model  World  EJ/yr
>>> ts = res.timeseries(check_duplicated=False)
>>> ts[ts.index.duplicated(keep=False)]
time                                                        2005-01-01  ...  2015-06-12
scenario   variable       model climate_model region unit               ...
a_scenario Primary Energy a_iam a_model       World  EJ/yr         1.0  ...         7.0
                                                     EJ/yr        -1.0  ...         1.0

Parameters

runs (list of ScmRun) – The runs to append. Values will be attempted to be cast to ScmRun.
inplace – If True, then the operation updates the first item in runs and returns None.
duplicate_msg – If True, raise a NonUniqueMetadataError error so the user can see the duplicate timeseries. If False, take the average and do not raise a warning or error. If "warn", raise a warning if duplicate data is detected.
metadata – If not None, override the metadata of the resulting ScmRun with metadata. Otherwise, the metadata for the runs are merged. In the case where there are duplicate metadata keys, the values from the first run are used.

Returns

If not inplace, the return value is the object containing the merged data. The resultant class will be determined by the type of the first object.

Return type