GAMS Transfer

GAMS Transfer is a package to maintain GAMS data outside a GAMS script in a programming language like Python or Matlab. It allows the user to add GAMS symbols (Sets, Aliases, Parameters, Variables and Equations), to manipulate GAMS symbols, as well as read/write symbols to different data endpoints. GAMS Transfer’s main focus is the highly efficient transfer of data between GAMS and the target programming language, while keeping those operations as simple as possible for the user. In order to achieve this, symbol records – the actual and potentially large-scale data sets – are stored in native data structures of the corresponding programming languages. The benefits of this approach are threefold: (1) The user is usually very familiar with these data structures, (2) these data structures come with a large tool box for various data operations, and (3) optimized methods for reading from and writing to GAMS can transfer the data as a bulk – resulting in the high performance of this package. This documentation describes, in detail, the use of GAMS Transfer within a Python environment.

Data within GAMS Transfer will be stored as Pandas DataFrame. The flexible nature of Pandas DataFrames makes them ideal for storing/manipulating sparse data. Pandas includes advanced operations for indexing and slicing, reshaping, merging and even visualization.

Pandas also includes a number of advanced data I/O tools that allow users to generate DataFrames directly from CSV (.csv), JSON (.json), HTML (.html), Microsoft Excel (.xls, .xlsx), SQL , pickle (.pkl), SPSS (.sav, .zsav), SAS (.xpt, .sas7bdat), etc.

Centering GAMS Transfer around the Pandas DataFrame gives GAMS users (on a variety of platforms – MacOS, Windows, Linux) access to tools to move data back and forth between their favorite environments for use in their GAMS models.

The goal of this documentation is to introduce the user to GAMS Transfer and its functionality. This documentation is not designed to teach the user how to effectively manipulate Pandas DataFrames; users seeking a deeper understanding of Pandas are referred to the extensive documentation.

# Install

The user must download and install the latest version of GAMS in order to install GAMS Transfer. GAMS Transfer is installed when the GAMS Python API is built and installed. The user is referred HERE for instructions on how to install the Python API files. GAMS Transfer and all GAMS Python API files are compatible with environment managers such as Anaconda.

# Design

Storing, manipulating, and transforming sparse data requires that it lives within an environment – this data can then be linked together to enable various operations. In GAMS Transfer we refer to this "environment" as the Container, it is the main repository for storing and linking our sparse data. Symbols can be added to the Container from a variety of GAMS starting points but they can also be generated directly within the Python environment using convenient function calls that are part of the GAMS Transfer package; a symbol can only belong to one container at a time.

The process of linking symbols together within a container was inspired by typical GAMS workflows but leverages aspects of object oriented programming to make linking data a natural process. Linking data enables data operations like implicit set growth, domain checking, data format transformations (to dense/sparse matrix formats), etc – all of these features are enabled by the use of ordered pandas.CategoricalDtype data types. All of these details will be discussed in the following sections.

## Naming Conventions

Methods – functions that operate on a object – are all verbs (i.e., getMaxAbsValue(), getUniverseSet(), etc.) and use camel case for identification purposes. Methods are, by convention, tools that "do things"; that is they involve some, potentially expensive, computations. Some GAMS Transfer methods accept arguments, while others are simply called using the () notation. Plural arguments (columns) hint that they can accept lists of inputs (i.e., a list of symbol names) while singular arguments (column) will only accept one input at a time.

Properties – inherent attributes of an object – are all nouns (i.e., name, number_records, etc.) and use snake case (lower case words separated by underscores) for identification purposes. Object properties (or "object attributes") are fundamental to the object and therefore they are not called like methods; object properties are simply accessed by other methods or user calls. By convention, properties only require trival amounts of computation to access.

Classes – the basic structure of an object – are all singular nouns and use camel case (starting with a capital first letter) for identification purposes.

# Quick Start

Reading in all symbols can be accomplished with one line of code (we reference data from the trnsport.gms example).

import gamstransfer as gt
m = gt.Container("trnsport.gdx")

All symbol data is organized in the data attribute (a Python dict) – m.data[<symbol_name>].records – records are stored as Pandas DataFrames.

## Write Symbol to CSV

Writing symbol records to a CSV can also be accomplished with one line.

m.data["x"].records.to_csv("x.csv")

## Write a New GDX

There are five symbol classes within GAMS Transfer: 1) Sets, 2) Parameters, 3) Variables, 4) Equations and 5) Aliases. For purposes of this quick start, we show how to recreate the distance data structure from the trnsport.gms model (the parameter d). This brief example shows how users can achieve "GAMS-like" functionality, but within a Python environment – GAMS Transfer leverages the object oriented programming to simplify syntax.

import gamstransfer as gt
import pandas as pd
m = gt.Container()
# create the sets i, j
i = gt.Set(m, "i", records=["seattle", "san-diego"], description="supply")
j = gt.Set(m, "j", records=["new-york", "chicago", "topeka"], description="markets")
# add "d" parameter -- domain linked to set objects i and j
d = gt.Parameter(m, "d", [i, j], description="distance in thousands of miles")
# create some data as a generic DataFrame
dist = pd.DataFrame(
[
("seattle", "new-york", 2.5),
("seattle", "chicago", 1.7),
("seattle", "topeka", 1.8),
("san-diego", "new-york", 2.5),
("san-diego", "chicago", 1.8),
("san-diego", "topeka", 1.4),
],
columns=["from", "to", "thousand_miles"],
)
# setRecords will automatically convert the dist DataFrame into a standard DataFrame format
d.setRecords(dist)
# write the GDX
m.write("out.gdx")

This example shows a few fundamental features of GAMS Transfer:

1. An empty Container is analogous to an empty GDX file
2. Symbols will always be linked to a Container (notice that we always pass the Container reference m to the symbol constructor)
3. Records can be added to a symbol with the setRecords() method or through the records constructor argument (internally calls setRecords()). GAMS Transfer will convert many common Python data structures into a standard format.
4. Domain linking is possible by passing domain set objects to other symbols – this will also enable domain checking (violations will show up as NaN)
5. Writing a GDX file can be accomplished in one line with the write() method.

This Quick Start introduced the reader to the GAMS Transfer syntax, but in the remaining sections we will present details about the core functionality and dig further into the syntax. Specifically, we will discuss:

1. How to create a Container
2. How to add symbols to a Container
3. How to validate the data that is in the Container
4. How to defined domains implicitly with domain_forwarding
5. How to describe the data that is in the Container
6. How to transform data that is in the Container
7. Understand what the universe set is (UEL list)
8. How to reorder symbols in the Container
9. How to remove symbols from the Container

# Create a Container

The main object class within GAMS Transfer is called Container. The Container is the vessel that allows symbols to be linked together (through their domain definitions), it enables implicit set definitions, it enables structural manipulations of the data (matrix generation), and it allows the user to perform different read/write operations.

## Container constructor

Argument Type Description Required Default
load_from str, GMD Object Handle, GamsDatabase Object, ConstContainer Points to the source of the data being read into the Container No None
system_directory str Absolute path to GAMS system directory No Attempts to find the GAMS installation by creating a GamsWorkspace object and loading the system_directory attribute.

Creating a Container is a simple matter of initializing an object. For example:

import gamstransfer as gt
m = gt.Container()

This new Container object, here called m, contains a number of convenient properties and methods that allow the user to interact with the symbols that are in the Container. Some of these methods are used to filter out different types of symbols, other methods are used to numerically characterize the data within each symbol.

## Container properties

Property Description Type Special Setter Behavior
data main dict that is used to store all symbol data dict -

Symbols are organized in the Container under the data Container attribute. The dot notation (m.data) is used to access the underlying dict; symbols in this dict can then be retrieved with the standard bracket notation (m.data[<symbol_name>]).

In [1]: m.data
Out[1]:
{'i': <src.gamstransfer.Set at 0x7fa2387750a0>,
'j': <src.gamstransfer.Set at 0x7fa238e74fa0>,
'a': <src.gamstransfer.Parameter at 0x7fa238e74cd0>,
'b': <src.gamstransfer.Parameter at 0x7fa238e746a0>,
'd': <src.gamstransfer.Parameter at 0x7fa23876b370>,
'f': <src.gamstransfer.Parameter at 0x7fa23876b400>,
'c': <src.gamstransfer.Parameter at 0x7fa23876b5e0>,
'x': <src.gamstransfer.Variable at 0x7fa23876b340>,
'z': <src.gamstransfer.Variable at 0x7fa23876b640>,
'cost': <src.gamstransfer.Equation at 0x7fa23876b2b0>,
'supply': <src.gamstransfer.Equation at 0x7fa23876b310>,
'demand': <src.gamstransfer.Equation at 0x7fa23876b460>}

## Container methods

Method Description Arguments/Defaults Returns
addAlias Container method to add an Alias name (str)
alias_with (Set,Alias)
Alias object
addEquation Container method to add an Equation name (str)
type (str)
domain=[] (str,list)
records=None (pandas.DataFrame,numpy.ndarry,None)
domain_forwarding=False (bool)
description="" (str)
Equation object
addParameter Container method to add a Parameter name (str)
domain=None (str,list,None)
records=None (pandas.DataFrame,numpy.ndarry,None)
domain_forwarding=False (bool)
description="" (str)
Parameter object
addSet Container method to add a Set name (str)
domain=None (str,list,None)
is_singleton=False (bool)
records=None (pandas.DataFrame,numpy.ndarry,None)
domain_forwarding=False (bool)
description="" (str)
Set object
addVariable Container method to add an Variable name (str)
type="free" (str)
domain=[] (str,list)
records=None (pandas.DataFrame,numpy.ndarry,None)
domain_forwarding=False (bool)
description="" (str)
Variable object
describeAliases create a summary table with descriptive statistics for Aliases symbols=None (None,str,list) - if None, assumes all aliases pandas.DataFrame
describeParameters create a summary table with descriptive statistics for Parameters symbols=None (None,str,list) - if None, assumes all parameters pandas.DataFrame
describEquations create a summary table with descriptive statistics for Equations symbols=None (None,str,list) - if None, assumes all equations pandas.DataFrame
describeSets create a summary table with descriptive statistics for Sets symbols=None (None,str,list) - if None, assumes all sets pandas.DataFrame
describeVariables create a summary table with descriptive statistics for Variables symbols=None (None,str,list) - if None, assumes all variables pandas.DataFrame
getSymbols returns a list of object refernces for symbols symbols (str,list) list
getUniverseSet provides a universe for all symbols, the symbols argument allows GAMS Transfer to create a partial universe if writing only a subset of symbols (currently only supported when writing to GamsDatabases or GMD Objects) symbols=None (None,str,list) list
isValid True if all symbols in the Container are valid - bool
listAliases list all aliases (is_valid=None), list all valid aliases (is_valid=True), list all invalid aliases (is_valid=False) in the container is_valid=None (bool,None) list
listEquations list all equations (is_valid=None), list all valid equations (is_valid=True), list all invalid equations (is_valid=False) in the container is_valid=None (bool,None)
types=None (list of equation types) - if None, assumes all types
list
listParameters list all parameters (is_valid=None), list all valid parameters (is_valid=True), list all invalid parameters (is_valid=False) in the container is_valid=None (bool,None) list
listSets list all sets (is_valid=None), list all valid sets (is_valid=True), list all invalid sets (is_valid=False) in the container is_valid=None (bool,None) list
listSymbols list all symbols (is_valid=None), list all valid symbols (is_valid=True), list all invalid symbols (is_valid=False) in the container is_valid=None (bool,None) list
listVariables list all variables (is_valid=None), list all valid variables (is_valid=True), list all invalid variables (is_valid=False) in the container is_valid=None (bool,None)
types=None (list of variable types) - if None, assumes all types
list
read main method to read load_from, can be provided with a list of symbols to read in subsets, records controls if symbol records are loaded or just metadata load_from (str,GMD Object Handle,GamsDatabase Object,ConstContainer)
symbols="all" (str, list)
records=True (bool)
None
removeSymbols symbols to remove from the Container, also sets the symbols ref_container to None symbols (str,list) None
renameSymbol rename a symbol in the Container old_name (str), new_name (str) None
reorderSymbols reorder symbols in order to avoid domain violations - None
write main bulk write method to a write_to target write_to (str,GamsDatabase,GMD Object)
write_symbols=None (None,str,list) - if None, assumes all symbols
compress=False (bool)
uel_priority=None (str,list)
merge_symbols=None (None,str,list)
None

# Create a Set

There are two different ways to create a GAMS set and add it to a Container.

1. Use Set constructor
2. Use the Container method addSet (which internally calls the Set constructor)

## Set Constructor

Argument Type Description Required Default
container Container A reference to the Container object that the symbol is being added to Yes -
name str Name of symbol Yes -
domain list List of domains given either as string ('*' for universe set) or as reference to a Set object No ["*"]
is_singleton bool Indicates if set is a singleton set (True) or not (False) No False
records many Symbol records No None
domain_forwarding bool Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) No False
description str Description of symbol No ""

## Set Properties

Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
is_singleton bool if symbol is a singleton set bool -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
ref_container reference to the Container that the symbol belongs to Container -
summary output a dict of only the metadata dict -

## Set Methods

Method Description Arguments/Defaults Returns
getCardinality get the full cartesian product of the domain - int
getSparsity get the sparsity of the symbol w.r.t the cardinality - float
findDomainViolations get the index of records that contain any domain violations - pandas.Index
isValid checks if the symbol is in a valid format, throw exceptions if verbose=True, recheck a symbol if force=True verbose=False
force=True
bool
setRecords main convenience method to set standard pandas.DataFrame formatted records records (many types) None

Three possibilities exist to assign symbol records to a set (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

If the data is in a convenient format, a user may want to pass the records directly within the set constructor. This is an optional keyword argument and internally the set constructor will simply call the setRecords method. The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format (see GAMS Transfer Standard Data Formats). Many native python data types can be easily transformed into DataFrames, so the setRecords method for Set objects will accept a number of different types for input. The setRecords method is called internally on any data structure that is passed through the records argument. We show a few examples of ways to create differently structured sets:

Example #1 - Create a 1D set from a list
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"])
# NOTE: the above syntax is equivalent to -
# i = gt.Set(m, "i")
# i.setRecords(["seattle", "san-diego"])
# NOTE: the above syntax is also equivalent to -
# NOTE: the above syntax is also equivalent to -
# i.setRecords(["seattle", "san-diego"])
# NOTE: the above syntax is also equivalent to -
# m.data["i"].setRecords(["seattle", "san-diego"])
In [1]: i.records
Out[1]:
uni_0 element_text
0 seattle
1 san-diego
Example #2 - Create a 1D set from a tuple
import gamstransfer as gt
m = gt.Container()
j = gt.Set(m, "j", records=("seattle", "san-diego"))
# NOTE: the above syntax is equivalent to -
# j = gt.Set(m, "j")
# j.setRecords(("seattle", "san-diego"))
# NOTE: the above syntax is also equivalent to -
# NOTE: the above syntax is also equivalent to -
# j.setRecords(("seattle", "san-diego"))
# NOTE: the above syntax is also equivalent to -
# m.data["j"].setRecords(("seattle", "san-diego"))
In [1]: j.records
Out[1]:
uni_0 element_text
0 seattle
1 san-diego
Example #3 - Create a 2D set from a list of tuples
import gamstransfer as gt
m = gt.Container()
k = gt.Set(m, "k", ["*", "*"], records=[("seattle", "san-diego")])
# NOTE: the above syntax is equivalent to -
# k = gt.Set(m, "k", ["*", "*"])
# k.setRecords([("seattle", "san-diego")])
# NOTE: the above syntax is also equivalent to -
# NOTE: the above syntax is also equivalent to -
# k.setRecords([("seattle", "san-diego")])
# NOTE: the above syntax is also equivalent to -
# m.data["k"].setRecords([("seattle", "san-diego")])
In [1]: k.records
Out[1]:
uni_0 uni_1 element_text
0 seattle san-diego
Example #4 - Create a 1D set from a DataFrame slice + .unique()
import gamstransfer as gt
m = gt.Container()
# note that the raw data is convenient to hold in a DataFrame
dist = pd.DataFrame(
[
("seattle", "new-york", 2.5),
("seattle", "chicago", 1.7),
("seattle", "topeka", 1.8),
("san-diego", "new-york", 2.5),
("san-diego", "chicago", 1.8),
("san-diego", "topeka", 1.4),
],
columns=["from", "to", "thousand_miles"],
)
l = gt.Set(m, "l", records=dist["from"].unique())
# NOTE: the above syntax is equivalent to -
# l = gt.Set(m, "l")
# l.setRecords(dist["from"].unique())
# NOTE: the above syntax is also equivalent to -
# NOTE: the above syntax is also equivalent to -
# l.setRecords(dist["from"].unique())
# NOTE: the above syntax is also equivalent to -
# m.data["l"].setRecords(dist["from"].unique())
In [1]: l.records
Out[1]:
uni_0 element_text
0 seattle
1 san-diego
Note
The .unique() method preserves the order of appearance, unlike set().

Set element text is very handy when labeling specific set elements within a set. A user can add a set element text directly with a set element. Note that it is not required to label all set elements, as can be seen in the following example.

Example #5 - Add set element text
import gamstransfer as gt
m = gt.Container()
i = gt.Set(
m,
"i",
records=[
("seattle", "home of sub pop records"),
("san-diego",),
("washington_dc", "former gams hq"),
],
)
# NOTE: the above syntax is equivalent to -
#
# i = gt.Set(m, "i")
# i_recs = [
# ("seattle", "home of sub pop records"),
# ("san-diego",),
# ("washington_dc", "former gams hq"),
# ]
#
# i.setRecords(i_recs)
# NOTE: the above syntax is also equivalent to -
# NOTE: the above syntax is also equivalent to -
# i.setRecords(i_recs)
# NOTE: the above syntax is also equivalent to -
# m.data["i"].setRecords(i_recs)
In [1]: i.records
Out[1]:
uni_0 element_text
0 seattle home of sub pop records
1 san-diego
2 washington_dc former gams hq

## Directly Set Records

The primary advantage of the setRecords method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Directly set records (1D set)
import gamstransfer as gt
import pandas as pd
m = gt.Container()
i = gt.Set(m, "i", description="supply")
# create a standard format dataframe
df_i = pd.DataFrame(
data=[("seattle", ""), ("san-diego", "")], columns=["uni_0", "element_text"]
)
# need to create categorical column type, referencing elements already in df_i
df_i["uni_0"] = df_i["uni_0"].astype(
pd.CategoricalDtype(categories=df_i["uni_0"].unique(), ordered=True)
)
# set the records directly
i.records = df_i
In [1]: i.isValid()
Out[1]: True

Stepping through this example we take the following steps:

1. Create an empty Container
2. Create a GAMS set i in the Container, but do not set the records
3. Create a Pandas DataFrame (manually, in this example) taking care to follow the standard format
4. The DataFrame has the right shape and column labels so we can proceed to set the records.
5. We need to cast the uni_0 column as a categorical data type, so we create a custom ordered categorty type using pandas.CategoricalDtype
6. Finally, we set the records directly by passing a reference to df_i into the symbol records attribute. The setter function of .records checks that a DataFrame is being set, but does not check validity. Thus, as a final step we call the .isValid() method to verify that the symbol is valid.
Attention
Users can debug their DataFrames by running <symbol_name>.isValid(verbose=True) to get feedback about their data.
Example #2 - Directly set records (1D subset)
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"], description="supply")
j = gt.Set(m, "j", i, description="supply")
# create a standard format dataframe
df_j = pd.DataFrame(data=[("seattle", "")], columns=["i_0", "element_text"])
# create the categorical column type
df_j["i_0"] = df_j["i_0"].astype(i.records["uni_0"].dtype)
# set the records
j.records = df_j
In [1]: j.isValid()
Out[1]: True

This example is more subtle in that we want to create a set j that is a subset of i. We create the set i using the setRecords method but then set the records directly for j. There are two important details to note: 1) the column labels in df_j now reflect the standard format for a symbol with a domain set (as opposed to the universe) and 2) we create the categorical dtype by referencing the parent set (i) for the categories (instead of referencing itself).

# Create a Parameter

There are two different ways to create a GAMS parameter and add it to a Container.

1. Use Parameter constructor
2. Use the Container method addParameter (which internally calls the Parameter constructor)

## Parameter Constructor

Parameter constructor
Argument Type Description Required Default
container Container A reference to the Container object that the symbol is being added to Yes -
name str Name of symbol Yes -
domain list List of domains given either as string ('*' for universe set) or as reference to a Set object, an empty domain list will create a scalar parameter No []
records many Symbol records No None
domain_forwarding bool Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) No False
description str Description of symbol No ""

## Parameter Properties

Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
is_scalar True if the len(self.domain) = 0 bool -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
ref_container reference to the Container that the symbol belongs to Container -
shape a tuple describing the array dimensions if records were converted with .toDense() tuple -
summary output a dict of only the metadata dict -

## Parameter Methods

Method Description Arguments/Defaults Returns
countEps total number of SpecialValues.EPS in value column - int
countNA total number of SpecialValues.NA in value column - int
countNegInf total number of SpecialValues.NEGINF in value column - int
countPosInf total number of SpecialValues.POSINF in value column - int
countUndef total number of SpecialValues.UNDEF in value column - int
findDomainViolations get the index of records that contain any domain violations - pandas.Index
findEps find index positions of SpecialValues.EPS in value column - pandas.Index
findNA find index positions of SpecialValues.NA in value column - pandas.Index
findNegInf find index positions of SpecialValues.NEGINF in value column - pandas.Index
findPosInf find index positions of SpecialValues.POSINF in value column - pandas.Index
findUndef find index positions of SpecialValues.Undef in value column - pandas.Index
getCardinality get the full cartesian product of the domain - int
getSparsity get the sparsity of the symbol w.r.t the cardinality - float
getMaxValue get the maximum value in value column - float
getMinValue get the minimum value in value column - float
getMeanValue get the mean value in value column - float
getMaxAbsValue get the maximum absolute value in value column - float
isValid checks if the symbol is in a valid format, throw exceptions if verbose=True, recheck a symbol if force=True verbose=False
force=True
bool
setRecords main convenience method to set standard pandas.DataFrame records records (many types) None
toDense convert symbol to a dense numpy.array format - numpy.array
toSparseCoo convert symbol to a sparse COOrdinate numpy.array format - sparse matrix format
whereMax find the domain entry of records with a maximum value (return first instance only) - list of str
whereMaxAbs find the domain entry of records with a maximum absolute value (return first instance only) - list of str
whereMin find the domain entry of records with a minimum value (return first instance only) - list of str

Three possibilities exist to assign symbol records to a parameter (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

If the data is in a convenient format, a user may want to pass the records directly within the parameter constructor. This is an optional keyword argument and internally the parameter constructor will simply call the setRecords method. The symbol method setRecords is a convenience method that transforms the given data into an approved Pandas DataFrame format (see GAMS Transfer Standard Data Formats). Many native python data types can be easily transformed into DataFrames, so the setRecords method for Set objects will accept a number of different types for input. The setRecords method is called internally on any data structure that is passed through the records argument. We show a few examples of ways to create differently structured parameters:

Example #1 - Create a GAMS scalar
import gamstransfer as gt
m = gt.Container()
pi = gt.Parameter(m, "pi", records=3.14159)
# NOTE: the above syntax is equivalent to -
# pi = gt.Parameter(m, "pi")
# pi.setRecords(3.14159)
# NOTE: the above syntax is also equivalent to -
# NOTE: the above syntax is also equivalent to -
# pi.setRecords(3.14159)
# NOTE: the above syntax is also equivalent to -
# m.data["pi"].setRecords(3.14159)
In [14]: pi.records
Out[14]:
value
0 3.14159
Note
GAMS Transfer will still convert scalar values to a standard format (i.e., a Pandas DataFrame with a single row and column).
Example #2 - Create a 1D parameter (defined over *) from a list of tuples
import gamstransfer as gt
m = gt.Container()
i = gt.Parameter(m, "i", ["*"], records=[("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is equivalent to -
# i = gt.Parameter(m, "i")
# i.setRecords([("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is also equivalent to -
# m.addParameter("i", records=[("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is also equivalent to -
# i.setRecords([("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is also equivalent to -
# m.data["i"].setRecords([("i" + str(i), i) for i in range(5)])
In [1]: i.records
Out[1]:
uni_0 value
0 i0 0.0
1 i1 1.0
2 i2 2.0
3 i3 3.0
4 i4 4.0
Example #3 - Create a 1D parameter (defined over a set) from a list of tuples
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=["i" + str(i) for i in range(5)])
a = gt.Parameter(m, "a", i, records=[("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is equivalent to -
# i = gt.Set(m, "i")
# i.setRecords(["i" + str(i) for i in range(5)])
# a = gt.Parameter(m, "a", i)
# a.setRecords([("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is also equivalent to -
# m.addSet("i", records=["i" + str(i) for i in range(5)])
# m.addParameter("a", i, records=[("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is also equivalent to -
# i.setRecords(["i" + str(i) for i in range(5)])
# a.setRecords([("i" + str(i), i) for i in range(5)])
# NOTE: the above syntax is also equivalent to -
# m.data["i"].setRecords(["i" + str(i) for i in range(5)])
# m.data["a"].setRecords([("i" + str(i), i) for i in range(5)])
In [1]: a.records
Out[1]:
i_0 value
0 i0 0.0
1 i1 1.0
2 i2 2.0
3 i3 3.0
4 i4 4.0
Example #4 - Create a 2D parameter (defined over a set) from a DataFrame slice
import gamstransfer as gt
import pandas as pd
dist = pd.DataFrame(
[
("seattle", "new-york", 2.5),
("seattle", "chicago", 1.7),
("seattle", "topeka", 1.8),
("san-diego", "new-york", 2.5),
("san-diego", "chicago", 1.8),
("san-diego", "topeka", 1.4),
],
columns=["from", "to", "thousand_miles"],
)
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=dist["from"].unique())
j = gt.Set(m, "j", ["*"], records=dist["to"].unique())
a = gt.Parameter(m, "a", [i, j], records=dist.loc[[0, 3], :])
# NOTE: the above syntax is equivalent to -
# i = gt.Set(m, "i")
# i.setRecords(dist["from"].unique())
# j = gt.Set(m, "j")
# j.setRecords(dist["to"].unique())
# a = gt.Parameter(m, "a", [i, j])
# a.setRecords(dist.loc[[0, 3], :])
# NOTE: the above syntax is also equivalent to -
# m.addParameter("a", i, records=dist.loc[[0, 3], :])
In [1]: a.records
Out[1]:
i_0 j_1 value
0 seattle new-york 2.5
3 san-diego new-york 2.5
Note
The original indexing is preserved when a user slices rows out of a reference dataframe.
Example #5 - Create a 2D parameter (defined over a set) from a matrix
import gamstransfer as gt
import pandas as pd
dist = pd.DataFrame(
[
("seattle", "new-york", 2.5),
("seattle", "chicago", 1.7),
("seattle", "topeka", 1.8),
("san-diego", "new-york", 2.5),
("san-diego", "chicago", 1.8),
("san-diego", "topeka", 1.4),
],
columns=["from", "to", "thousand_miles"],
)
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=dist["from"].unique())
j = gt.Set(m, "j", ["*"], records=dist["to"].unique())
a = gt.Parameter(m, "a", [i, j], records=dist)
In [1]: a.toDense()
Out[1]:
array([[2.5, 1.7, 1.8],
[2.5, 1.8, 1.4]])
# use a.toDense() to create a new (and identicial) parameter a2
a2 = gt.Parameter(m, "a2", [i, j], records=a.toDense())
# check that a.records is identical to a2.records
In [1]: a.records.equals(a2.records)
Out[1]: True
Example #6 - Create a 2D parameter from an array using setRecords
import gamstransfer as gt
ipmort numpy as np
import pandas as pd
m = gt.Container()
i = gt.Set(m, "i", records=["i" + str(i) for i in range(5)])
j = gt.Set(m, "j", records=["j" + str(j) for j in range(5)])
# create the parameter with linked domains (these will control the .shape of the symbol)
a = gt.Parameter(m, "a", [i, j])
# here we use the .shape property to easily generate a dense random array in numpy
a.setRecords(np.random.uniform(low=1, high=10, size=a.shape))
In [1]: a.toDense()
Out[1]:
array([[3.6694495 , 5.17395381, 1.99129484, 3.28315433, 1.44793791],
[1.06953243, 6.56331121, 5.26162554, 5.98098795, 8.30006 ],
[3.77213221, 5.82144901, 9.30035479, 9.12534285, 8.51970747],
[8.47965504, 7.84426304, 5.2442471 , 6.96666622, 6.55194415],
[5.62682779, 4.92509183, 8.94579609, 2.7724934 , 9.99576081]])

## Directly Set Records

As with sets, the primary advantage of the setRecords method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Correctly set records (directly)
import gamstransfer as gt
import pandas as pd
import numpy as np
df = pd.DataFrame(
data=[
("h" + str(h), "m" + str(m), "s" + str(s))
for h in range(8760)
for m in range(60)
for s in range(60)
],
columns=["h_0", "m_1", "s_2"],
)
df["value"] = np.random.uniform(0, 100, len(df))
m = gt.Container()
hrs = gt.Set(m, "h", records=df["h_0"].unique())
mins = gt.Set(m, "m", records=df["m_1"].unique())
secs = gt.Set(m, "s", records=df["s_2"].unique())
df["h_0"] = df["h_0"].astype(hrs.records["uni_0"].dtype)
df["m_1"] = df["m_1"].astype(mins.records["uni_0"].dtype)
df["s_2"] = df["s_2"].astype(secs.records["uni_0"].dtype)
a = gt.Parameter(m, "a", [hrs, mins, secs])
# set records
a.records = df
In [1]: a.isValid()
Out[1]: True

In this example we create a large parameter (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to a parameter with a.records. GAMS Transfer requires that all domain columns must be a categorical data type, furthermore, this categorical must be ordered. The records setter function does very little work other than checking if the object being set is a DataFrame. This places more responsibility on the user to create a DataFrame that complies with the standard format. In Example #1 we take care to properly reference the categorical data types from the domain sets – and in the end a.isValid() = True.

Users will need to use the .isValid(verbose=True) method to debug any structural issues. As an example we incorrectly generate categorical data types by passing the DataFrame constructor the generic dtype="category" argument. This creates categorical column types but they are not ordered and they do not reference the underlying domain set. These errors result in a being invalid.

Example #2 - Incorrectly set records (directly)
import gamstransfer as gt
import pandas as pd
import numpy as np
df = pd.DataFrame(
data=[
("h" + str(h), "m" + str(m), "s" + str(s))
for h in range(8760)
for m in range(60)
for s in range(60)
],
columns=["h_0", "m_1", "s_2"],
dtype="category"
)
df["value"] = np.random.uniform(0, 100, len(df))
m = gt.Container()
hrs = gt.Set(m, "h", records=df["h_0"].unique())
mins = gt.Set(m, "m", records=df["m_1"].unique())
secs = gt.Set(m, "s", records=df["s_2"].unique())
a = gt.Parameter(m, "a", [hrs, mins, secs])
# set the records directly
a.records = df
In [1]: a.isValid()
Out[1]: False
In [2]: a.isValid(verbose=True)
Out[2]: Exception: Domain information in column 'h_0' for 'records' must be an ORDERED categorical type (i.e., <symbol_object>.records[h_0].dtype.ordered = True)

# Create a Variable

There are two different ways to create a GAMS variable and add it to a Container.

1. Use Variable constructor
2. Use the Container method addVariable (which internally calls the Variable constructor)

## Variable Constructor

Variable constructor
Argument Type Description Required Default
container Container A reference to the Container object that the symbol is being added to Yes -
name str Name of symbol Yes -
type str Type of variable being created [binary, integer, positive, negative, free, sos1, sos2, semicont, semiint] No free
domain list List of domains given either as string (* for universe set) or as reference to a Set object, an empty domain list will create a scalar variable No []
records many Symbol records No None
domain_forwarding bool Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) No False
description str Description of symbol No ""

## Variable Properties

Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
ref_container reference to the Container that the symbol belongs to Container -
shape a tuple describing the array dimensions if records were converted with .toDense() tuple -
summary output a dict of only the metadata dict -
type str type of variable dict -

## Variable Methods

Method Description Arguments/Defaults Returns
countEps total number of SpecialValues.EPS across all columns columns="level" (str,list) int
countNA total number of SpecialValues.NA across all columns columns="level" (str,list) int
countNegInf total number of SpecialValues.NEGINF across all columns columns="level" (str,list) int
countPosInf total number of SpecialValues.POSINF across all columns columns="level" (str,list) int
countUndef total number of SpecialValues.UNDEF across all columns columns="level" (str,list) int
findDomainViolations get the index of records that contain any domain violations - pandas.Index
findEps find index positions of SpecialValues.EPS in column column="level" (str) pandas.Index
findNA find index positions of SpecialValues.NA in column column="level" (str) pandas.Index
findNegInf find index positions of SpecialValues.NEGINF in column column="level" (str) pandas.Index
findPosInf find index positions of SpecialValues.POSINF in column column="level" (str) pandas.Index
findUndef find index positions of SpecialValues.Undef in column column="level" (str) pandas.Index
getCardinality get the full cartesian product of the domain - int
getSparsity get the sparsity of the symbol w.r.t the cardinality - float
getMaxValue get the maximum value across all columns columns="level" (str,list) float
getMinValue get the minimum value across all columns columns="level" (str,list) float
getMeanValue get the mean value across all columns columns="level" (str,list) float
getMaxAbsValue get the maximum absolute value across all columns columns="level" (str,list) float
isValid checks if the symbol is in a valid format, throw exceptions if verbose=True, recheck a symbol if force=True verbose=False
force=True
bool
setRecords main convenience method to set standard pandas.DataFrame records records (many types) None
toDense convert column to a dense numpy.array format column="level" (str) numpy.array
toSparseCoo convert column to a sparse COOrdinate numpy.array format column="level" (str) sparse matrix format
whereMax find the domain entry of records with a maximum value (return first instance only) column="level" (str) list of str
whereMaxAbs find the domain entry of records with a maximum absolute value (return first instance only) column="level" (str) list of str
whereMin find the domain entry of records with a minimum value (return first instance only) column="level" (str) list of str

Three possibilities exist to assign symbol records to a variable (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

If the data is in a convenient format, a user may want to pass the records directly within the variable constructor. This is an optional keyword argument and internally the variable constructor will simply call the setRecords method. In contrast to the setRecords methods in in either the Set or Parameter classes the setRecords method for variables will only accept Pandas DataFrames and specially structured dict for creating records from matrices. This restriction is out of necessity because to properly set a record for a Variable the user must pass data for the level, marginal, lower, upper and scale attributes. That said, any missing attributes will be filled in with the GAMS default record values (see: Variable Types), default scale value is always 1, and the default level and marginal values are 0 for all variable types). We show a few examples of ways to create differently structured variables:

Example #1 - Create a GAMS scalar variable
import gamstransfer as gt
m = gt.Container()
pi = gt.Variable(m, "pi", records=pd.DataFrame(data=[3.14159], columns=["level"]))
# NOTE: the above syntax is equivalent to -
# pi = gt.Variable(m, "pi", "free")
# pi.setRecords(pd.DataFrame(data=[3.14159], columns=["level"]))
# NOTE: the above syntax is also equivalent to -
In [1]: pi.records
Out[1]:
level marginal lower upper scale
0 3.14159 0.0 -inf inf 1.0
Example #2 - Create a 1D variable (defined over *) from a list of tuples

In this example we only set the marginal values.

import gamstransfer as gt
m = gt.Container()
v = gt.Variable(
m,
"v",
"free",
domain=["*"],
records=pd.DataFrame(
data=[("i" + str(i), i) for i in range(5)], columns=["domain", "marginal"]
),
)
In [1]: v.records
Out[1]:
uni_0 level marginal lower upper scale
0 i0 0.0 0.0 -inf inf 1.0
1 i1 0.0 1.0 -inf inf 1.0
2 i2 0.0 2.0 -inf inf 1.0
3 i3 0.0 3.0 -inf inf 1.0
4 i4 0.0 4.0 -inf inf 1.0
Example #3 - Create a 1D variable (defined over a set) from a list of tuples
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=["i" + str(i) for i in range(5)])
v = gt.Variable(
m,
"v",
"free",
domain=i,
records=pd.DataFrame(
data=[("i" + str(i), i) for i in range(5)], columns=["domain", "marginal"]
),
)
In [1]: v.records
Out[1]:
i_0 level marginal lower upper scale
0 i0 0.0 0.0 -inf inf 1.0
1 i1 0.0 1.0 -inf inf 1.0
2 i2 0.0 2.0 -inf inf 1.0
3 i3 0.0 3.0 -inf inf 1.0
4 i4 0.0 4.0 -inf inf 1.0
Example #4 - Create a 2D positive variable, specifying no numerical data
import gamstransfer as gt
import pandas as pd
m = gt.Container()
v = gt.Variable(
m,
"v",
"positive",
["*", "*"],
)
In [1]: v.records
Out[1]:
uni_0 uni_1 level marginal lower upper scale
0 seattle san-diego 0.0 0.0 0.0 inf 1.0
1 chicago madison 0.0 0.0 0.0 inf 1.0
Example #5 - Create a 2D variable (defined over a set) from a matrix
import gamstransfer as gt
import pandas as pd
import numpy as np
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=["i" + str(i) for i in range(5)])
j = gt.Set(m, "j", ["*"], records=["j" + str(i) for i in range(5)])
a = gt.Parameter(
m,
"a",
[i, j],
records=[("i" + str(i), "j" + str(j), i + j) for i in range(5) for j in range(5)],
)
# create a free variable and set the level and marginal attributes from matricies
v = gt.Variable(
m, "v", domain=[i, j], records={"level": a.toDense(), "marginal": a.toDense()}
)
# if not specified, the toDense() method will convert the level values to a matrix
In [1]: v.toDense()
Out[1]:
array([[0., 1., 2., 3., 4.],
[1., 2., 3., 4., 5.],
[2., 3., 4., 5., 6.],
[3., 4., 5., 6., 7.],
[4., 5., 6., 7., 8.]])

## Directly Set Records

As with sets, the primary advantage of the setRecords method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Correctly set records (directly)
import gamstransfer as gt
import pandas as pd
import numpy as np
df = pd.DataFrame(
data=[
("h" + str(h), "m" + str(m), "s" + str(s))
for h in range(8760)
for m in range(60)
for s in range(60)
],
columns=["h_0", "m_1", "s_2"],
)
# it is necessary to specify all variable attributes if setting records directly
# NOTE: all numeric data must be type float
df["level"] = np.random.uniform(0, 100, len(df))
df["marginal"] = 0.0
df["lower"] = gt.SpecialValues.NEGINF
df["upper"] = gt.SpecialValues.POSINF
df["scale"] = 1.0
m = gt.Container()
hrs = gt.Set(m, "h", records=df["h_0"].unique())
mins = gt.Set(m, "m", records=df["m_1"].unique())
secs = gt.Set(m, "s", records=df["s_2"].unique())
df["h_0"] = df["h_0"].astype(hrs.records["uni_0"].dtype)
df["m_1"] = df["m_1"].astype(mins.records["uni_0"].dtype)
df["s_2"] = df["s_2"].astype(secs.records["uni_0"].dtype)
a = gt.Variable(m, "a", domain=[hrs, mins, secs])
# set records
a.records = df
In [1]: a.isValid()
Out[1]: True
Attention
All numeric data in the records will need to be type float in order to maintain a valid symbol.

In this example we create a large variable (31,536,000 records and 8880 unique domain elements – we mimic data that is labeled for every second in one year) and assign it to a variable with a.records. GAMS Transfer requires that all domain columns must be a categorical data type, furthermore this categorical must be ordered. The records setter function does very little work other than checking if the object being set is a DataFrame. This places more responsibility on the user to create a DataFrame that complies with the standard format. In Example #1 we take care to properly reference the categorical data types from the domain sets – and in the end a.isValid() = True. As with Set and Parameters, users can use the .isValid(verbose=True) method to debug any structural issues.

# Create an Equation

There are two different ways to create a GAMS equation and add it to a Container.

1. Use Equation constructor
2. Use the Container method addEquation (which internally calls the Equation constructor)

## Equation Constructor

Equation constructor
Argument Type Description Required Default
container Container A reference to the Container object that the symbol is being added to Yes -
name str Name of symbol Yes -
type str Type of equation being created [eq (or E/e), geq (or G/g), leq (or L/l), nonbinding (or N/n), external (or X/x)] Yes -
domain list List of domains given either as string (* for universe set) or as reference to a Set/Alias object, an empty domain list will create a scalar equation No []
records many Symbol records No None
domain_forwarding bool Flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) No False
description str Description of symbol No ""

## Equation Properties

Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
ref_container reference to the Container that the symbol belongs to Container -
shape a tuple describing the array dimensions if records were converted with .toDense() tuple -
summary output a dict of only the metadata dict -
type str type of variable dict -

## Equation Methods

Method Description Arguments/Defaults Returns
countEps total number of SpecialValues.EPS across all columns columns="level" (str,list) int
countNA total number of SpecialValues.NA across all columns columns="level" (str,list) int
countNegInf total number of SpecialValues.NEGINF across all columns columns="level" (str,list) int
countPosInf total number of SpecialValues.POSINF across all columns columns="level" (str,list) int
countUndef total number of SpecialValues.UNDEF across all columns columns="level" (str,list) int
findDomainViolations get the index of records that contain any domain violations - pandas.Index
findEps find index positions of SpecialValues.EPS in column column="level" (str) pandas.Index
findNA find index positions of SpecialValues.NA in column column="level" (str) pandas.Index
findNegInf find index positions of SpecialValues.NEGINF in column column="level" (str) pandas.Index
findPosInf find index positions of SpecialValues.POSINF in column column="level" (str) pandas.Index
findUndef find index positions of SpecialValues.Undef in column column="level" (str) pandas.Index
getCardinality get the full cartesian product of the domain - int
getSparsity get the sparsity of the symbol w.r.t the cardinality - float
getMaxValue get the maximum value across all columns columns="level" (str,list) float
getMinValue get the minimum value across all columns columns="level" (str,list) float
getMeanValue get the mean value across all columns columns="level" (str,list) float
getMaxAbsValue get the maximum absolute value across all columns columns="level" (str,list) float
isValid checks if the symbol is in a valid format, throw exceptions if verbose=True, recheck a symbol if force=True verbose=False
force=True
bool
setRecords main convenience method to set standard pandas.DataFrame records records (many types) None
toDense convert column to a dense numpy.array format column="level" (str) numpy.array
toSparseCoo convert column to a sparse COOrdinate numpy.array format column="level" (str) sparse matrix format
whereMax find the domain entry of records with a maximum value (return first instance only) column="level" (str) list of str
whereMaxAbs find the domain entry of records with a maximum absolute value (return first instance only) column="level" (str) list of str
whereMin find the domain entry of records with a minimum value (return first instance only) column="level" (str) list of str

Adding equation records mimics that of variables – three possibilities exist to assign symbol records to an equation (roughly ordered in complexity):

1. Setting the argument records in the set constructor/container method (internally calls setRecords) - creates a data copy
2. Using the symbol method setRecords - creates a data copy
3. Setting the property records directly - does not create a data copy

Setting equation records require the user to be explicit with the type of equation that is being created; in contrast to setting variable records (where the default variable is considered to be free).

If the data is in a convenient format, a user may want to pass the records directly within the equation constructor. This is an optional keyword argument and internally the equation constructor will simply call the setRecords method. In contrast to the setRecords methods in in either the Set or Parameter classes the setRecords method for variables will only accept Pandas DataFrames and specially structured dict for creating records from matrices. This restriction is out of necessity because to properly set a record for an Equation the user must pass data for the level, marginal, lower, upper and scale attributes. That said, any missing attributes will be filled in with the GAMS default record values (level = 0.0, marginal = 0.0, lower = -inf, upper = inf, scale = 1.0). We show a few examples of ways to create differently structured variables:

Example #1 - Create a GAMS scalar equation
import gamstransfer as gt
m = gt.Container()
# here we create an equality (=E=) equation
z = gt.Equation(m, "z", "eq", records=pd.DataFrame(data=[3.14159], columns=["level"]))
# NOTE: the above syntax is equivalent to -
# pi = gt.Equation(m, "pi", "eq")
# pi.setRecords(pd.DataFrame(data=[3.14159], columns=["level"]))
# NOTE: the above syntax is also equivalent to -
In [1]: pi.records
Out[1]:
level marginal lower upper scale
0 3.14159 0.0 -inf inf 1.0
Example #2 - Create a 1D Equation (defined over *) from a list of tuples

In this example we only set the marginal values.

import gamstransfer as gt
m = gt.Container()
# here we define a greater than or equal (=G=) equation
i = gt.Equation(
m,
"i",
"geq",
domain=["*"],
records=pd.DataFrame(
data=[("i" + str(i), i) for i in range(5)], columns=["domain", "marginal"]
),
)
In [1]: i.type
Out[1]: 'geq'
In [2]: i.records
Out[2]:
uni_0 level marginal lower upper scale
0 i0 0.0 0.0 -inf inf 1.0
1 i1 0.0 1.0 -inf inf 1.0
2 i2 0.0 2.0 -inf inf 1.0
3 i3 0.0 3.0 -inf inf 1.0
4 i4 0.0 4.0 -inf inf 1.0
Example #3 - Create a 1D Equation (defined over a set) from a list of tuples
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=["i" + str(i) for i in range(5)])
# here we define a less than or equal (=L=) equation
e = gt.Equation(
m,
"e",
"leq",
domain=i,
records=pd.DataFrame(
data=[("i" + str(i), i) for i in range(5)], columns=["domain", "marginal"]
),
)
In [1]: i.type
Out[1]: 'leq'
In [5]: e.records
Out[5]:
i_0 level marginal lower upper scale
0 i0 0.0 0.0 -inf inf 1.0
1 i1 0.0 1.0 -inf inf 1.0
2 i2 0.0 2.0 -inf inf 1.0
3 i3 0.0 3.0 -inf inf 1.0
4 i4 0.0 4.0 -inf inf 1.0
Example #4 - Create a 2D equation, specifying no numerical data
import gamstransfer as gt
import pandas as pd
m = gt.Container()
e = gt.Equation(
m,
"e",
"eq",
["*", "*"],
)
In [1]: e.records
Out[1]:
uni_0 uni_1 level marginal lower upper scale
0 seattle san-diego 0.0 0.0 -inf inf 1.0
1 chicago madison 0.0 0.0 -inf inf 1.0
Example #5 - Create a 2D equation (defined over a set) from a matrix
import gamstransfer as gt
import pandas as pd
import numpy as np
m = gt.Container()
i = gt.Set(m, "i", ["*"], records=["i" + str(i) for i in range(5)])
j = gt.Set(m, "j", ["*"], records=["j" + str(i) for i in range(5)])
a = gt.Parameter(
m,
"a",
[i, j],
records=[("i" + str(i), "j" + str(j), i + j) for i in range(5) for j in range(5)],
)
# create a nonbinding (=N=) equation and set the level and marginal attributes from matricies
e = gt.Equation(
m, "e", "nonbinding", domain=[i, j], records={"level": a.toDense(), "marginal": a.toDense()}
)
In [1]: e.records
Out[1]:
i_0 j_1 level marginal lower upper scale
0 i0 j1 1.0 1.0 -inf inf 1.0
1 i0 j2 2.0 2.0 -inf inf 1.0
2 i0 j3 3.0 3.0 -inf inf 1.0
3 i0 j4 4.0 4.0 -inf inf 1.0
4 i1 j0 1.0 1.0 -inf inf 1.0
5 i1 j1 2.0 2.0 -inf inf 1.0
6 i1 j2 3.0 3.0 -inf inf 1.0
7 i1 j3 4.0 4.0 -inf inf 1.0
8 i1 j4 5.0 5.0 -inf inf 1.0
9 i2 j0 2.0 2.0 -inf inf 1.0
10 i2 j1 3.0 3.0 -inf inf 1.0
11 i2 j2 4.0 4.0 -inf inf 1.0
12 i2 j3 5.0 5.0 -inf inf 1.0
13 i2 j4 6.0 6.0 -inf inf 1.0
14 i3 j0 3.0 3.0 -inf inf 1.0
15 i3 j1 4.0 4.0 -inf inf 1.0
16 i3 j2 5.0 5.0 -inf inf 1.0
17 i3 j3 6.0 6.0 -inf inf 1.0
18 i3 j4 7.0 7.0 -inf inf 1.0
19 i4 j0 4.0 4.0 -inf inf 1.0
20 i4 j1 5.0 5.0 -inf inf 1.0
21 i4 j2 6.0 6.0 -inf inf 1.0
22 i4 j3 7.0 7.0 -inf inf 1.0
23 i4 j4 8.0 8.0 -inf inf 1.0
# if not specified, the toDense() method will convert the level values to a matrix
In [2]: e.toDense()
Out[2]:
array([[0., 1., 2., 3., 4.],
[1., 2., 3., 4., 5.],
[2., 3., 4., 5., 6.],
[3., 4., 5., 6., 7.],
[4., 5., 6., 7., 8.]])

## Directly Set Records

As with set, parameters and variables, the primary advantage of the setRecords method is that GAMS Transfer will convert many different (and convenient) data types into the standard data format (a Pandas DataFrame). Users that require higher performance will want to directly pass the Container a reference to a valid Pandas DataFrame, thereby skipping some of these computational steps. This places more burden on the user to pass the data in a valid standard form, but it speeds the records setting process and it avoids making a copy of the data in memory. In this section we walk the user through an example of how to set records directly.

Example #1 - Correctly set records (directly)
import gamstransfer as gt
import pandas as pd
import numpy as np
df = pd.DataFrame(
data=[
("h" + str(h), "m" + str(m), "s" + str(s))
for h in range(8760)
for m in range(60)
for s in range(60)
],
columns=["h_0", "m_1", "s_2"],
)
# it is necessary to specify all variable attributes if setting records directly
# NOTE: all numeric data must be type float
df["level"] = np.random.uniform(0, 100, len(df))
df["marginal"] = 0.0
df["lower"] = gt.SpecialValues.NEGINF
df["upper"] = gt.SpecialValues.POSINF
df["scale"] = 1.0
m = gt.Container()
hrs = gt.Set(m, "h", records=df["h_0"].unique())
mins = gt.Set(m, "m", records=df["m_1"].unique())
secs = gt.Set(m, "s", records=df["s_2"].unique())
df["h_0"] = df["h_0"].astype(hrs.records["uni_0"].dtype)
df["m_1"] = df["m_1"].astype(mins.records["uni_0"].dtype)
df["s_2"] = df["s_2"].astype(secs.records["uni_0"].dtype)
a = gt.Equation(m, "a", "eq", domain=[hrs, mins, secs])
# set records
a.records = df
In [1]: e.isValid()
Out[1]: True
Attention
All numeric data in the records will need to be type float in order to maintain a valid symbol.

In this example we create a large equation (31,536,000 records and 8880 unique domain elements) and assign it to a variable with a.records. GAMS Transfer requires that all domain columns must be a categorical data type, furthermore this categorical must be ordered. The records setter function does very little work other than checking if the object being set is a DataFrame. This places more responsibility on the user to create a DataFrame that complies with the standard format. In Example #1 we take care to properly reference the categorical data types from the domain sets – and in the end a.isValid() = True. As with Set and Parameters, users can use the .isValid(verbose=True) method to debug any structural issues.

# Create an Alias

There are two different ways to create a GAMS equation and add it to a Container.

1. Use Alias constructor
2. Use the Container method addAlias (which internally calls the Alias constructor)

## Alias Constructor

Alias constructor
Argument Type Description Required Default
container Container A reference to the Container object that the symbol is being added to Yes -
name str Name of symbol Yes -
alias_with Set object set object from which to create an alias Yes -
Example - Creating an alias from a set

GAMS Transfer only stores the reference to the parent set as part of the alias structure – most properties that are called from an alias object simply point to the properties of the parent set (with the exception of ref_container, name, and alias_with). It is possible to create an alias from another alias object. In this case a recursive search will be performed to find the root parent set – this is the set that will ultimately be stored as the alias_with property. We can see this behavior in the following example:

import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i" + str(i) for i in range(5)])
ip = gt.Alias(m, "ip", i)
ipp = gt.Alias(m, "ipp", ip)
In [1]: ip.alias_with.name
Out[1]: 'i'
In [2]: ipp.alias_with.name
Out[2]: 'i'

## Alias Properties

Property Description Type Special Setter Behavior
alias_with aliased object Set -
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
is_singleton if symbol is a singleton set bool -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
ref_container reference to the Container that the symbol belongs to Container -
summary output a dict of only the metadata dict -

## Alias Methods

Method Description Arguments/Defaults Returns
getCardinality get the full cartesian product of the domain - int
getSparsity get the sparsity of the symbol w.r.t the cardinality - float
isValid checks if the symbol is in a valid format, throw exceptions if verbose=True, recheck a symbol if force=True verbose=False
force=True
bool
setRecords main convenience method to set standard pandas.DataFrame formatted records records (many types) None

The linked structure of Aliases offers some unique opportunies to access some of the setter functionality of the parent set. Specifically, GAMS Transfer allows the user to change the domain, description, dimension, and records of the underlying parent set as a shorthand notation. We can see this behavior if we look at a modified Example #1 from Adding Set Records.

Example - Creating set records through an alias link
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i")
ip = gt.Alias(m, "ip",i)
ip.description = "adding new descriptive set text"
ip.domain = ["*", "*"]
ip.setRecords([("i" + str(i), "j" + str(j)) for i in range(3) for j in range(3)])
In [1]: i.description
Out[1]: 'adding new descriptive set text'
In [2]: i.domain
Out[2]: ['*', '*']
In [3]: i.records
Out[3]:
uni_0 uni_1 element_text
0 i0 j0
1 i0 j1
2 i0 j2
3 i1 j0
4 i1 j1
5 i1 j2
6 i2 j0
7 i2 j1
8 i2 j2
Note
An alias .isValid()=True when the underlying parent set is also valid – if the parent set is removed from the Container the alias will no longer be valid.

# Validating Data

GAMS Transfer requires that the records for all symbols exist in a standard format (GAMS Transfer Standard Data Formats) in order for them to be understood and written successfully. It is certainly possible that the data could end up in a state that is inconsistent with the standard format (especially if setting symbol attributes directly). GAMS Transfer includes the .isValid() method in order to determine if a symbol is valid and ready for writing; this method returns a bool. For example, we create two valid sets and then check them with .isValid() to be sure.

Note
It is possible to run .isValid() on both the Container as well as the symbol object – .isValid() will also return a bool if there are any invalid symbols in the Container object.
Example (valid data)
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego", "washington_dc"])
j = gt.Set(m, "j", i, records=["san-diego", "washington_dc"])
In [1]: i.isValid()
Out[1]: True
In [2]: j.isValid()
Out[2]: True
In [3]: m.isValid()
Out[3]: True

Now we create some data that is invalid due to domain violations in the set j.

Example (intentionally create domain violations)
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego", "washington_dc"])
j = gt.Set(m, "j", i, records=["grayslake", "washington_dc"])
In [1]: i.isValid()
Out[1]: True
In [2]: j.isValid()
Out[2]: False
In [3]: m.isValid()
Out[3]: False

In this example, we know that the validity of the data is compromised by the domain violations, but there could be other subtle discrepancies that must be remedied before writing data. The user can get more detailed error reporting if the verbose argument is set to True. For example:

In [1]: j.isValid(verbose=True)
Exception: Symbol 'records' contain domain violations; ensure that all domain elements have been mapped properly to a category

The .isValid() method checks:

1. If the symbol belongs to a Container
2. If all domain set symbols exist in the Container
3. If all domain set symbols objects are valid
4. If records are a DataFrame (or None)
5. The shape of the records is congruent with the dimensionality of the symbol
6. If records column headings are in stanard format
7. If all domain columns are type category and also ordered
8. If all domain categorical dtypes are referenced properly (.records for referenced domain sets cannot be None in order to create categoricals properly)
9. If there are any domain violations
10. If there are any duplicate domain members
11. That all data columns are type float
12. To make sure that all domain categories are type str

# Domain Forwarding

GAMS includes the ability to define sets directly from data using the implicit set notation (see: Implicit Set Definition (or: Domain Defining Symbol Declarations)). This notation has an analogue in GAMS Transfer called domain_forwarding.

Note
It is possible to recursively update a subset tree in GAMS Transfer.

Domain forwarding is available as an argument to all symbol object constructors; the user would simply need to pass domain_forwarding=True.

In this example we have raw data that in the dist DataFrame and we want to send the domain information into the i and j sets – we take care to pass the set objects as the domain for parameter c.

import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i")
j = gt.Set(m, "j")
dist = pd.DataFrame(
[
("seattle", "new-york", 2.5),
("seattle", "chicago", 1.7),
("seattle", "topeka", 1.8),
("san-diego", "new-york", 2.5),
("san-diego", "chicago", 1.8),
("san-diego", "topeka", 1.4),
],
columns=["from", "to", "thousand_miles"],
)
c = gt.Parameter(m, "c", [i, j], records=dist, domain_forwarding=True)
In [1]: i.records
Out[1]:
uni_0 element_text
0 seattle
1 san-diego
In [2]: j.records
Out[2]:
uni_0 element_text
0 new-york
1 chicago
2 topeka
In [3]: c.records
Out[3]:
i_0 j_1 value
0 seattle new-york 2.5
1 seattle chicago 1.7
2 seattle topeka 1.8
3 san-diego new-york 2.5
4 san-diego chicago 1.8
5 san-diego topeka 1.4
Note
The element order in the sets i and j mirrors that in the raw data.

In this example we show that domain forwarding will also work recursively to update the entire set lineage – the domain forwarding occurs at the creation of every symbol object. The correct order of elements in set i is [z, a, b, c] because the records from j are forwarded first, and then the records from k are propagated through (back to i).

import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i")
j = gt.Set(m, "j", i, records=["z"], domain_forwarding=True)
k = gt.Set(m, "k", j, records=["a", "b", "c"], domain_forwarding=True)
In [1]: i.records
Out[1]:
uni_0 element_text
0 z
1 a
2 b
3 c
In [2]: j.records
Out[2]:
i_0 element_text
0 z
1 a
2 b
3 c
In [3]: k.records
Out[3]:
j_0 element_text
0 a
1 b
2 c

# Describing Data

The methods describeSets, describeParameters, describeVariables, and describeEquations allow the user to get a summary view of key data statistics. The returned DataFrame aggregates the output for a number of other methods (depending on symbol type). A description of each Container method is provided in the following subsections:

## describeSets

Argument Type Description Required Default
symbols list, str, NoneType A list of sets in the Container to include in the output. describeSets will include aliases if they are explicitly passed by the user. No None (if None specified, will assume all sets – not aliases)

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Property / Statistic Description
name name of the symbol
is_singleton bool if the set/alias is a singleton set (or an alias of a singleton set)
alias_with [OPTIONAL if users passes an alias name as part of symbols] name of the parent set (for alias only), None otherwise
domain domain labels for the symbol
domain_type none, relaxed or regular depending on the symbol state
dim dimension
num_recs number of records in the symbol
cardinality cartesian product of the domain information
sparsity 1 - num_recs/cardinality
Example #1
import gamstransfer as gt
m = gt.Container("trnsport.gdx")
In [1]: m.describeSets()
Out[1]:
name is_singleton domain domain_type dim num_recs cardinality sparsity
0 i False [*] none 1 2 None None
1 j False [*] none 1 3 None None
Example #2 – with aliases
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i" + str(i) for i in range(1, 10)])
j = gt.Set(m, "j", records=["j" + str(i) for i in range(1, 10)])
ip = gt.Alias(m, "ip", i)
jp = gt.Alias(m, "jp", j)
In [1]: m.describeSets()
Out[1]:
name is_singleton domain domain_type dim num_recs cardinality sparsity
0 i False [*] none 1 9 None None
1 j False [*] none 1 9 None None
In [2]: m.describeSets(m.listSets() + m.listAliases())
Out[2]:
name is_singleton is_alias alias_with domain domain_type dim num_recs cardinality sparsity
0 i False False None [*] none 1 9 None None
1 ip False True i [*] none 1 9 None None
2 j False False None [*] none 1 9 None None
3 jp False True j [*] none 1 9 None None

## describeParameters

Argument Type Description Required Default
symbols list, str, NoneType A list of parameters in the Container to include in the output No None (if None specified, will assume all parameters)

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Property / Statistic Description
name name of the symbol
is_scalar bool if the symbol is a scalar (i.e., dimension = 0)
domain domain labels for the symbol
domain_type none, relaxed or regular depending on the symbol state
dim dimension
num_recs number of records in the symbol
min_value min value in data
mean_value mean value in data
max_value max value in data
where_min domain of min value (if multiple, returns only first occurance)
where_max domain of max value (if multiple, returns only first occurance)
count_eps number of SpecialValues.EPS in data
count_na number of SpecialValues.NA in data
count_undef number of SpecialValues.UNDEF in data
cardinality cartesian product of the domain information
sparsity 1 - num_recs/cardinality
Example
import gamstransfer as gt
m = gt.Container("trnsport.gdx")
In [1]: m.describeParameters()
Out[1]:
name is_scalar domain domain_type dim num_recs min_value mean_value max_value where_min where_max count_eps count_na count_undef cardinality sparsity
0 a False [i] regular 1 2 350.000 475.000 600.000 [seattle] [san-diego] 0 0 0 2 0.0
1 b False [j] regular 1 3 275.000 300.000 325.000 [topeka] [new-york] 0 0 0 3 0.0
2 c False [i, j] regular 2 6 0.126 0.176 0.225 [san-diego, topeka] [seattle, new-york] 0 0 0 6 0.0
3 d False [i, j] regular 2 6 1.400 1.950 2.500 [san-diego, topeka] [seattle, new-york] 0 0 0 6 0.0
4 f True [] none 0 1 90.000 90.000 90.000 None None 0 0 0 None None

## describeVariables

Argument Type Description Required Default
symbols list, str, NoneType A list of variables in the Container to include in the output No None (if None specified, will assume all variables)

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Property / Statistic Description
name name of the symbol
type type of variable (i.e., binary,integer,positive,negative,free,sos1,sos2,semicont,semiint)
domain domain labels for the symbol
domain_type none, relaxed or regular depending on the symbol state
dim dimension
num_recs number of records in the symbol
cardinality cartesian product of the domain information
sparsity 1 - num_recs/cardinality
min_level min value in the level
mean_level mean value in the level
max_level max value in the level
where_max_abs_level domain of max(abs(level)) in data
count_eps_level number of SpecialValues.EPS in level
min_marginal min value in the marginal
mean_marginal mean value in the marginal
max_marginal max value in the marginal
where_max_abs_marginal domain of max(abs(marginal)) in data
count_eps_marginal number of SpecialValues.EPS in marginal
Example
import gamstransfer as gt
m = gt.Container("trnsport.gdx")
In [1]: m.describeVariables()
Out[1]:
name type domain domain_type dim num_recs cardinality sparsity min_level mean_level max_level where_max_abs_level count_eps_level min_marginal mean_marginal max_marginal where_max_abs_marginal count_eps_marginal
0 x positive [i, j] regular 2 6 6 0.0 0.000 150.000 300.000 [seattle, chicago] 0 0.0 0.008 0.036 [seattle, topeka] 0
1 z free [] none 0 1 None None 153.675 153.675 153.675 None 0 0.0 0.000 0.000 None 0

## describeEquations

Argument Type Description Required Default
symbols list, str, NoneType A list of equations in the Container to include in the output No None (if None specified, will assume all equations)

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return.

Property / Statistic Description
name name of the symbol
type type of variable (i.e., binary, integer, positive, negative, free, sos1, sos2, semicont, semiint)
domain domain labels for the symbol
domain_type none, relaxed or regular depending on the symbol state
dim dimension
num_recs number of records in the symbol
cardinality cartesian product of the domain information
sparsity 1 - num_recs/cardinality
min_level min value in the level
mean_level mean value in the level
max_level max value in the level
where_max_abs_level domain of max(abs(level)) in data
count_eps_level number of SpecialValues.EPS in level
min_marginal min value in the marginal
mean_marginal mean value in the marginal
max_marginal max value in the marginal
where_max_abs_marginal domain of max(abs(marginal)) in data
count_eps_marginal number of SpecialValues.EPS in marginal
Example
import gamstransfer as gt
m = gt.Container("trnsport.gdx")
In [1]: m.describeEquations()
Out[1]:
name type domain domain_type dim num_recs cardinality sparsity min_level mean_level max_level where_max_abs_level count_eps_level min_marginal mean_marginal max_marginal where_max_abs_marginal count_eps_marginal
0 cost eq [] none 0 1 None None -0.0 0.0 -0.0 None 1 1.000 1.000 1.000 None 0
1 demand geq [j] regular 1 3 3 0.0 275.0 300.0 325.0 [new-york] 0 0.126 0.168 0.225 [new-york] 0
2 supply leq [i] regular 1 2 2 0.0 350.0 450.0 550.0 [san-diego] 0 0.000 0.000 0.000 [seattle] 1

## describeAliases

Argument Type Description Required Default
symbols list, str, NoneType A list of alias (only) symbols in the Container to include in the output No None (if None specified, will assume all aliases – not sets)

Returns: pandas.DataFrame

The following table includes a short description of the column headings in the return. All data is referenced from the parent set that the alias is created from.

Property / Statistic Description
name name of the symbol
is_singleton bool if the set/alias is a singleton set (or an alias of a singleton set)
alias_with name of the parent set (for alias only), None otherwise
domain domain labels for the symbol
domain_type none, relaxed or regular depending on the symbol state
dim dimension
num_recs number of records in the symbol
cardinality cartesian product of the domain information
sparsity 1 - num_recs/cardinality
Example
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i" + str(i) for i in range(5)])
j = gt.Set(m, "j", records=["j" + str(j) for j in range(10)])
ip = gt.Alias(m, "ip", i)
ipp = gt.Alias(m, "ipp", ip)
jp = gt.Alias(m, "jp", j)
In [1]: m.describeAliases()
Out[1]:
name alias_with is_singleton domain domain_type dim num_recs cardinality sparsity
0 ip i False [*] none 1 5 None None
1 ipp i False [*] none 1 5 None None
2 jp j False [*] none 1 10 None None

# Matrix Generation

GAMS Transfer stores data in a "flat" format, that is, one record entry per DataFrame row. However, it is often necessary to convert this data format into a matrix format – GAMS Transfer enables users to do this with relative ease using the toDense and the toSparseCoo symbol methods. The toDense method will return a dense N-dimensional numpy array with each dimension corresponding to the GAMS symbol dimension; it is possible to output an array up to 20 dimensions (a GAMS limit). The toSparseCoo method will return the data in a sparse scipy COOrdinate format, which can then be efficiently converted into other sparse matrix formats.

Attention
Both the toDense and toSparseCoo methods do not transform the underlying DataFrame in any way, they only return the transformed data.
Note
toSparseCoo will only convert 2-dimensional data to the scipy COOrdinate format. A user interested in sparse data for an N-dimensional symbol will need to decide how to reshape the dense array in order to generate the 2D sparse format.
Attention
In order to use the toSparseCoo method the user will need to install the scipy package. Scipy is not provided with GMSPython.

Both the toDense and toSparseCoo method leverage the indexing that comes along with using categorical data types to store domain information. This means that linking symbols together (by passing symbol objects as domain information) impacts the size of the matrix. This is best demonstrated by a few examples.

Example (1D data w/o domain linking (i.e., a relaxed domain))
import gamstransfer as gt
m = gt.Container()
a = gt.Parameter(m, "a", "i", records=[("a", 1), ("c", 3)])
In [1]: a.records
Out[1]:
i_0 value
0 a 1.0
1 c 3.0
In [2]: a.toDense()
Out[2]: array([1., 3.])
In [3]: a.toSparseCoo()
Out[3]:
<1x2 sparse matrix of type '<class 'numpy.float64'>'
with 2 stored elements in COOrdinate format>

Note that the parameter a is not linked to another symbol, so when converting to a matrix, the indexing is referenced to the data structure in a.records. Defining a sparse parameter a over a set i allows us to extract information from the i domain and construct a very different dense matrix, as the following example shows:

Example (1D data w/ domain linking (i.e., a regular domain))
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["a", "b", "c", "d"])
a = gt.Parameter(m, "a", i, records=[("a", 1), ("c", 3)])
In [1]: i.records
Out[1]:
uni_0 element_text
0 a
1 b
2 c
3 d
In [2]: a.records
Out[2]:
i_0 value
0 a 1.0
1 c 3.0
In [3]: a.toDense()
Out[3]: array([1., 0., 3., 0.])
In [4]: a.toSparseCoo()
Out[4]:
<1x4 sparse matrix of type '<class 'numpy.float64'>'
with 2 stored elements in COOrdinate format>
Example (2D data w/ domain linking)
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["a", "b", "c", "d"])
a = gt.Parameter(m, "a", [i, i], records=[("a", "a", 1), ("c", "c", 3)])
In [1]: i.records
Out[1]:
uni_0 element_text
0 a
1 b
2 c
3 d
In [2]: a.records
Out[2]:
i_0 i_1 value
0 a a 1.0
1 c c 3.0
In [3]: a.toDense()
Out[3]:
array([[1., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 3., 0.],
[0., 0., 0., 0.]])
In [4]: a.toSparseCoo()
Out[4]:
<4x4 sparse matrix of type '<class 'numpy.float64'>'
with 2 stored elements in COOrdinate format>

# The Universe Set

A Unique Element List (UEL) (aka the "universe" or "universe set") is an (i,s) pair where i is an identification number for a string s. GAMS uses UELs to efficiently store domain entries of a record by storing the UEL ID i of a domain entry instead of the actual string s. This avoids storing the same string multiple times. The concept of UELs also exists in Python/Pandas and is called a "categorical array". GAMS Transfer leverages these types in order to efficiently store strings and enable domain checking within the Python environment.

Each domain column in a DataFrame can be assigned a unique categorical type, the effect is that each symbol maintains its own list of UELs per dimension. It is possible to convert a categorical column to its ID number representation by using the categorical accessor x.records[<domain_column_label>].cat.codes; however, this type of data manipulation is not necessary within GAMS Transfer, but could be handy when debugging data.

Pandas offers the possibility to create categorical column types that are ordered or not; GAMS Transfer relies exclusively on ordered categorical data types (in order for a symbol to be valid it must have only ordered categories). By using ordered categories, GAMS Transfer will order the UEL such that elements appear in the order in which they appeared in the data (which is how GAMS defines the UEL). GAMSTransfer allows the user to reorder the UELs with the uel_priority argument in the .write() method.

GAMS Transfer does not actually keep track of the UEL separately from other symbols in the Container, it will be created interal to the .write() method and is based on the order in which data is added to the container. The user can access the current state of the UEL with the .getUniverseSet() container method. For example, we set a two dimensional set:

import gamstransfer as gt
m = gt.Container()
j = gt.Set(m, "j", ["*", "*"], records=[("i" + str(n), "j" + str(n)) for n in range(2)])
In [1]: j.records
Out[1]:
uni_0 uni_1 element_text
0 i0 j0
1 i1 j1
In [2]: m.getUniverseSet()
Out[2]: ['i0', 'j0', 'i1', 'j1']

Pandas also includes a number of methods that allow categories to be renamed, appended, etc. These methods may be useful for advanced users, but most users will probably find that modifying the original data structures and resetting the symbol records provides a simpler solution. The design of GAMS Transfer should enable the user to quickly move data back and forth, without worrying about the deeper mechanics of categorical data.

# Reordering Symbols

The order of the Container file requires the symbols to be sorted such that, for example, a Set used as domain of another symbol appears before that symbol. The Container will try to establish a valid ordering when writing the data. This type of situation could be encountered if the user is adding and removing many symbols (and perhaps rewriting symbols with the same name) – users should attempt to only add symbols to a Container once, and care must be taken when creating symbol names. The method reorderSymbols attempts to fix symbol ordering problems. The following example shows how this can occur:

Example Symbol reordering
import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["i" + str(i) for i in range(5)])
j = gt.Set(m, "j", i, records=["i" + str(i) for i in range(3)])
In [1]: m.data
Out[1]:
{'i': <src.gamstransfer.Set at 0x7f90c068a8e0>,
'j': <src.gamstransfer.Set at 0x7f908084ceb0>}
# now we remove the set i and recreate the data
m.removeSymbols("i")
i = gt.Set(m, "i", records=["i" + str(i) for i in range(5)])

The symbols are now out of order in .data and must be reordered:

In [1]: m.data
Out[1]:
{'j': <src.gamstransfer.Set at 0x7f90c068a8e0>,
'i': <src.gamstransfer.Set at 0x7f908084ceb0>}
# calling reorderSymbols() will order the dictionary properly, but the domain reference in j is now broken
m.reorderSymbols()
# fix the domain reference in the set j
j.domain = i
In [1]: m.isValid()
Out[1]: True

# Rename Symbols

It is possible to rename a symbol even after it has been added to a Container. There are two methods that can be used to achieve the desired outcome:

• using the container method renameSymbol
• directly changing the name symbol property

We create a Container with two sets:

import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"])
j = gt.Set(m, "j", records=["new-york", "chicago", "topeka"])
Example #1 - Change the name of a symbol with the container method
In [1]: m.renameSymbol("i","h")
In [2]: m.data
Out[2]:
{'h': <src.gamstransfer.Set at 0x7f9fc01fc070>,
'j': <src.gamstransfer.Set at 0x7f9f9080a220>}
Example #2 - Change the name of a symbol with the .name attribute
In [1]: i.name = "h"
In [2]: m.data
Out[2]:
{'h': <src.gamstransfer.Set at 0x7f9fc01fc070>,
'j': <src.gamstransfer.Set at 0x7f9f9080a220>}
Note
Note that the renamed symbols maintain the original symbol order, this will prevent unnecessary reordering operations later in the workflow.

# Removing Symbols

Removing symbols from a container is easy when using the removeSymbols container method; this method accepts either a str or a list of str.

Attention
Once a symbol has been removed, it is possible to have hanging references as domain links in other symbols. The user will need to repair these other symbols with the proper domain links in order to avoid validity errors.

# Full Example

It is possible to use everything we now know about GAMS Transfer to recreate the trnsport.gms results in GDX form. As part of this example we also introduce the write method (and generate new.gdx). We will discuss it in more detail in the following section: Data Exchange with GDX.

import gamstransfer as gt
# create an empty Container object
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"], description="supply")
j = gt.Set(m, "j", records=["new-york", "chicago", "topeka"], description="markets")
a = gt.Parameter(m, "a", ["*"], description="capacity of plant i in cases")
b = gt.Parameter(m, "b", j, description="demand at market j in cases")
d = gt.Parameter(m, "d", [i, j], description="distance in thousands of miles")
f = gt.Parameter(
m, "f", records=90, description="freight in dollars per case per thousand miles"
)
c = gt.Parameter(
m, "c", [i, j], description="transport cost in thousands of dollars per case"
)
# set parameter records
cap = pd.DataFrame([("seattle", 350), ("san-diego", 600)], columns=["plant", "n_cases"])
a.setRecords(cap)
dem = pd.DataFrame(
[("new-york", 325), ("chicago", 300), ("topeka", 275)],
columns=["market", "n_cases"],
)
b.setRecords(dem)
dist = pd.DataFrame(
[
("seattle", "new-york", 2.5),
("seattle", "chicago", 1.7),
("seattle", "topeka", 1.8),
("san-diego", "new-york", 2.5),
("san-diego", "chicago", 1.8),
("san-diego", "topeka", 1.4),
],
columns=["from", "to", "thousand_miles"],
)
d.setRecords(dist)
# c(i,j) = f * d(i,j) / 1000;
cost = d.records.copy(deep=True)
cost["value"] = f.records.loc[0, "value"] * cost["value"] / 1000
c.setRecords(cost)
q = pd.DataFrame(
[
("seattle", "new-york", 50, 0),
("seattle", "chicago", 300, 0),
("seattle", "topeka", 0, 0.036),
("san-diego", "new-york", 275, 0),
("san-diego", "chicago", 0, 0.009),
("san-diego", "topeka", 275, 0),
],
columns=["from", "to", "level", "marginal"],
)
x = gt.Variable(
m, "x", "positive", [i, j], records=q, description="shipment quantities in cases",
)
z = gt.Variable(
m,
"z",
records=pd.DataFrame(data=[153.675], columns=["level"]),
description="total transportation costs in thousands of dollars",
)
cost = gt.Equation(m, "cost", "eq", description="define objective function")
supply = gt.Equation(m, "supply", "leq", i, description="observe supply limit at plant i")
demand = gt.Equation(m, "demand", "geq", j, description="satisfy demand at market j")
# set equation records
cost.setRecords(
pd.DataFrame(data=[[0, 1, 0, 0]], columns=["level", "marginal", "lower", "upper"])
)
supplies = pd.DataFrame(
[
("seattle", 350, "eps", float("-inf"), 350),
("san-diego", 550, 0, float("-inf"), 600),
],
columns=["from", "level", "marginal", "lower", "upper"],
)
supply.setRecords(supplies)
demands = pd.DataFrame(
[
("new-york", 325, 0.225, 325),
("chicago", 300, 0.153, 300),
("topeka", 275, 0.126, 275),
],
columns=["from", "level", "marginal", "lower"],
)
demand.setRecords(demands)
m.write("new.gdx")

With the wide range of I/O tools included in Pandas it is possible to easly draw data down from CSV data sources. We provide two examples here that pull data into GAMS Transfer from an HTML source and a POSTGRES SQL source.

Example #1 - Create symbols from HTML
import gamstransfer as gt
import pandas as pd
url = "https://www.fdic.gov/resources/resolutions/bank-failures/failed-bank-list"
# pandas will create a list of dataframe depending on the target URL, we just need the first one
df = dfs[0]
m = gt.Container()
b = gt.Set(m, "b", ["*"], records=df["Bank NameBank"].unique(), description="Bank Name")
s = gt.Set(
m,
"s",
["*"],
records=df["StateSt"].sort_values().unique(),
description="States (alphabetical order)",
)
c = gt.Set(
m,
"c",
["*"],
records=df["CityCity"].sort_values().unique(),
description="Cities (alphabetical order)",
)
c_to_s = gt.Set(
m,
"c_to_s",
[c, s],
records=df[["CityCity", "StateSt"]]
.drop_duplicates()
.sort_values(by=["StateSt", "CityCity"]),
description="City/State pair",
)
bf = gt.Parameter(
m,
"bf",
b,
records=df[["Bank NameBank", "FundFund"]]
.drop_duplicates(subset="Bank NameBank")
.sort_values(by=["Bank NameBank"]),
description="Bank Namd & Fund #",
)
In [1]: m.isValid()
Out[1]: True
Note
Users can chain Pandas operations together and pass those operations through to the records argument or the setRecords method.
Example #2 - Create symbols from a POSTGRES SQL Database (sqlalchemy)
import gamstransfer as gt
from sqlalchemy import create_engine
import pandas as pd
# connect to postgres (assuming a localhost)
engine = create_engine("postgresql://localhost:5432/" + <database_name>)
# create the Container and add symbol
m = Container()
p = Parameter(m, <sql_table_name>)
# we need to figure out the symbol dimensionality (potentially from the shape of the dataframe)
r, c = df.shape
p.dimension = c - 1
# set the records
p.setRecords(df)
# write out the GDX file
m.write("out.gdx")

# GAMS Special Values

The GAMS system contains five special values: UNDEF (undefined), NA (not available), EPS (epsilon), +INF (positive infinity), -INF (negative infinity). These special values must be mapped to their Python equivalents. GAMS Transfer follows the following convention to generate the 1:1 mapping:

• +INF is mapped to float("inf")
• -INF is mapped to float("-inf")
• EPS is mapped to -0.0 (mathematically identical to zero)
• NA is mapped to a special NaN
• UNDEF is mapped to float("nan")

GAMS Transfer syntax is designed to quickly get data into a form that is usable in further analyses or visualization; this mapping also highlights the preference for data that is of type float, which offers performance benefits within Pandas/NumPy. The user does not need to remember these constants as they are provided within the class SpecialValues as SpecialValues.POSINF, SpecialValues.NEGINF, SpecialValues.EPS, SpecialValues.NA, and SpecialValues.UNDEF. The SpecialValues class also contains methods to test for these special values. Some examples are shown below; already, we, begin to introduce some of the GAMS Transfer syntax.

Example (special values in a parameter)
import gamstransfer as gt
m = gt.Container()
x = gt.Parameter(
m,
"x",
["*"],
records=[
("i1", 1),
("i2", SpecialValues.POSINF),
("i3", SpecialValues.NEGINF),
("i4", SpecialValues.EPS),
("i5", SpecialValues.NA),
("i6", SpecialValues.UNDEF),
],
description="special values",
)

The following DataFrame for x would look like:

In [1]: x.records
Out[1]:
uni_0 value
0 i1 1.0
1 i2 inf
2 i3 -inf
3 i4 -0.0
4 i5 NaN
5 i6 NaN

The user can now easily test for specific special values in the value column of the DataFrame (returns a boolean array):

In [1]: SpecialValues.isNA(x.records["value"])
Out[1]: array([False, False, False, False, True, False])

Other data structures can be passed into these methods as long as these structures can be converted into a numpy array with dtype=float. It follows that:

In [1]: SpecialValues.isEps(SpecialValues.EPS)
Out[1]: True
In [2]: SpecialValues.isPosInf(SpecialValues.POSINF)
Out[2]: True
In [3]: SpecialValues.isNegInf(SpecialValues.NEGINF)
Out[3]: True
In [4]: SpecialValues.isNA(SpecialValues.NA)
Out[4]: True
In [5]: SpecialValues.isUndef(SpecialValues.UNDEF)
Out[5]: True
In [6]: SpecialValues.isUndef(SpecialValues.NA)
Out[6]: False
In [6]: SpecialValues.isNA(SpecialValues.UNDEF)
Out[6]: False

Pandas DataFrames allow data columns to exist with mixed type (dtype=object) – GAMS Transfer leverages this convenience feature to enable users to import string representations of EPS, NA, and UNDEF. GAMS Transfer is tolerant of any mixed-case special value string representation. Python offers additional flexiblity when representing negative/positive infinity. Any string x where float(x) == float("inf") evaluates to True can be used to represent positive infinity. Similarly, any string x where float(x) == float("-inf") evaluates to True can be used to represent negative infinity. Allowed values include inf, +inf, INFINITY, +INFINITY, -inf, -INFINITY and all mixed-case eqivalents.

Example (special values defined by strings)
import gamstransfer as gt
m = gt.Container()
x = gt.Parameter(
m,
"x",
["*"],
records=[
("i1", 1),
("i2", "+inf"),
("i3", "-infinity"),
("i4", "eps"),
("i5", "na"),
("i6", "undef"),
],
description="special values",
)

These special strings will be immediately mapped to their float equivalents from the SpecialValues class in order to ensure that all data entries are float types.

# GAMS Transfer Standard Data Formats

This section is meant to introduce the standard format that GAMS Transfer expects for symbol records. It has already been mentioned that we store data as a Pandas DataFrame, but there is an assumed structure to the column headings and column types that will be important to understand. GAMS Transfer includes convenience functions in order to ease the burden of converting data from a user-centric format to one that is understood by GAMS Transfer. However, advanced users will want to convert their data first and add it directly to the Container to avoid making extra copies of (potentially large) data sets.

Set Records Standard Format

All set records (including singleton sets) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 1. The first n-1 columns include the domain elements while the last column includes the set element explanatory text. Records are organized such that there is one record per row.

The names of the domain columns follow a pattern of <set_name>_<index_position>; a symbol dimension that is referenced to the universe is labeled uni_<index position>. The explanatory text column is called element_text and must take the last position in the DataFrame.

All domain columns must be a categorical data type and the element_text column must be a object type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however GAMS Transfer requires that all these are type str. All rows in the element_text column must be type str.

Some examples:

import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"])
j = gt.Set(m, "j", [i, "*"], records=[("seattle", "new-york"), ("san-diego", "st-louis")])
k = gt.Set(m, "k", [i], is_singleton=True, records=["seattle"])
In [1]: i.records
Out[1]:
uni_0 element_text
0 seattle
1 san-diego
In [2]: j.records
Out[2]:
i_0 uni_1 element_text
0 seattle new-york
1 san-diego st-louis
In [3]: k.records
Out[3]:
i_0 element_text
0 seattle
Parameter Records Standard Format

All parameter records (including scalars) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 1. The first n-1 columns include the domain elements while the last column includes the numerical value of the records. Records are organized such that there is one record per row. Scalar parameters have zero dimension, therefore they only have one column and one row.

The names of the domain columns follow a pattern of <set_name>_<index_position>; a symbol dimension that is referenced to the universe is labeled uni_<index_position>. The value column is called value and must take the last position in the DataFrame.

All domain columns must be a categorical data type and the value column must be a float type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however GAMS Transfer requires that all these are type str.

Some examples:

import gamstransfer as gt
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"])
a = gt.Parameter(m, "a", ["*"], records=[("seattle", 50), ("san-diego", 100)])
b = gt.Parameter(
m,
"b",
[i, "*"],
records=[("seattle", "new-york", 32.2), ("san-diego", "st-louis", 123)],
)
c = gt.Parameter(m, "c", records=90)
In [1]: a.records
Out[1]:
uni_0 value
0 seattle 50.0
1 san-diego 100.0
In [2]: b.records
Out[2]:
i_0 uni_1 value
0 seattle new-york 32.2
1 san-diego st-louis 123.0
In [3]: c.records
Out[3]:
value
0 90.0
Variable/Equation Records Standard Format

Variables and equations share the same standard data format. All records (including scalar variables/equations) are stored as a Pandas DataFrame with n number of columns, where n is the dimensionality of the symbol + 5. The first n-5 columns include the domain elements while the last five columns include the numerical values for different attributes of the records. Records are organized such that there is one record per row. Scalar variables/equations have zero dimension, therefore they have five columns and one row.

The names of the domain columns follow a pattern of <set_name>_<index position>; a symbol dimension that is referenced to the universe is labeled uni_<index_position>. The attribute columns are called level, marginal, lower, upper, and scale. These attribute columns must appear in this order. Attributes that are not supplied by the user will be assigned the default GAMS values for that variable/equation type; it is possible to not pass any attributes, GAMS Transfer would then simply assign default values to all attributes.

All domain columns must be a categorical data type and all the attribute columns must be a float type. Pandas allows the categories (basically the unique elements of a column) to be various data types as well, however GAMS Transfer requires that all these are type str.

Some examples:

import gamstransfer as gt
import pandas as pd
m = gt.Container()
i = gt.Set(m, "i", records=["seattle", "san-diego"])
a = gt.Variable(
m,
"a",
"free",
domain=[i],
records=pd.DataFrame(
[("seattle", 50), ("san-diego", 100)], columns=["city", "level"]
),
)
In [1]: a.records
Out[1]:
i_0 level marginal lower upper scale
0 seattle 50.0 0.0 -inf inf 1.0
1 san-diego 100.0 0.0 -inf inf 1.0

# Data Exchange with GDX

Up until now, we have been focused on using GAMS Transfer to create symbols in an empty Container using the symbol constructors (or their corresponding container methods). These tools will enable users to ingest data from many different formats and add them to a Container – however, it is also possible to read in symbol data directly from GDX files using the read container method. In the following sections, we will discuss this method in detail as well as the write method, which allows users to write out to new GDX files.

There are two main ways to read in GDX based data.

• Pass the file path directly to the Container constructor (will read all symbols and records)
• Pass the file path directly to the read method (default read all symbols, but can read partial files)

The first option here is provided for convenience and will, internally, call the read method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GDX file and into your Python environment. For the following examples we leverage the GDX output generated from the trnsport.gms model file.

Example (reading full data w/ Container constructor)
import gamstransfer as gt
m = gt.Container("trnsport.gdx")
In [1]: m.data
Out[1]:
{'i': <src.gamstransfer.Set at 0x7fdd21858d60>,
'j': <src.gamstransfer.Set at 0x7fdd21858dc0>,
'a': <src.gamstransfer.Parameter at 0x7fdd21858df0>,
'b': <src.gamstransfer.Parameter at 0x7fdd21858d90>,
'd': <src.gamstransfer.Parameter at 0x7fdd21858e80>,
'f': <src.gamstransfer.Parameter at 0x7fdd21858eb0>,
'c': <src.gamstransfer.Parameter at 0x7fdd21858ee0>,
'x': <src.gamstransfer.Variable at 0x7fdd21858f10>,
'z': <src.gamstransfer.Variable at 0x7fdd21858e50>,
'cost': <src.gamstransfer.Equation at 0x7fdd21858f70>,
'supply': <src.gamstransfer.Equation at 0x7fdd21858fa0>,
'demand': <src.gamstransfer.Equation at 0x7fdd21858fd0>}
In [1]: m.describeParameters()
Out[1]:
name is_scalar domain domain_type dim num_recs min_value mean_value max_value where_min where_max count_eps count_na count_undef cardinality sparsity
0 a False [i] regular 1 2 350.000 475.000 600.000 [seattle] [san-diego] 0 0 0 2 0.0
1 b False [j] regular 1 3 275.000 300.000 325.000 [topeka] [new-york] 0 0 0 3 0.0
2 c False [i, j] regular 2 6 0.126 0.176 0.225 [san-diego, topeka] [seattle, new-york] 0 0 0 6 0.0
3 d False [i, j] regular 2 6 1.400 1.950 2.500 [san-diego, topeka] [seattle, new-york] 0 0 0 6 0.0
4 f True [] none 0 1 90.000 90.000 90.000 None None 0 0 0 None None

A user could also read in data with the read method as shown in the following example.

import gamstransfer as gt
m = gt.Container()
In [1]: m.data
Out[1]:
{'i': <src.gamstransfer.Set at 0x7fdd21858d60>,
'j': <src.gamstransfer.Set at 0x7fdd21858dc0>,
'a': <src.gamstransfer.Parameter at 0x7fdd21858df0>,
'b': <src.gamstransfer.Parameter at 0x7fdd21858d90>,
'd': <src.gamstransfer.Parameter at 0x7fdd21858e80>,
'f': <src.gamstransfer.Parameter at 0x7fdd21858eb0>,
'c': <src.gamstransfer.Parameter at 0x7fdd21858ee0>,
'x': <src.gamstransfer.Variable at 0x7fdd21858f10>,
'z': <src.gamstransfer.Variable at 0x7fdd21858e50>,
'cost': <src.gamstransfer.Equation at 0x7fdd21858f70>,
'supply': <src.gamstransfer.Equation at 0x7fdd21858fa0>,
'demand': <src.gamstransfer.Equation at 0x7fdd21858fd0>}

It is also possible to read in a partial GDX file with the read method, as shown in the following example:

m = gt.Container()
In [1]: m.data
Out[1]: {'x': <src.gamstransfer.Variable at 0x7fa728a2d9d0>}
In [2]: m.data["x"].records
Out[2]:
i_0 j_1 level marginal lower upper scale
0 seattle new-york 50.0 0.000 0.0 inf 1.0
1 seattle chicago 300.0 0.000 0.0 inf 1.0
2 seattle topeka 0.0 0.036 0.0 inf 1.0
3 san-diego new-york 275.0 0.000 0.0 inf 1.0
4 san-diego chicago 0.0 0.009 0.0 inf 1.0
5 san-diego topeka 275.0 0.000 0.0 inf 1.0

This syntax assumes that the user will always want to read in both the metadata as well as the actual data records, but it is possible to skip the reading of the records by passing the argument records=False.

m = gt.Container()
In [1]: m.data
Out[1]: {'x': <src.gamstransfer.Variable at 0x7fa728a37220>}
In [2]: m.data["x"].summary
Out[2]:
{'name': 'x',
'type': 'positive',
'domain_objects': ['i', 'j'],
'domain_names': ['i', 'j'],
'dimension': 2,
'description': 'shipment quantities in cases',
'number_records': None,
'domain_type': 'relaxed'}
In [3]: type(m.data["x"].records)
Out[3]: <class 'NoneType'>
Attention
The read method attempts to link the domain objects together (in order to have a "regular" domain_type) but if domain sets are not part of the read operation there is no choice but to default to a "relaxed" domain_type. This can be seen in the last example where we only read in the variable x and not the domain sets (i and j) that the variable is defined over. All the data will be available to the user, but domain checking is no longer possible. The symbol x will remain with "relaxed" domain type even if the user were to read in sets i and j in a second read call.

## Writing to GDX

A user can write data to a GDX file by simply passing a file path (as a string). The write method will then create the GDX and write all data in the Container.

Note
It is not possible to write the Container when any of its symbols are invalid. If any symbols are invalid an error will be raised and the user will need to inspect the problematic symbols (perhaps using a combination of the listSymbols(isValid=False) and isValid(verbose=True) methods).
Example
m.write("path/to/file.gdx")
Example (write a compressed GDX file)
m.write("path/to/file.gdx", compress=True)

Advanced users might want to specify an order to their UEL list (i.e., the universe set); recall that the UEL ordering follows that dictated by the data. As a convenience, it is possible to prepend the UEL list with a user specified order using the uel_priority argument.

Example (change the order of the UEL)
m = gt.Container()
i = gt.Set(m, "i", records=["a", "b", "c"])
m.write("foo.gdx", uel_priority=["a", "c"])

The original UEL order for this GDX file would have been ["a", "b", "c"], but this example reorders the UEL with uel_priority – the positions of b and c have been swapped. This can be verified with the gdxdump utility (using the uelTable argument):

gdxdump foo.gdx ueltable=foo

Set foo /
'a' ,
'c' ,
'b' /;
$onEmpty Set i(*) / 'a', 'c', 'b' /;$offEmpty


# Data Exchange with GamsDatabase and GMD Objects (Embedded Python Code)

We have discussed how to create symbols in an empty Container and we have discussed how to exchange data with GDX files, however it is also possible to read and write data directly in memory by interacting with a GamsDatabase/GMD object – this allows GAMS Transfer to be used to read/write data within an Embedded Python Code environment or in combination with the Python OO API. There are some important differences when compared to data exchange with GDX since we are working with data representations in memory.

## Reading from GamsDatabase and GMD Objects

Just as with a GDX, there are two main ways to read in data that is in a GamsDatabase/GMD object.

• Pass the GamsDatabase/GMD object directly to the Container constructor (will read all symbols and records)
• Pass the GamsDatabase/GMD object directly to the read method (default read all symbols, but can read partial files)

The first option here is provided for convenience and will, internally, call the read method. This method will read in all symbols as well as their records. This is the easiest and fastest way to get data out of a GamsDatabase/GMD object and into your Python environment. While it is possible to generate a custom GamsDatabase/GMD object from scratch (using the gmdcc API), most users will be interacting with a GamsDatabase/GMD object that has already been instantiated internally when he/she is using Embedded Python Code or the GamsDatabase class in the Python OO API. Our examples will show how to access the GamsDatabase/GMD object – we leverage the some of the data from the trnsport.gms model file.

Example (reading full data w/ Container constructor)
m = gt.Container(gams.db)
Note
Embedded Python Code users will want pass the GamsDatabase object that is part of the GAMS Database object – this will always be referenced as gams.db regardless of the model file.

The following example uses embedded Python code to create a new Container, read in all symbols, and display some summary statistics as part of the gams log output.

Set
i 'canning plants' / seattle,  san-diego /
j 'markets'        / new-york, chicago, topeka /;

Parameter
a(i) 'capacity of plant i in cases'
/ seattle    350
san-diego  600 /

b(j) 'demand at market j in cases'
/ new-york   325
chicago    300
topeka     275 /;

Table d(i,j) 'distance in thousands of miles'
new-york  chicago  topeka
seattle         2.5      1.7     1.8
san-diego       2.5      1.8     1.4;

$onembeddedCode Python: import gamstransfer as gt m = gt.Container(gams.db) print(m.describeSets()) print(m.describeParameters())$offEmbeddedCode


The gams log output will then look as such (the extra print calls are just providing nice spacing for this example):

GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- matrix.gms(29) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
name  is_singleton domain domain_type  dim  num_recs cardinality sparsity
0    i         False    [*]        none    1         2        None     None
1    j         False    [*]        none    1         3        None     None
name  is_scalar  domain domain_type  dim  num_recs  min_value  mean_value  max_value            where_min            where_max  count_eps  count_na  count_undef  cardinality  sparsity
0    a      False     [i]     regular    1         2      350.0      475.00      600.0            [seattle]          [san-diego]          0         0            0            2       0.0
1    b      False     [j]     regular    1         3      275.0      300.00      325.0             [topeka]           [new-york]          0         0            0            3       0.0
2    d      False  [i, j]     regular    2         6        1.4        1.95        2.5  [san-diego, topeka]  [seattle, new-york]          0         0            0            6       0.0

--- Starting execution - empty program
*** Status: Normal completion

[3 rows x 16 columns]

--- Starting execution - empty program
*** Status: Normal completion


A user could also read in a subset of the data located in the GamsDatabase object with the read method as shown in the following example. Here we only read in the sets i and j, as a result the .describeParameters() method will return None.

Set
i 'canning plants' / seattle,  san-diego /
j 'markets'        / new-york, chicago, topeka /;

Parameter
a(i) 'capacity of plant i in cases'
/ seattle    350
san-diego  600 /

b(j) 'demand at market j in cases'
/ new-york   325
chicago    300
topeka     275 /;

Table d(i,j) 'distance in thousands of miles'
new-york  chicago  topeka
seattle         2.5      1.7     1.8
san-diego       2.5      1.8     1.4;

$onembeddedCode Python: import gamstransfer as gt m = gt.Container() m.read(gams.db, symbols=["i","j"]) gams.printLog("") print(m.describeSets()) print(m.describeParameters())$offEmbeddedCode

GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- matrix.gms(29) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   name  is_singleton domain domain_type  dim  num_recs cardinality sparsity
0    i         False    [*]        none    1         2        None     None
1    j         False    [*]        none    1         3        None     None
None

--- Starting execution - empty program
*** Status: Normal completion


All the typical functionality of the Container exists when working with GamsDatabase/GMD objects. This means that domain linking, matrix conversion, and other more advanced options are available to the user at either compilation time or execution time (depending on the Embedded Code syntax being used, see: Syntax). The next example generates a 1000x1000 matrix and then takes its inverse using the Numpy linalg package.

Example (Matrix Generation and Inversion)
set i / i1*i1000 /;
alias(i,j);

parameter a(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;

embeddedCode Python:
import gamstransfer as gt
import numpy as np
import time

gams.printLog("")
s = time.time()
m = gt.Container(gams.db)
gams.printLog(f"read data: {round(time.time() - s, 3)} sec")

s = time.time()
A = m.data["a"].toDense()
gams.printLog(f"create matrix A: {round(time.time() - s, 3)} sec")

s = time.time()
invA = np.linalg.inv(A)
gams.printLog(f"generate inv(A): {round(time.time() - s, 3)} sec")

endEmbeddedCode

Note
In this example, the assignment of the a parameter is done during execution time so we must use the execution time syntax for embedded code in order to get the numerical records properly.
GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(27) 3 Mb
--- Starting execution: elapsed 0:00:00.003
--- test.gms(9) 36 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---
--- create matrix A: 0.02 sec
--- generate inv(A): 0.031 sec
*** Status: Normal completion


We will extend this example in the next section to write the inverse matrix A back into a GAMS parameter.

## Writing Data to GamsDatabase and GMD

A user can write to a GamsDatabase/GMD object with the .write() method just as he/she would write a GDX file – however there are some important differences. When a user writes a GDX file the entire GDX file represents a complete data environment (all domains have been resolved, etc.) thus, GAMS Transfer does not need to worry about merge/replace operations. It is possible to merge/replace symbol records when a user is writing data to in-memory data representations with GamsDatabase/GMD. We show a few examples to illustrate this behavior.

Example (Populating a set in GAMS)
* note that we need to declare the set i over "*" in order to provide hints about the symbol dimensionality
set i(*);

$onembeddedCode Python: import gamstransfer as gt m = gt.Container() i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)]) m.write(gams.db)$offEmbeddedCode i

embeddedCode Python:
import gamstransfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m.data["i"].records)

endEmbeddedCode

Note
In general, it is possible to use GAMS Transfer to create new symbols in a GamsDatabase and GMD object (and not necessarily merge symbols) but embedded code best practices necessitate the declaration of any GAMS symbols on the GAMS side first, then the records can be filled with GAMS Transfer.

If we break down this example we can see that the set i is declared within GAMS (with no records) and then the records for i are set by writing a Container to the gams.db GamsDatabase object (we do this at compile time). The second embedded Python code block runs at execution time and is simply there to read all the records on the set i – printing the sets this way adds the output to the .log file (we could also use the more common display i; operation in GAMS to display the set elements in the LST file).

GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(10) 2 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(20) 3 Mb
--- Starting execution: elapsed 0:00:01.464
--- test.gms(13) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   uni_0 element_text
0    i0
1    i1
2    i2
3    i3
4    i4
5    i5
6    i6
7    i7
8    i8
9    i9

*** Status: Normal completion

Example (Merging set records)
set i / i1, i2 /;

$onmulti$onembeddedCode Python:
import gamstransfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db, merge_symbols="i")

$offEmbeddedCode i$offmulti

embeddedCode Python:
import gamstransfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m.data["i"].records)

endEmbeddedCode


In this example we need to make use of $onMulti/$offMulti in order to merge new set elements into the the set i (the same would be true if we were merging other symbol types) – any symbol that already has records defined (in GAMS) and is being added to with Python (and GAMS Transfer) must be wrapped with $onMulti/$offMulti. As with the previous example, the second embedded Python code block runs at execution time and is simply there to read all the records on the set i. Note that the UEL order will be different in this case (i1 and i2 come before i0).

GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(11) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(21) 3 Mb
--- Starting execution: elapsed 0:00:01.535
--- test.gms(14) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   uni_0 element_text
0    i1
1    i2
2    i0
3    i3
4    i4
5    i5
6    i6
7    i7
8    i8
9    i9

*** Status: Normal completion

Example (Replacing set records)
set i / x1, x2 /;

$onmultiR$onembeddedCode Python:
import gamstransfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
m.write(gams.db)

$offEmbeddedCode i$offmulti

embeddedCode Python:
import gamstransfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m.data["i"].records)

endEmbeddedCode


In this example we want to replace the x1 and x2 set elements and built up a totally new element list with set elements from the Container. Instead of $onMulti/$offMulti we must use $onMultiR/$offMulti to ensure that the replacement happens in GAMS; we also need to remove the set i from the merge_symbols argument.

Attention
If the user seeks to replace all records in a symbol they must use the $onMultiR syntax. It is not sufficient to simply remove them from the merge_symbols argument in GAMS Transfer. If the user mistakenly uses $onMulti the symbols will end up merging without total replacement.
GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(11) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(21) 3 Mb
--- Starting execution: elapsed 0:00:01.482
--- test.gms(14) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   uni_0 element_text
0    i0
1    i1
2    i2
3    i3
4    i4
5    i5
6    i6
7    i7
8    i8
9    i9

*** Status: Normal completion

Example (Merging parameter records)
set i;
parameter a(i<) /
i1 1.23
i2 5
/;

$onmulti$onembeddedCode Python:
import gamstransfer as gt

m = gt.Container()
i = gt.Set(m, "i", records=["i"+str(i) for i in range(10)])
a = gt.Parameter(m, "a", domain=i, records=[("i"+str(i),i) for i in range(10)])
m.write(gams.db, merge_symbols="a")

$offEmbeddedCode i, a$offmulti

embeddedCode Python:
import gamstransfer as gt

m = gt.Container(gams.db)
gams.printLog("")
print(m.data["a"].records)
endEmbeddedCode

In this example we also need to make use of $onMulti/$offMulti in order to merge new set elements into the the set i, however the set i also needs to contain the elements that are defined in the parameter – here we make use of the < operator that will add the set elements from a(i) into the set i

Note
It would also be possible to run this example by explicitly defining the set i /i1, i2/; before the parameter declaration.
Attention
GAMS Transfer will overwrite all duplicate records when merging. The original values of a("i1") and a("i2") have been replaced with their new values when writing the Container in this example (see output below).
GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- test.gms(16) 3 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
--- test.gms(25) 3 Mb
--- Starting execution: elapsed 0:00:01.467
--- test.gms(19) 4 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---   i_0  value
0  i1    1.0
1  i2    2.0
2  i3    3.0
3  i4    4.0
4  i5    5.0
5  i6    6.0
6  i7    7.0
7  i8    8.0
8  i9    9.0

*** Status: Normal completion

Example (Advanced Matrix Generation and Inversion w/ Write Operation)
set i / i1*i1000 /;
alias(i,j);

parameter a(i,j);
a(i,j) = 1 / (ord(i)+ord(j) - 1);
a(i,i) = 1;

parameter inv_a(i,j);
parameter ident(i,j);

embeddedCode Python:
import gamstransfer as gt
import numpy as np
import time

gams.printLog("")
gams.printLog("")

s = time.time()
m = gt.Container(gams.db)
gams.printLog(f"read data: {round(time.time() - s, 3)} sec")

s = time.time()
A = m.data["a"].toDense()
gams.printLog(f"create matrix A: {round(time.time() - s, 3)} sec")

s = time.time()
invA = np.linalg.inv(A)
gams.printLog(f"calculate inv(A): {round(time.time() - s, 3)} sec")

s = time.time()
m.data["inv_a"].setRecords(invA)
gams.printLog(f"convert matrix to records for inv(A): {round(time.time() - s, 3)} sec")

s = time.time()
I = np.dot(A,invA)
tol = 1e-9
I[np.where((I<tol) & (I>-tol))] = 0
gams.printLog(f"calculate A*invA + small number cleanup: {round(time.time() - s, 3)} sec")

s = time.time()
m.data["ident"].setRecords(I)
gams.printLog(f"convert matrix to records for I: {round(time.time() - s, 3)} sec")

s = time.time()
m.write(gams.db, ["inv_a","ident"])
gams.printLog(f"write to GamsDatabase: {round(time.time() - s, 3)} sec")

gams.printLog("")
endEmbeddedCode inv_a, ident

display ident;


In this example we extend the example shown in Reading from GamsDatabase and GMD Objects to read data from GAMS, calculate a matrix inversion, do the matrix multiplication, and then write both the A^-1 and A*A^-1 (i.e., the identity matrix) back to GAMS for display in the LST file. This data round trip highlights the benefits of using a GAMS Transfer Container (and the linked symbol structure) as the mechanism to move data – converting back and forth from a records format to a matrix format can be cumbersome, but here, GAMS Transfer takes care of all the indexing for the user.

The first few lines of GAMS code generates a 1000x1000 A matrix as a parameter (at execution time), we then define two more parameters that we will fill with results of the embedded Python code – specifically we want to fill a parameter with the matrix A^-1 and we want to verify that another parameter (ident) contains the identity matrix (i.e., I). Stepping through the code:

1. We start the embedded Python code section (execution time) by importing both GAMS Transfer and Numpy and by reading all the symbols that currently exist in the GamsDatabase. We must read in all this information in order to get the domain set information – GAMS Transfer needs these domain sets in order to generate matricies with the proper size.
2. Generate the matrix A by calling .toDense() on the symbol object in the Container.
3. Take the inverse of A with np.linalg.inv().
4. The Parameter symbol for inv_a already exists in the Container, but it does not have any records (i.e., m.data["inv_a"].records is None will evaluate to True). We use .setRecords() to convert the invA back into a records format.
5. We continue the computations by performing the matrix multiplication using np.dot() – we must clean up a lot of small numbers in I.
6. The Parameter symbol for ident already exists in the Container, but it does not have any records. We use .setRecords() to convert I back into a records format.
7. Since we are calculating these parameter values at execution time, it is not possible to modify the domain set information (or even merge/replace it). Therefore we only want to write the parameter values to GAMS. We achieve this by writing a subset of the Container symbols out with the m.write(gams.db, ["inv_a","ident"]) call. This partial write preserves symbol validity in the Container and it does not violate other GAMS requirements.
8. Finally, we can verify that the (albeit large) identity matrix exists in the LST file (or in another GDX file).
Note
It was not possible to just use np.round because small negative numbers that round to -0.0 will be interpreted by GAMS Transfer as the GAMS EPS special value.

The output for this example is shown below:

GAMS 38.1.0   Copyright (C) 1987-2022 GAMS Development. All rights reserved
--- Starting compilation
--- matrix.gms(52) 3 Mb
--- Starting execution: elapsed 0:00:00.004
--- matrix.gms(11) 36 Mb
--- Initialize embedded library libembpycclib64.dylib
--- Execute embedded library libembpycclib64.dylib
---
---
--- create matrix A: 0.016 sec
--- calculate inv(A): 0.032 sec
--- convert matrix to records for inv(A): 0.176 sec
--- calculate A*invA + small number cleanup: 0.027 sec
--- convert matrix to records for I: 0.17 sec
--- write to GamsDatabase: 1.937 sec
---
--- matrix.gms(52) 68 Mb
*** Status: Normal completion


In the Create a Container section we describe how to use the main object class of GAMS Transfer – the Container. Many users of GAMS Transfer will rely on the Container for building their data pipeline, however some users will only be interested in post-processing data from a GAMS model run. This one-directional flow of data means that these users do not need some of the advanced Container features such as domain linking, matrix generation, domain checking, etc. The ConstContainer (i.e., a Constant Container) object class is a data-focused read-only object that will provide a snapshot of the data target being read – the ConstContainer can be created by reading a GDX file or a GamsDatabase/GMD object (an in memory representation of data used e.g. in embedded Python code).

## Creating a ConstContainer

The ConstContainer shares many of the same methods and attributes that are in the Container class, which makes moving between the ConstContainer and the Container very simple. There are some important differences though:

1. The ConstContainer does not link any symbol data
2. The ConstContainer can only read from one source at a time – every new call of .read() will clear the data dict
3. The ConstContainer constructor will not read in any symbol records – this enables users to browse an unknown data source quickly (similar behavior to gdxdump).
4. The ConstContainer does not have a .write() method – a ConstContainer can be passed to the constructor of a Container which will enable data writing (however a copy of the data will be generated).
5. The user will never need to instantiate a symbol object and add it to the ConstContainer – the ConstContainer will internally generate its own set of (simplified) symbol classes and hold them in the .data attribute.

All of these differences were inspired by users that want to read the data as fast as possible and probe unknown data files without worrying about memory issues – ConstContainer provides users with a high level view of the data very quickly.

ConstContainer constructor

Creating a ConstContainer is a simple matter of initializing an object. For example:

import gamstransfer as gt
h = gt.ConstContainer("out.gdx")
Note
This new ConstContainer object, here called h, will load all the symbol data from out.gdx but it will not load any of the records. To load records, users must use the .read() method.

The ConstContainer constructor arguments are:

Argument Type Description Required Default
load_from str or GamsDatabase/GMD Object Points to the source of the data being read into the ConstContainer No None
system_directory str Absolute path to GAMS system_directory No Attempts to find the GAMS installation by creating a GamsWorkspace object and loading the system_directory attribute.

The ConstContainer contains many of the same methods that are in the Container class, specifically:

ConstContainer Methods
Method Description Arguments/Defaults Returns
describeAliases create a summary table with descriptive statistics for Aliases symbols=None (None,str,list) - if None, assumes all aliases pandas.DataFrame
describeParameters create a summary table with descriptive statistics for Parameters symbols=None (None,str,list) - if None, assumes all parameters pandas.DataFrame
describEquations create a summary table with descriptive statistics for Equations symbols=None (None,str,list) - if None, assumes all equations pandas.DataFrame
describeSets create a summary table with descriptive statistics for Sets symbols=None (None,str,list) - if None, assumes all sets pandas.DataFrame
describeVariables create a summary table with descriptive statistics for Variables symbols=None (None,str,list) - if None, assumes all variables pandas.DataFrame
listAliases list all aliases - list
listEquations list all equations types=None (list of equation types) - if None, assumes all types list
listParameters list all parameters - list
listSets list all sets - list
listSymbols list all symbols - list
listVariables list all variables types=None (list of variable types) - if None, assumes all types list
read main method to read load_from, can be provided with a list of symbols to read in subsets, records controls if symbol records are loaded or just metadata load_from (str,GMD Object Handle,GamsDatabase Object)
symbols="all" (str, list)
records=True (bool)
None

The structure of the DataFrames that are returned from the describe* methods mirrors that in the Container; the user should reference Describing Data for detailed descriptions of the columns.

## ConstContainer Symbol Objects

The ConstContainer uses a simplified symbol class structure to hold symbol specific information. The user will never need to directly instantiate these symbol classes (called SimpleSet, SimpleParameter, SimpleVariable, SimpleEquation and SimpleAlias); these symbol classes are nested under the ConstContainer class. This class structure is used to provide the feel of a read-only object.

While users do not need to instantiate any of the Simple* symbol objects directly, they are available for users to probe. Many of the same Container symbol methods that generate summary statistics exist for the ConstContainer symbols. Specifically:

SimpleSet Properties
Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
is_singleton bool if symbol is a singleton set bool -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
summary output a dict of only the metadata dict -
SimpleSet Methods

None

SimpleParameter Properties
Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
is_scalar True if the len(self.domain) = 0 bool -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
summary output a dict of only the metadata dict -
SimpleParameter Methods
Method Description Arguments/Defaults Returns
countEps total number of SpecialValues.EPS across all columns - int
countNA total number of SpecialValues.NA across all columns - int
countNegInf total number of SpecialValues.NEGINF across all columns - int
countPosInf total number of SpecialValues.POSINF across all columns - int
countUndef total number of SpecialValues.UNDEF across all columns - int
findEps find index positions of SpecialValues.EPS in value column - pandas.Index
findNA find index positions of SpecialValues.NA in value column - pandas.Index
findNegInf find index positions of SpecialValues.NEGINF in value column - pandas.Index
findPosInf find index positions of SpecialValues.POSINF in value column - pandas.Index
findUndef find index positions of SpecialValues.Undef in value column - pandas.Index
getMaxValue get the maximum value across all columns - float
getMinValue get the minimum value across all columns - float
getMeanValue get the mean value across all columns - float
getMaxAbsValue get the maximum absolute value across all columns - float
whereMax find the domain entry of records with a maximum value (return first instance only) - list of str
whereMaxAbs find the domain entry of records with a maximum absolute value (return first instance only) - list of str
whereMin find the domain entry of records with a minimum value (return first instance only) - list of str
SimpleVariable Properties
Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
summary output a dict of only the metadata dict -
type str type of variable dict -
SimpleVariable Methods
Method Description Arguments/Defaults Returns
countEps total number of SpecialValues.EPS across all columns columns="level" (str,list) int
countNA total number of SpecialValues.NA across all columns columns="level" (str,list) int
countNegInf total number of SpecialValues.NEGINF across all columns columns="level" (str,list) int
countPosInf total number of SpecialValues.POSINF across all columns columns="level" (str,list) int
countUndef total number of SpecialValues.UNDEF across all columns columns="level" (str,list) int
findEps find index positions of SpecialValues.EPS in column column="level" (str) pandas.Index
findNA find index positions of SpecialValues.NA in column column="level" (str) pandas.Index
findNegInf find index positions of SpecialValues.NEGINF in column column="level" (str) pandas.Index
findPosInf find index positions of SpecialValues.POSINF in column column="level" (str) pandas.Index
findUndef find index positions of SpecialValues.Undef in column column="level" (str) pandas.Index
getMaxValue get the maximum value across all columns columns="level" (str,list) float
getMinValue get the minimum value across all columns columns="level" (str,list) float
getMeanValue get the mean value across all columns columns="level" (str,list) float
getMaxAbsValue get the maximum absolute value across all columns columns="level" (str,list) float
whereMax find the domain entry of records with a maximum value (return first instance only) column="level" (str) list of str
whereMaxAbs find the domain entry of records with a maximum absolute value (return first instance only) column="level" (str) list of str
whereMin find the domain entry of records with a minimum value (return first instance only) column="level" (str) list of str
SimpleEquation Properties
Property Description Type Special Setter Behavior
description description of symbol str -
dimension dimension of symbol int setting is a shorthand notation to create ["*"] * n domains in symbol
domain_forwarding flag that forces set elements to be recursively included in all parent sets (i.e., implicit set growth) bool no effect after records have been set
domain_labels column headings for the records DataFrame list of str -
domain_names string version of domain names list of str -
domain_type none, relaxed or regular depending on state of domain links str -
name name of symbol str sets the GAMS name of the symbol
number_records number of symbol records (i.e., returns len(self.records) if not None) int -
records the main symbol records pandas.DataFrame responsive to domain_forwarding state
summary output a dict of only the metadata dict -
type str type of variable dict -
SimpleEquation Methods
Method Description Arguments/Defaults Returns
countEps total number of SpecialValues.EPS across all columns columns="level" (str,list) int
countNA total number of SpecialValues.NA across all columns columns="level" (str,list) int
countNegInf total number of SpecialValues.NEGINF across all columns columns="level" (str,list) int
countPosInf total number of SpecialValues.POSINF across all columns columns="level" (str,list) int
countUndef total number of SpecialValues.UNDEF across all columns columns="level" (str,list) int
findEps find index positions of SpecialValues.EPS in column column="level" (str) pandas.Index
findNA find index positions of SpecialValues.NA in column column="level" (str) pandas.Index
findNegInf find index positions of SpecialValues.NEGINF in column column="level" (str) pandas.Index
findPosInf find index positions of SpecialValues.POSINF in column column="level" (str) pandas.Index
findUndef find index positions of SpecialValues.Undef in column column="level" (str) pandas.Index
getMaxValue get the maximum value across all columns columns="level" (str,list) float
getMinValue get the minimum value across all columns columns="level" (str,list) float
getMeanValue get the mean value across all columns columns="level" (str,list) float
getMaxAbsValue get the maximum absolute value across all columns columns="level" (str,list) float
whereMax find the domain entry of records with a maximum value (return first instance only) column="level" (str) list of str
whereMaxAbs find the domain entry of records with a maximum absolute value (return first instance only) column="level" (str) list of str
whereMin find the domain entry of records with a minimum value (return first instance only) column="level" (str) list of str
Example (reading only meta data w/ ConstContainer constructor)
import gamstransfer as gt
h = gt.ConstContainer("trnsport.gdx")
In [1]: h.data
Out[1]:
{'i': <src.gamstransfer.ConstContainer.SimpleSet at 0x7fba484bf5b0>,
'j': <src.gamstransfer.ConstContainer.SimpleSet at 0x7fba484bffd0>,
'a': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba484bf880>,
'b': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0f460>,
'd': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0ff40>,
'f': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0fa00>,
'c': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0f160>,
'x': <src.gamstransfer.ConstContainer.SimpleVariable at 0x7fba48b0f7c0>,
'z': <src.gamstransfer.ConstContainer.SimpleVariable at 0x7fba48b0f3a0>,
'cost': <src.gamstransfer.ConstContainer.SimpleEquation at 0x7fba48b0f7f0>,
'supply': <src.gamstransfer.ConstContainer.SimpleEquation at 0x7fba48b0fd30>,
'demand': <src.gamstransfer.ConstContainer.SimpleEquation at 0x7fba48b0fc70>}
In [2]: h.describeParameters()
Out[2]:
name is_scalar domain domain_type dim num_recs sparsity min_value mean_value max_value where_min where_max count_eps count_na count_undef
0 a False [i] regular 1 2 0.0 None None None None None None None None
1 b False [j] regular 1 3 0.0 None None None None None None None None
2 c False [i, j] regular 2 6 0.0 None None None None None None None None
3 d False [i, j] regular 2 6 0.0 None None None None None None None None
4 f True [] none 0 1 0.0 None None None None None None None None

Note that in this example we make use of the convenience notation contained in the constructor to read in only the metadata of the trnsport.gdx file. This allows users to quickly explore the symbols contained in a file (or in-memory object) and it also explains why there are many None values in the columns of the .describeParameters() method.

import gamstransfer as gt
h = gt.ConstContainer()
In [1]: h.data
Out[1]:
{'i': <src.gamstransfer.ConstContainer.SimpleSet at 0x7fba484bf5b0>,
'j': <src.gamstransfer.ConstContainer.SimpleSet at 0x7fba484bffd0>,
'a': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba484bf880>,
'b': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0f460>,
'd': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0ff40>,
'f': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0fa00>,
'c': <src.gamstransfer.ConstContainer.SimpleParameter at 0x7fba48b0f160>,
'x': <src.gamstransfer.ConstContainer.SimpleVariable at 0x7fba48b0f7c0>,
'z': <src.gamstransfer.ConstContainer.SimpleVariable at 0x7fba48b0f3a0>,
'cost': <src.gamstransfer.ConstContainer.SimpleEquation at 0x7fba48b0f7f0>,
'supply': <src.gamstransfer.ConstContainer.SimpleEquation at 0x7fba48b0fd30>,
'demand': <src.gamstransfer.ConstContainer.SimpleEquation at 0x7fba48b0fc70>}
In [2]: h.describeParameters()
Out[2]:
name is_scalar domain domain_type dim num_recs sparsity min_value mean_value max_value where_min where_max count_eps count_na count_undef
0 a False [i] regular 1 2 0.0 350.000 475.000 600.000 [seattle] [san-diego] 0 0 0
1 b False [j] regular 1 3 0.0 275.000 300.000 325.000 [topeka] [new-york] 0 0 0
2 c False [i, j] regular 2 6 0.0 0.126 0.176 0.225 [san-diego, topeka] [seattle, new-york] 0 0 0
3 d False [i, j] regular 2 6 0.0 1.400 1.950 2.500 [san-diego, topeka] [seattle, new-york] 0 0 0
4 f True [] none 0 1 0.0 90.000 90.000 90.000 None None 0 0 0

In this example we make use of the .read() method to retrieve both the metadata and the numerical records for all symbols in the GDX file – the .describeParameters() method will now populate the DataFrame with additional summary statistics.

GAMS Development Corp.
GAMS Software GmbH

General Information and Sales
U.S. (+1) 202 342-0180
Europe: (+49) 221 949-9170