dataiter

The following functions are shorthand helpers for use in conjunction with DataFrame.aggregate(), see the guide on aggregation for details.

all() any() count() count_unique() first() last() max() mean() median() min() mode() nth() quantile() std() sum() var()

The following read functions are convenience aliases to the correspoding methods of the classes generally most suitable for the particular file type, i.e. DataFrame for CSV, NPZ and Parquet, GeoJSON for GeoJSON and ListOfDicts for JSON.

read_csv() read_geojson() read_json() read_npz() read_parquet()

The following constants can be used to customize certain defaults, such as formatting and limits for printing.

dataiter.PRINT_MAX_WIDTH dataiter.PRINT_THOUSAND_SEPARATOR dataiter.PRINT_TRUNCATE_WIDTH dataiter.USE_NUMBA dataiter.USE_NUMBA_CACHE

dataiter.PRINT_MAX_WIDTH = 80

Maximum amount of columns to wrap print output to. This is only a fallback in case Python’s shutil.get_terminal_size fails to detect the width of your terminal. By default the detected full width is used.

dataiter.PRINT_THOUSAND_SEPARATOR = ''

Thousand separator to use when printing numbers. By default this is blank, meaning no thousand separators are rendered.

dataiter.PRINT_TRUNCATE_WIDTH = 36

Maximum width to truncate string columns to in DataFrame print output. When this is exceeded, strings will be cut and an ellipsis (…) rendered at the cut point.

dataiter.USE_NUMBA = False

True to use Numba, if available, to speed up aggregations, False to only use pure Python code.

dataiter.USE_NUMBA_CACHE = True

True to use Numba cache for JIT-compiled aggregations, False to only keep compiled code in memory for the duration of the session.

dataiter.all(x)

Return whether all elements of x evaluate to True.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.all, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.all.html

>>> di.all(di.Vector([True, False]))
False
>>> di.all(di.Vector([True, True]))
True
>>> di.all("x")
<function all.<locals>.aggregate at 0x7fe88fbfc4a0>

dataiter.any(x)

Return whether any element of x evaluates to True.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.any, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.any.html

>>> di.any(di.Vector([False, False]))
False
>>> di.any(di.Vector([True, False]))
True
>>> di.any("x")
<function any.<locals>.aggregate at 0x7fd7928bc4a0>

dataiter.count(x='', *, drop_na=False)

Return the amount of elements in x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x. Since all columns in a data frame should have the same amount of elements (i.e. rows), you can just leave the x argument at its default blank string, which will give you that row count.

>>> di.count(di.Vector([1, 2, 3]))
3
>>> di.count()
<function count.<locals>.aggregate at 0x7f04798a84a0>

dataiter.count_unique(x, *, drop_na=False)

Return the amount of unique elements in x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.count_unique(di.Vector([1, 2, 2, 3, 3, 3]))
3
>>> di.count_unique("x")
<function count_unique.<locals>.aggregate at 0x7f281eb6c4a0>

dataiter.first(x, *, drop_na=False)

Return the first element of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.first(di.Vector([1, 2, 3]))
1
>>> di.first("x")
<function nth.<locals>.aggregate at 0x7fe8a7c304a0>

dataiter.last(x, *, drop_na=False)

Return the last element of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.last(di.Vector([1, 2, 3]))
3
>>> di.last("x")
<function nth.<locals>.aggregate at 0x7fc1669ac4a0>

dataiter.max(x, *, drop_na=True)

Return the maximum of elements in x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.max(di.Vector([4, 5, 6]))
6
>>> di.max("x")
<function max.<locals>.aggregate at 0x7f3f4f4184a0>

dataiter.mean(x, *, drop_na=True)

Return the arithmetic mean of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.mean, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.mean.html

>>> di.mean(di.Vector([1, 2, 10]))
4.333333333333333
>>> di.mean("x")
<function mean.<locals>.aggregate at 0x7fd92a4f84a0>

dataiter.median(x, *, drop_na=True)

Return the median of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.median, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.median.html

>>> di.median(di.Vector([5, 1, 2]))
2.0
>>> di.median("x")
<function median.<locals>.aggregate at 0x7f406b7644a0>

dataiter.min(x, *, drop_na=True)

Return the minimum of elements in x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.min(di.Vector([4, 5, 6]))
4
>>> di.min("x")
<function min.<locals>.aggregate at 0x7f8566b744a0>

dataiter.mode(x, *, drop_na=True)

Return the most common value in x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.mode(di.Vector([1, 2, 2, 3, 3, 3]))
3
>>> di.mode("x")
<function mode.<locals>.aggregate at 0x7f45be1144a0>

dataiter.nth(x, index, *, drop_na=False)

Return the element of x at index (zero-based).

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.nth(di.Vector([1, 2, 3]), 1)
2
>>> di.nth("x", 1)
<function nth.<locals>.aggregate at 0x7fd86016c4a0>

dataiter.quantile(x, q, *, drop_na=True)

Return the qth quantile of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.quantile, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.quantile.html

>>> di.quantile(di.Vector([1, 5, 6]), 0.5)
5.0
>>> di.quantile("x", 0.5)
<function quantile.<locals>.aggregate at 0x7fe9558fc4a0>

dataiter.read_csv(path, *, encoding='utf-8', sep=',', header=True, columns=[], dtypes={})

Return a new data frame from CSV file path.

Will automatically decompress if path ends in .bz2|.gz|.xz. columns is an optional list of columns to limit to. dtypes is an optional dict mapping column names to NumPy datatypes.

Note

read_csv() is a convenience alias for DataFrame.read_csv().

dataiter.read_geojson(path, *, encoding='utf-8', columns=[], dtypes={}, **kwargs)

Return data from GeoJSON file path.

Will automatically decompress if path ends in .bz2|.gz|.xz. columns is an optional list of columns to limit to. dtypes is an optional dict mapping column names to NumPy datatypes. kwargs are passed to json.load.

Note

read_geojson() is a convenience alias for GeoJSON.read().

dataiter.read_json(path, *, encoding='utf-8', keys=[], types={}, **kwargs)

Return a new list from JSON file path.

Will automatically decompress if path ends in .bz2|.gz|.xz. keys is an optional list of keys to limit to. types is an optional dict mapping keys to datatypes. kwargs are passed to json.load.

Note

read_json() is a convenience alias for ListOfDicts.read_json().

dataiter.read_npz(path, *, allow_pickle=True)

Return a new data frame from NumPy file path.

See numpy.load for an explanation of allow_pickle: https://numpy.org/doc/stable/reference/generated/numpy.load.html

Note

read_npz() is a convenience alias for DataFrame.read_npz().

dataiter.read_parquet(path, *, columns=[], dtypes={})

Return a new data frame from Parquet file path.

columns is an optional list of columns to limit to. dtypes is an optional dict mapping column names to NumPy datatypes.

Note

read_parquet() is a convenience alias for DataFrame.read_parquet().

dataiter.std(x, *, ddof=0, drop_na=True)

Return the standard deviation of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.std, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.std.html

>>> di.std(di.Vector([3, 6, 7]))
1.699673171197595
>>> di.std("x")
<function std.<locals>.aggregate at 0x7f7fcf2c84a0>

dataiter.sum(x, *, drop_na=True)

Return the sum of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

>>> di.sum(di.Vector([1, 2, 3]))
6
>>> di.sum("x")
<function sum.<locals>.aggregate at 0x7f90dcaac4a0>

dataiter.var(x, *, ddof=0, drop_na=True)

Return the variance of x.

If x is a string, return a function usable with DataFrame.aggregate() that operates group-wise on column x.

Uses numpy.var, see the NumPy documentation for details: https://numpy.org/doc/stable/reference/generated/numpy.var.html

>>> di.var(di.Vector([3, 6, 7]))
2.888888888888889
>>> di.var("x")
<function var.<locals>.aggregate at 0x7f58a1eb44a0>