This release adds multi-source support, request of (#131): a tidync object can now present a consolidated lazy view across a collection of NetCDF sources, extending the filter-then-read model unchanged from one file to many.
tidync() now accepts a vector of file paths together with a
concat_dim argument, building a consolidated view across sources along
that dimension (typically time). Downstream operations (hyper_filter(),
hyper_array(), hyper_tibble(), hyper_tbl_cube()) work transparently
across the collection, and only the files needed for the current
selection are opened at read time. A single file with concat_dim is a
valid degenerate collection.
Fast mode for large collections:
tidync(files, concat_dim = "time", fast = TRUE) skips full metadata
scans of sources 2..N, reading only the concat dimension coordinate from
each file. Validation is deferred to data-read time, where dimension
mismatches are reported with the offending file named. In fast mode the
time coordinate is presented numerically (no timestamp column), since
per-source calendar metadata is not read; use the default full mode for
CFtime timestamps.
Zero file I/O construction: supply coordinate values directly with
concat_dim = list(name = "time", values = dates) and only the first
source is opened (as the template). Values can be numeric, Date, or
POSIXct, and hyper_filter() operates on them directly. This is aimed
at file-database workflows (for example raadfiles) where per-file
coordinates are already known. This construction assumes one step per
source; files holding more steps on the concat dimension are detected at
read time and reported as an error rather than silently subset.
Read-time validation is armed automatically for fast mode and for values-supplied construction: shared dimensions and the per-source concat dimension length are checked against the consolidated view as each file is opened, with clear errors naming the file.
Optional parallel reads via mirai: when mirai::daemons() are active,
per-source reads in hyper_array() run in parallel with
mirai::mirai_map(), falling back to sequential reads when mirai is not
installed or no daemons are set. Parallelism is entirely under user
control; tidync never starts daemons itself. Errors raised during
parallel reads (including validation failures) propagate normally.
Sources are kept in the order supplied; the consolidated index on the
concat dimension is file order. Value-based filters work regardless of
ordering, but supply sources in coordinate order for a monotonic view.
tidync() on a tidync_data object (round-trip) preserves the source
table and concat_dim.
hyper_array() time, consistent
with single-source behaviour. Non-contiguous selections that span whole
sources (for example dropping intermediate files) are supported and read
only the files required.Fixed hyper_tibble() producing constant dimension values when
drop = TRUE (the default) collapses arrays to bare vectors;
expansion is now based on length() rather than prod(dim()).
Fixed the print method showing NA for min/max of Date/POSIXct
coordinate columns; non-numeric coordinates are now converted with
as.numeric() (Date as days since epoch) rather than replaced with NA.
Repaired mangled pipe operators in the package overview help examples.
Removed commented-out browser() calls.
Update to CFTime as_timestamp(), thanks to @fabern.
Support for CF time metadata via package CFtime thanks to @pvanlaake, see https://github.com/ropensci/tidync/pull/124
Suppress ncdf4 warnings on open, issue #119.
Minimum R version is now 4.1.0 (native pipe |>). Removed forcats,
magrittr, and purrr dependencies; %>% replaced with |> throughout
and purrr::safely() replaced with tryCatch().
ncmeta (>= 0.5.0) is now required, for extended (CFtime) metadata and for correct handling of sources with no variables (such as compound-type L3 bin files), which errored uninformatively with earlier ncmeta under current dplyr.
Minimum dplyr version is 1.1.0 (multiple argument in joins) and
minimum tidyr version is 1.0.0 (cols argument in unnest()); all
version-gating conditionals removed.
mirai added to Suggests.
print.tidync now returns its input invisibly, as always claimed.
hyper_array() and hyper_tibble() and hyper_tbl_cube(),
and allowed user-controlled option to avoid this check. Thanks to Alessandro Bigi for feedback.tbl_cube() thanks to Hadley Wickham PR #102.Fix broken tests due to changes in RNetCDF, thanks to CRAN for notification.
Added example to readme and docs, and checks for non-unique hyper_filter() names, as per #93.
Thanks to @everydayduffy for the suggestion.
Now depends on R 3.5.0.
Removed obscure numeric 'what' logic for activate().
Fixed tidync method for hyper_array() output.
Found a really bad bug in hyper_array() (#92), now fixed. Axis order in
transforms was sometimes reversed, which caused garbled results from hyper_tibble().
The effect is to ruin any ggplot2 figures for some source files. It's very likely
that no other system yet uses the hyper_array() format so impact is low.
tidync is now part of the rOpenSci project.
Fixes in tests and examples to avoid version-incapable NetCDF problems on Solaris.
A number of improved tests and documentation fixes.
Deal with warnings from tidyr version > 0.8.3.
A huge thank you to all rOpenSci contributions, especially very helpful reviews from
@Nowosad and @timcdlucas as well as organizers @karthik, @stefaniebutland and @sckott.
Helpful input was also provided via issues from @adrfantini, @JustBerkhout, @matteodefelice, @rensa, @rmendels, @sw-rifai, and @tremenyi.
Package improvements thanks to CRAN feedback, clarified Description and added
more examples. Replaced cat() and print() calls with message() and warning().
New class 'tidync_data' for output of hyper_array(), no underlying change to
the object which is simply a list of arrays from each variable, and axis transforms
stored in an attribute.
Old deprecated function axis_transforms() now Defunct.
FIRST RELEASE, tidync was greatly improved via help from the rOpenSci review process.
New function hyper_grids() to report available grid names.
A printing error of dimension value ranges is now fixed, thanks to James Goldie (#84).
Now supports 'NC_CHAR' type, by exploding these into the array size expected.
Breaking change: when using tidync$gridit's now expected that this must be
tidyr::unnest()ed in order to expand out the grid list per variable, in line
with https://github.com/hypertidy/ncmeta/issues/26.
The hyper_array function now stores the relevant transforms table as an attribute
transforms so that objects can be constructed directly from the native array
output.
Function rename hyper_array() now matches hyper_tibble() indicating the form
of the output (rather than the action used, was hyper_slice()).
New functions hyper_vars() and hyper_dims() for reporting on the currently active
variables and dimensions and their status.
Now dependent on ncmeta >= 0.0.2, partly to avoid crashing on invalid source or file strings
Removed hyper_index() and incorporated that into hyper_filter(), there's
now only one delay-capable class which is 'tidync'
Function hyper_filter() now uses a selection idiom, to record the state of
the axis rather than explicitly filter it. This means we can have more flexibility
on what the axis transform tables can be used for, and removes some unwieldy
handling code. All the available axes are on the object from first contact, which
means we can program against the entire space in the source which will help for
complex mapping scenarios.
Function hyper filter() print now handles the case of char-type coordinate
values by setting the min and max to NA_real_
Various improvements and fixes for the print method for tidync
Support coordinate-less dimensions has been added, there is new information in the print summary about which dimensions are a "coord_dim" and this results in the axis transform tables using the index as the coordinate value.
Functions hyper_slice() and hyper_tibble() now return all variables that
exist within a grid
This version sees a new model where activation is on 'grids', effectively a space composed of dimensions. In addition to the variables, dimension, and attributes entities we add 'grid' defined by a set of dimensions, and 'axis' which is an instance of a particular dimension as used by a variable.
Sources without recognizable variables now gracefully handled, with help from ncmeta.
Now imports ncdump > 0.0.3.
Installed external example data from Unidata website
First working version now has tidync(), and hyper_*() family of functions.
Migrated from ncdump.