New function rb3_doctor() checks the health of the cache (#77). With a template, it finds the gaps in
the time series of a dataset layer (input or staging): the business days with no data, each with a
reason (never fetched, download failed, not read, no data, or read but not in the layer), so missing
dates can be fetched with fetch_marketdata(). It also reports stale download records (such as the old
b3-reference-rates records with curve_name), records whose raw file is missing, and raw files that
no record uses. It only reports and never changes the cache. See the new vignette "Checking the Cache".
Reading the download records is much faster (dget() on a text connection replaced by parse()).
?yc_get explains what to do with caches created before rb3 0.1.1, which still hold the old
"Taxas Referenciais" HTML files of b3-reference-rates: they are no longer read, and the dates can be
fetched again from the swap rates file (#144).
fetch_marketdata() ends with a summary: how many files have data, and the arguments of every file
whose download failed, that was not read, or that had no data (#86). It now returns (invisibly) a tibble
with template, download_args and status for each combination of arguments, instead of NULL.
New template b3-trade-information-consolidated (#126): B3's TradeInformationConsolidated file, with every
instrument of every segment (stocks, options, futures, forwards, ...) and every column of the file
(low, high, average and last price, oscillation, settlement price and rate, reference price, number of
trades, contracts, financial volume). B3 keeps only the last ~360 sessions (about 17 months): older dates
get an empty file and are reported as failed downloads.
fetch_marketdata() now downloads again a file that had no usable data (reported as invalid) when it
was downloaded less than 7 days after its refdate (#142). For example, settlement prices downloaded
before B3 published them are fetched again by the next call, instead of being lost. Files downloaded
later are final and are not retried.
fetch_marketdata() now downloads again the current year's file of b3-cotahist-yearly and
b3-indexes-historical-data when the cached copy is from an earlier day (#143). B3 updates these files
every session, so before this change the data stopped at the day of the first download. Past years are
still downloaded only once. Templates opt in with downloader.refresh-current-year: true.
An empty HTTP response (no Content-Type header) no longer stops fetch_marketdata() with "Unknown error";
it is saved as an empty file and reported as a failed download.
b3-reference-rates now downloads B3's "Taxas de Mercado para Swaps" file (TS<yymmdd>.ex_ on the
"Pesquisa por pregão" page) because the "Taxas Referenciais" page was discontinued (#125).
The curve_name argument is gone: one file holds every curve (about 120). The input layer schema
changed (refdate, curve_name, cur_days, biz_days, rate, partitioned by refdate), so delete
<cachedir>/db/input/b3-reference-rates and <cachedir>/db/staging/b3-reference-rates before
fetching again. yc_brl_get() now returns NA for r_360.b3-futures-settlement-prices now reads the "Consolidated trades of the session" table of the
Daily Market Bulletin (https://arquivos.b3.com.br/bdi/) because the "Ajustes do pregão" page was
discontinued (#125). The bulletin keeps only about the last 21 sessions. Raw files of the old
"Ajustes do pregão" HTML page already in the cache are still read, so reprocess = TRUE keeps the
history back to 2000.b3-bdi-derivatives-trades: the same Daily Market Bulletin table used by
b3-futures-settlement-prices, with every column (open, high, low, close, average and settlement prices,
number of trades, contracts, financial volume, best bid/ask, ...) and every instrument, options included.
Like its source, it covers only about the last 21 sessions; use b3-bvbg-086 for older dates.
b3-futures-settlement-prices is unchanged.fetch_marketdata() tries that date again.
Before, the page was saved, the meta was marked invalid and the date was never downloaded again..ex_ files are unpacked;
an empty archive is now reported as a failed download instead of stopping fetch_marketdata().rb3_clear_cache() deletes all downloaded files, datasets and metadata in the cache
folder, keeping the folders and the empty metadata table (#129).rb3_bootstrap() closes the open metadata connection, so changing rb3.cachedir and calling it again
uses the database of the new cache folder (#103).*_get()
help page).proto and is fully adapted to downloader and reader functions.arrow for efficient data storage and retrieval.rvest, proto, ascii, readxl and other packages that aren't heavy used.glue environment list requiring explicit passing as an environment.read_marketdata.test-company.R with exception handling for empty downloaded files.scraper-company.R.company_cash_dividends_get does not return all cash dividendsverifyssl. It defaults to TRUE, always use ssl, but when it is FALSE an option is set in httr to skip ssl verification.futures_get and futures_mget to handle equity futurescompany_cash_dividends_get, company_info_get, company_stock_dividends_get, company_subscriptions_get to get company's informationsindex_get to download historical data from B3 indexes (Issue #39)rb3.silent (defaults to FALSE) hide alert messages and progress barrb3.clear.cache (defaults to FALSE) remove files with invalid content from cache folderGetPortfolioDay_IndexStatistics historical time series for B3 indexesfutures_get and maturity2date to use calendar Brazil/BMFmaturity2date has a new argument refdate, it must be passed when converting old maturities like JAN0, FEV0, ...code2month and maturity2date now accept old 4 characters maturity code, before 2006cotahist_options_by_symbol_superset joins options data, equity data and interest rates for each option for a given symbol (Issue #50)read_marketdata lost the argument cachedir, the RDS file with parsed data is saved in the directory of the given file.
do_cache = FALSE to not save the RDS file, it defaults to TRUE.COTAHIST_YEARLY, it uses cache wrongly (Issue #44)rates was renamed to implied_rateen to templates: COTAHIST_*NegociosIntraday intraday listed market tradesNegociosBalcao intraday OTC (Debentures) tradesNegociosBTB intraday lending tradesrb3.cachedir to set default cache directory in order to use cached files across multiple sessionsGetStockIndex to get the composition of B3 indexes.GetTheoricalPortfolio to get composition and weights of B3 indexes.GetPortfolioDay to get composition, weights and segments of B3 indexes.CenariosCurva for scenarios of term structures of interest rates.CenariosPrecoReferencia for reference prices scenarios.IndexReport indexes daily market data.PriceReport daily prices and market data.GetListedSupplementCompany supplement data for listed companies.GetDetailsCompany to get companies details (name, codeCVM, ...).GetListedCashDividends to get list of cash dividends.yc_ipca_get and yc_ipca_mget for real interest ratesyc_usd_get and yc_usd_mget for USD interest rates in Brazilcotahist_get_symbols to get stocks by a list of symbolscotahist_etfs_get, cotahist_fiis_get, cotahist_fidcs_get, cotahist_fiagros_getcotahist_funds_get has been replaced by these ones.index_comp_get returns the index compositionindex_weights_get returns the index weightsindex_by_segment_get returns indexes assets grouped by segmentsindexes_get lists the available indexesindexes_last_update returns the date when the indexes have been updatedindexreport_get and indexreport_mget download index report datacotahist_equity_options_superset joins options data, equity data and interest rates for each option - this is useful to run option and volatility models.yc_superset, yc_usd_superset, yc_ipca_superset mark futures maturities in yield curve.sapply with purrr::map_xxx.destdir = NULL to convert_to function.yc_get / yc_mget and futures_get / futures_mget (Issue #26).fields creation (Issue #27).GetStockIndex, JSON file with relations between stocks and indexes.