DBnomics fetcher toolbox

DBnomics is a database of macro-economic data aggregated from a great number of world-wide providers (see the complete list).

Its usage is free of charge, its source code is licensed under the GNU Affero GPL v3+, and data is redistributed freely under the same conditions as the original provider.

In DBnomics, data acquisition is done by fetchers, small programs that download data from the provider infrastructure, and convert it to a common data model and format. Each fetcher covers a particular data provider. Fetchers are scheduled on a regular basis in order to keep DBnomics data up to date.

dbnomics-fetcher-toolbox is a Python package containing several tools and patterns to ease the development of a fetcher. It helps to capitalize between fetchers and reduce the burden in solving bugs.

Contents:

• Installation
• Design goals
  • Be self-contained
  • Store provider data as-is
  • Convert data to DBnomics data model
  • Keep past revisions
  • Avoid false revisions
  • Ensure revision consistency
  • Process resources
  • Error handling
  • Have a report on process execution
  • Scheduling
• Writing a fetcher
  • Install and configure environment
  • Good practices
  • Main steps of a script
  • Run the fetcher
  • Pin dependencies versions
  • Technical details
• dbnomics_fetcher_toolbox package
  • aiohttp_utils module
  • arguments module
  • cli module
  • data_model module
  • file_system_utils module
  • formats module
  • logging_utils module
  • parts module
  • resources module
  • sdmx_v2_0 module
  • status module
  • utils module

Indices and tables

• Index

• Module Index

• Search Page