2023 AUGUST 01

In this month’s issue:

1. Problems with the MPC public server

2. MPC data sources

3. Suggestions for Users

1. Problems with the MPC public server

Over the last couple of days, starting August 30th, 2023, the MPC has experienced problems with our public server (specifically with Ruby on Rails). Those problems were preventing the use of a number of MPC online services, including receiving any submissions sent directly to our server via cURL or using our web forms (e.g. via https://minorplanetcenter.net/submit_xml or https://minorplanetcenter.net/submit_psv).
The MPC has now restored the server used to receive and process submissions: observations can now be sent again directly to https://minorplanetcenter.net/submit_obs, https://minorplanetcenter.net/submit_xml and https://minorplanetcenter.net/submit_psv via cURL or to https://minorplanetcenter.net/submit_xml and https://minorplanetcenter.net/submit_psv via web form. Observers who would like to submit their observations to the above urls are welcome to do so. If any problems are experienced or any unusual behavior is observed, please let us know via our Jira helpdesk.
We emphasize that the MPC is still experiencing other problems with our public server, in particular those services that require data from our public database. We are continuing to work on fixing these problems. For more updated information, please see our Status page.

2. MPC data sources

We would like to give a brief overview of the different data sources available from the MPC and how we are planning to switch to a more consistent set of information.

The website

Our most visible data source is given by the MPC website. The website contains useful information for both new and more experienced observers, links to our main services (e.g. Ephemeris Service, MPChecker) and orbits and observations for every single object (database search).

Important notes:

• The website runs on a combination of Ruby on Rails, Django, cgi, and HTML scripts. Most importantly, the data are currently fetched from a Maria database (MariaDB), an open source relational database.

• This MariaDB is not the primary source of data for MPC processing, but rather is a legacy system that is updated only at the time of each MPC publication (MPEC, mid-month or monthly circulars).

• As a consequence the data in the MariaDB is not kept exactly in-sync with the data contained in the MPC’s main postgres database already mentioned in our March and August (2023) Newsletters, which is kept updated constantly.

Suggestions for Users:

• If you use the MPC’s MariaDB as the primary source of your data, please try to move away from it and use the more recent and more frequently updated postgres database (see section Postgres Database).

  • We have already given instructions regarding postgres access and replication in our Newsletters and on our website. We plan to add more soon, but please use Jira to let us know if you have any questions or concerns.

  • We are working on creating new APIs that will supersede our db_search functionality. Those new APIs will be able to return designations, orbits and observations for every object in the MPC postgres database.

• Please note that the “obs_sbn” table in the postgres database, not the MariaDB, is the definitive source for observation data. The MariaDB could for example contain unwanted duplicate observations.

It is our goal to deprecate MariaDB, with a target date for turn-off of Jan 1st 2025. Before this date we will migrate to using our internal postgres database as the primary source for our website. This 2025 turn-off date is designed to allow the community sufficient time to adjust to the new changes.

The flat files

The MPC website includes many flat files that can be downloaded. These contain all of the orbits (MPCORB.dat) and observations for every object that the MPC has ever published. In addition to that, there are auxiliary files that are useful to understand the MPC processing and to fit your own orbits if needed. All the data are available from the DATA section of the MPC website.

Important notes:

• The MPC is moving towards a database-centric approach, where all the data will be derived and fetched from our internal postgres tables, but at the moment a large part of our code still relies on the presence of flat files. This includes all our publications.

• The MPC has spent a large amount of time comparing the data contained in the flat files with the ones in the postgres database. By now the number of differences between the two sources is definitely small. We are still working on fixing the latest discrepancies and we appreciate all the feedback that we can get from the community.

Our suggestions for Users:

• If you are using the flat files as the primary source of data, please try to move away from them and use the more frequently updated postgres database (see section Postgres Database).

  • Differences between flat files and postgres tables are now very tiny and you can consider the postgres tables as completely reliable.

• As stated before, please don’t try to compare flat files with the data from MariaDB.

Postgres Database

The MPC maintains an internal postgres database that contains, among other things, tables of observations and orbits. As of September 2023, the observation table contains more than 400 million observations. Even though the MPC system is still not completely database-centric, the database is updated when every process runs. We are working towards using the postgres tables as input for all our processes and data products. In addition to that, multiple scripts performing comparisons between the postgres tables and the flat files are run every day, to ensure full consistency between our main data sources.

Our internal postgres database is replicated to the Small Body Node: see also the MPC-Database. The replicated observations include all those that have been validated (via orbit-fitting) as belonging to a designated object, as well as those that are in the Isolated Tracklet File. Additional information on the table and examples of how to perform queries are provided on our website.

Important notes:

• Both MPC and SBN are working on creating APIs that will be able to directly query the MPC postgres database and the SBN replicated version, respectively. This will allow users to get all the data they need without having to use MariaDB or replicate the tables on their own laptops.

• More tables are going to be added for replication in the future. These include additional auxiliary tables, such as program codes, observatory codes and other information currently available only via flat files or website pages.

Our suggestions for Users:

• Please familiarize yourself with the new replicated tables, or at least with their content. More information is available through the tables, such as, for example, astrometric uncertainties when observations are submitted using the ADES format, and uncertainties on the orbital elements for MPC computed orbits.

• Please report any problems or discrepancies you encounter when working with these new data. Feedback is always welcome.

3. Suggestions for Users

Always apply for an observatory code

We require observers at all permanent observing sites to apply for an observatory code. The Roving Observer format should be used only by observers at temporary sites.

Instructions on how to apply for an observatory code and how to get started have been updated and are available on our website.

Please apply for an observatory code if you are planning to observe NEOCP objects.

Submit magnitude for NEOCP observations

We strongly encourage observers to submit magnitude values, especially in the case of NEOCP objects and new NEOs. Magnitudes represent a very valuable measure of the nature of the object, and as such they should not be neglected.