This document records a history of all issues found with CaSSIS data. These issues more relate to the implementation of the PDS standard and in particular the xml label files rather than the data files.
CaSSIS Raw data is being ingested into the PSA daily and is processed by a data processing pipeline (DPS) at ESAC. Currently the latest version of all Raw data files are version 2.0, even if this is the first version, as this assignment was given to all data following a bulk re-processing at ESAC in late 2019/early 2020 and is now assigned operationally to all new data.
Since version 2.0 of the data has been produced there have been ongoing improvements and bug fixes to the data processing, most significantly following the first CaSSIS data review in April 2020. This means for example that a file produced in May 2020 will contain improvements compared to one produced in February 2020 but both will be data version 2.0, therefore it is also useful to look at the Data Processing System (DPS) version in the file (see below).
The CaSSIS team do not deliver calibrated data in PDS format but instead use an internal xml based format. At ESAC a parser is run to convert the CaSSIS data into PDS4 files and hence the calibrated products are also assigned version 2.0 currently.
All accumulated improvements will be applied in the next bulk re-processing expected at the end of 2020 and over the summer of 2020, due to the accumulated issues with this version, all version 1.0 data will be deleted from the PSA.
Data Delivery Status
- Raw data without full (geometry) labels are generated daily
- The labels are updated with the geometry and instrument status information following deliveries to the ExoMars team at ESAC from the CaSSIS team at Bern
- CaSSIS deliveries from Bern also include calibrated framelets and stitched calibrated products
- All products have associated browse products except for the calibrated framelets, here the stitched product is considered the reference
- Currently CaSSIS have delivered calibrated products up to February 22nd 2020
Following the first CaSSIS data review in April 2020, all CaSSIS products are public six months after observation date.
The table below lists the known current and previous issues with the CaSSIS data. For the fixed issues please take note of the DPS version and the date. The date should be used for guidance and may be out by a few days as the release is deployed. The DPS version can be found in the Mission Area of the label e.g.
<psa:processing_software_title>EM16 Data Processing System</psa:processing_software_title>
|C3||Currently there are multiple problems associated with the 'on the fly' generated collection and bundle inventory files. The issues are understood, how to implement to the fix is in analysis.||2.0||In analysis||P1381|
|C4||The .csv inventory files do not end with CRLF as required by the PDS standard||2.0||Fixed in PSA 5.10||P1462|
|C5||PDS is changing its LID convention for instruments. This will require a change from|
Coordination is required to make this update as it has an impact on ingestion and the PSA database. Coordination between all missions and instruments is required.
|2.0||Will be done with next bulk reprocessing in late 2020||A|
is specified. This needs to be corrected to:
|2.0||Pending input from CaSSIS team||A|
Currently all files have:
It is being studied if we can insert either 'targeted' and 'ride-along' pointing types plus an appropriate description into these attributes.
|2.0||In Analysis, low priority||D|
|C14||While it is not a requirement for CaSSIS calibrated data to work with ISIS, what is required to make it work with ISIS is under study.||2.0||In analysis by CaSSIS team||A|
|When Fixed||Data Processing Fix Version|
|Version 2 Issues|
|F2.1||NASA URLs were quoted as the http address in CaSSIS labels. NASA has supported both http and https but are planning to phase out http. CaSSIS data now quotes the https address.||2.0||2020-01-20||DPS-3.4|
The CASSIS calibrated product should contain 'Calibrated' instead of 'Raw' in the Processing Level attribute of the PDS4 label.
Note: This issue affected deliveries from CaSSIS in the range April-September 2018.
|F2.3||The current context lid_reference|
has been deprecated by PDS. The new context/LID will be:
The pipeline was setting 'Center' for all the pixel intercepts. The order of the pixel intercepts is the order of the corners in our xml headers is the following: (left upper, right upper, right lower, left lower)
In the <DERIVED_HEADER_DATA Description="Derived data from HK"> section. The field geometry_reference_time_utc is now correctly implemented.
|F2.7||CaSSIS data should be SingedLSB2 very old files used SignedByte, more recent UnsignedMSB2 (see O1.4)||2.0||2020-05-11||DPS-3.9|
In the CaSSIS data files from Cruise phase, a NASA PDS target logical identifier (LID) was not available for Landolt SA 104 and Theta Ceti
Therefore the following local context with LIDs were generated:
Since then official PDS context with LIDs have been created:
The CaSSIS observations of these objects will be updated to use the PDS LIDs in future.
Labels currently do not have:
which provides Display_Direction.
|F2.10||A request has been made to add descriptions to HK fields. These are listed in the EAICD, adding descriptions to the data products is under study.||2.0||2020-06-10||DPS-3.10|
|F2.11||In the FTP the orbit range does not match the orbits. For example, directory with: /Orbit_Range_2201_2300/ actually contain directories to orbits 2200 -> 2299.||2.0||2020-06-10||DPS-3.10|
|Version 1 Issues|
|O1.1||Version 1 does not have accurate times associated with the observations, hence in some cases the version 2 files will show a different orbit number.||1.0|
|O1.2||In very old Raw data files the lid_reference:|
was implemented. This was soon replaced by the PDS version:
|O1.3||The data type in old Raw products is incorrectly specified as SignedByte, newer products have specified UnsignedMSB2||1.0||2018-11-20||DPS-2.4.2|
|O1.4||In old Raw products the logical identifier has an illegal capital letter T in the time part of the LID.||1.0||2018-12-13||DPS 2.5|
|O1.5||In old Raw products the record length in the label may not be correct and be lower than the location of the last field||1.0||2018-12-13||DPS 2.5|
|O1.6||The file size specified in old Raw product labels is incorrect and is lower than the actual file size||1.0|
In very old Raw data the context LID reference is incorrect:
It has been changed to the correct reference:
When specifying multiple schematrons, old versions of data files only specify the schematypens once but this needs to be done each time a schematron is specified e.g.
<?xml-model href="http://pds.nasa.gov/pds4/pds/v1/PDS4_PDS_1800.sch" href="http://pds.nasa.gov/pds4/geom/v1/PDS4_GEOM_1700_1401.sch" href="http://psa.esa.int/psa/em16/v1/PDS4_PSA_EM16_1000.sch" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<?xml-model href="http://pds.nasa.gov/pds4/pds/v1/PDS4_PDS_1800.sch" schematypens="http://purl.oclc.org/dsdl/schematron"?>
Old Raw files have
This has been deprecated by NASA and for CaSSIS has been replaced by: