Wider Meeting: SOWG #10 - Jul 2017
- David Williams
Gethyn Lewis (SWA/UCL)
Francisco Espinosa (EPD/U Alcalá)
- Isaias Carrasco (MAG)
Koen Stegen (EUI/ROB)
Gordon Hurford (STIX/FHNW)
- Nathan Rich (SoloHI/NRL)
- Jim Raines (SWA/U Michigan)
- Vincenzo Andretta (METIS/INAF)
- Chandrasekhar Anekallu (SWA/UCL-MSSL)
- Ali Varsani (IWF Graz)
- Gill Watson (SWA/UCL-MSSL)
- Oliver Grimm (STIX/ETHZ-FHNW)
- Lauri Panitzsch (EDP-STEP/University of Kiel)
- Christoph Terasa (EDP-STEP/University of Kiel)
- Christian Drews (EDP-SiS/University of Kiel)
- Alessandra Giunta (SPICE/STFC RAL Space)
- Sonny Lion (RPW/LESIA-Obs de Paris)
- Dennis Wang (SoloHI/NRL)
- Terje Fredvik (SPICE)
- Outstanding actions.
- Proposal - Products which can never be known to be complete. Mark as third category, neither I not C.
- Keywords/variable to ignore in LLVM tests to move to ICDs.
- VM outputs in both products and failed
VM delivery/testing status.
|Actions||The action outstanding was on Andrew Walsh to investigate the presence of duplicate files - one complete, one incomplete for each product in EPD output. On examination after meeting, it seems that this was a mis-reading by the SOC of the output. Action will be closed.|
|Unknowable completion category|
RC described the issue, approximately as follows:
Currently all product files output by LL VMs are marked (in the file name) as either C=Complete or I=Incomplete. As well as simply adding clarity to the status of a product, this aids the SOC in deciding how to treat versions of the same product (both same type and same observation/pseudoday) produced by a VM when processing data from different passes (because the successful downlink for that product is not complete in a single pass). For example, when a complete product results from a later pass, any earlier incomplete product is superseded. When a complete product exists, an incomplete product created later (packets may occasionally be delivered twice) can be discarded.
However the C/I distinction is premised on the assumption that if we (or a VM) have in fact all the packets corresponding to a product, then we can know that we have all the packets. Thus we can label the product as complete. Otherwise we know it is incomplete.
However the SOC understands that there are some products for which it is never possible from telemetry to know whether they are complete. These are products for which the total number of corresponding packets created cannot be derived. For such products it would never be possible to know (except in very special circumstances) that they are complete. As such these products can never be labelled Complete, and the SOC considers that in these cases it is deceptive to label them as Incomplete.
An example would be a 'Burst Product' which covers an unpredictable period of a pseudo-day, and where instrument housekeeping does not report the necessary information to determine how many corresponding packets were generated.
The SOC proposes that such products always include a 'U' (in the absence of a rival suggestion) in place of the 'C' or 'I'.
Thus there would be two categories of product. Firstly, products for which it is, in normal circumstances, possible to know whether the product is complete or not. These are marked in the filename as C or I. Secondly there are products for which in normal circumstances it is not possible to know whether they are complete. These will have a U rather than C or I in the filename.
Instruments for which C and I designations are fully appropriate would not have to make any changes to their pipelines.
NB It was not sure that the above was fully understood in the telecon. If the above description does not make sense to you, please let the SOC know.
|Ignore list -> ICD|
RC indicated that the SOC intends to move the definition of which FITS keywords and CDF variables are ignored when comparing test output to reference. Instead of in the SEGU, this information will be places in the respective ICDs.
There was no objection to this.
|Both products and failed|
RC pointed out that in some circumstances it makes sense for a LL VM to output, in response to a processing request, both to products and to failed. Most obviously, this would be expected in cases where some products could successfully be generated, but others could not. The SOC was only now taking this possibility into account, but however some instruments were already planning on taking this approach - or indeed considered it obvious.
IC asked if the SOC would be routinely monitoring VM log files for (for example) warnings. RC indicated that the SOC was not planning to routinely do this. Rather the logs are seen as a place to seek further details when a failure occurred (either failed output, or no output). KH indicated that this corresponded to his approach. He expected EUI log files to be largely for EUI interpretation. However IC wanted to know if there was a mechanism for a VM to present a warning. RC will discuss this inside the SOC and respond.
|Delivery Status||The status of the currently delivered VMs for each instrument was reviewed as usual. This was recorded in the status table distributed with these minutes.|
NR asked what the next milestone for the SOC was re the Low Latency. RC indicated that our next focus was on the post-processing - thus the need for products and VMs with which to work.
It was decided that the next Telecon should be on Weds 27th Sept at 15:00.
No new actions.