Processing predicates of BT-761-Lot and BT-771-Lot

[akashSrivastava-SAP]

Hi,

We encountered a scenario where our visual model (of noticeType - 17) contained the fields BT-761-Lot and BT-771-Lot.
Their xpaths are :
BT-761-Lot : "/*/cac:ProcurementProjectLot[cbc:ID/@schemeName='Lot']/cac:TenderingTerms/cac:TendererQualificationRequest[not(cac:SpecificTendererRequirement)]/cbc:CompanyLegalFormCode"

BT-771-Lot : "/*/cac:ProcurementProjectLot[cbc:ID/@schemeName='Lot']/cac:TenderingTerms/cac:TendererQualificationRequest[not(cbc:CompanyLegalFormCode)]/cac:SpecificTendererRequirement[cbc:TendererRequirementTypeCode/@listname='missing-info-submission']/cbc:TendererRequirementTypeCode"

Our code generated the below xml as it was unaware of the predicates:

<cac:TendererQualificationRequest>
    <cbc:CompanyLegalFormCode listName="required">false</cbc:CompanyLegalFormCode>
    <cac:SpecificTendererRequirement>
         <cbc:TendererRequirementTypeCode listName="reserved-procurement">none</cbc:TendererRequirementTypeCode>
    </cac:SpecificTendererRequirement>
</cac:TendererQualificationRequest> 

The xml which it should have generated (as we figured) was :

<cac:TendererQualificationRequest>
     <cbc:CompanyLegalFormCode listName="required">false</cbc:CompanyLegalFormCode>
</cac:TendererQualificationRequest>
<cac:TendererQualificationRequest>
     <cac:SpecificTendererRequirement>
         <cbc:TendererRequirementTypeCode listName="reserved-procurement">none</cbc:TendererRequirementTypeCode>
     </cac:SpecificTendererRequirement>
</cac:TendererQualificationRequest> 

Can you suggest guidelines for dealing with predicates or any algorithm to determine when to create new elements(cac:TendererQualificationRequest) and when to override existing ones?

[rouschr]

Hello,

<cac:TendererQualificationRequest> (this is "ND-TendererLegalForm")
     <cbc:CompanyLegalFormCode listName="required">false</cbc:CompanyLegalFormCode> (this is "BT-761-Lot")
</cac:TendererQualificationRequest>

<cac:TendererQualificationRequest> (this is "ND-LateTendererInformation")
     <cac:SpecificTendererRequirement>
         <cbc:TendererRequirementTypeCode listName="reserved-procurement">none</cbc:TendererRequirementTypeCode> (this is "BT-771-Lot")
     </cac:SpecificTendererRequirement>
</cac:TendererQualificationRequest> 

It seems you are using xpath to generate the XML, as you have seen, using xpath and ignoring predicates leads to problems.
To solve these problems we added what we call nodes that are found in the xmlStructure of the fields.json.
Looking at the SDK fields.json xmlStructure and fields (and their parentNodeId and repeatable boolean) you get this hierarchy:

"ND-Lot" (repeatable true), there can be many of these node instances in parallel, (LOT-0001, LOT-0002, ...) in the XML this is cac:ProcurementProjectLot

	"ND-LotTenderingTerms" (repeatable false), in the XML this is cac:TenderingTerms (unique in a lot instance)
	
		"ND-TendererLegalForm" (repeatable false), in the XML this is cac:TendererQualificationRequest
			BT-761-Lot
			
		"ND-LateTendererInformation" (repeatable false), in the XML this is cac:TendererQualificationRequest
			BT-771-Lot

As you can see while both fields "BT-761-Lot" and "BT-771-Lot" use the XML element "cac:TendererQualificationRequest",
the fields "BT-761-Lot" and "BT-771-Lot" have a different "parentNodeId"
"BT-761-Lot" has parentNodeId "ND-TendererLegalForm", and the node has parentId "ND-LotTenderingTerms"
"BT-771-Lot" has parentNodeId "ND-LateTendererInformation", and the node has parentId "ND-LotTenderingTerms"
As the node "ND-TendererLegalForm" is not repeatable and the node "ND-LateTendererInformation" is not repeatable, this also indicates that they are separate instances of the XML element "cac:TendererQualificationRequest"

So for everything under a repeatable node (repeatable true), something like "ND-Lot" in our example, you could start by creating the XML element of a field and then build the parent (container) XML elements around it until you reach their repeatable parent node (container).
So you would find fields like BT-... then look for their parentNodeId, build the container around the field, look for the node parentId until you find a repeatable node.
Another way could be to keep knowledge about the nodeId of an xml element, so that you can differentiate the instances of the "cac:TendererQualificationRequest", for example inside the XML element itself (as an attribute that you remove later) or in a map.
The temporary attribute also allows to visualise the difference in the generated XML if you keep it for debugging purposes.

For example:
<cac:TendererQualificationRequest nodeId="ND-TendererLegalForm"> ... fieldId="BT-761-Lot" ...</...>
<cac:TendererQualificationRequest nodeId="ND-LateTendererInformation"> ... fieldId="BT-771-Lot" ...</...>

This will not be valid in terms of the schema, but you can ignore validation and use it to debug or during generation before validation, just an idea.

[akashSrivastava-SAP]

Didn't expect this ellaborate explanation , really appreciate it. This helps. Thanks

[akashSrivastava-SAP]

Hi @rouschr , sorry for hijacking this old thread but we wanted to discuss a similar issue. We followed the idea of considering predicates while building xml document from the xpaths. But this isn't working for us in one particular scenario where we have both BT-10-Procedure-Buyer and BT-610-Procedure-Buyer. They have the following relative xpaths
BT-10-Procedure-Buyer = "cac:ContractingActivity/cbc:ActivityTypeCode[@listname='authority-activity']"
BT-610-Procedure-Buyer = "cac:ContractingActivity/cbc:ActivityTypeCode[@listname='entity-activity']"

If you see cac:ContractingActivity doesn't really have any differentiating predicates for the above 2 fields and neither the 2 fields have different parent nodes. Because of this incorrect xml is being generated.
We need to have 2 separate ContractingActivity elements here.

<cac:ContractingParty>
  <cac:ContractingActivity>
     <cbc:ActivityTypeCode listName="authority-activity">defence</cbc:ActivityTypeCode>
     <cbc:ActivityTypeCode listName="entity-activity">electricity</cbc:ActivityTypeCode>
  </cac:ContractingActivity>
</cac:ContractingParty>

can you give some suggestion to handle this ?

[akashSrivastava-SAP]

Shouldn't the 2 fields have xpaths like :
BT-10-Procedure-Buyer:

"xpathAbsolute": "/*/cac:ContractingParty/cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='entity-activity'])]/cbc:ActivityTypeCode[@listName='authority-activity']",
"xpathRelative": "cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='entity-activity'])]/cbc:ActivityTypeCode[@listName='authority-activity']"

BT-610-Procedure-Buyer:

"xpathAbsolute": "/*/cac:ContractingParty/cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='authority-activity'])]/cbc:ActivityTypeCode[@listName='entity-activity']",
"xpathRelative": "cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='authority-activity'])]/cbc:ActivityTypeCode[@listName='entity-activity']"

I see that similar solutions have been applied in other xpaths too.

[rouschr]

Hello again,

you should use the "fields.json" of the SDK, there you have fields and nodes.
We will not take form business rules into account here to focus only on the algorithm to generate XML, so we will be assuming your input is correct, the output should be done like this:

• you are in the correct context (following the nodes and fields from fields.json)
• some repeatable nodes have an identifier
• if repeatable you should create a new instance for each identifier (and reuse existing one if same identifier)

you have to parse the relative path and create intermediary XML element so /a/b/field-xyz you need to create "a", then "b" inside then put field-xyz inside. You also need a bit of extra logic for the attributes. But once you have this it will be always the same logic.

ContractingActivity must be repeated as it is part of the relative path:
"xpathRelative" : "cac:ContractingActivity/cbc:ActivityTypeCode[@listname='authority-activity']/@listname",

So for each cbc:ActivityTypeCode, always first build a cac:ContractingActivity, then put cbc:ActivityTypeCode inside.
You need some code to handle the building of the relative path as there could be more than one intermediary XML element

In fields.json you have the fields and the xpathRelative, use the xpath relative:

{
    "id" : "BT-10-Procedure-Buyer-List",
    "parentNodeId" : "ND-ContractingParty",
    "name" : "Activity Authority Listname",
    "btId" : "BT-10",
    "xpathAbsolute" : "/*/cac:ContractingParty/cac:ContractingActivity/cbc:ActivityTypeCode[@listName='authority-activity']/@listName",
    "xpathRelative" : "cac:ContractingActivity/cbc:ActivityTypeCode[@listName='authority-activity']/@listName",
    "xsdSequenceOrder" : [ { "cac:ContractingActivity" : 4 }, { "cbc:ActivityTypeCode" : 2 } ],
    "type" : "text",
    "attributeName" : "listName",
    "attributeOf" : "BT-10-Procedure-Buyer",
    "presetValue" : "authority-activity",
    "businessEntityId" : "buyer",
    "legalType" : "CODE",
    "repeatable" : {
      "value" : false,
      "severity" : "ERROR"
    }
  } ...

And this:

{
    "id" : "BT-610-Procedure-Buyer-List",
    "parentNodeId" : "ND-ContractingParty",
    "name" : "Activity Entity Listname",
    "btId" : "BT-610",
    "xpathAbsolute" : "/*/cac:ContractingParty/cac:ContractingActivity/cbc:ActivityTypeCode[@listName='entity-activity']/@listName",
    "xpathRelative" : "cac:ContractingActivity/cbc:ActivityTypeCode[@listName='entity-activity']/@listName",
    "xsdSequenceOrder" : [ { "cac:ContractingActivity" : 4 }, { "cbc:ActivityTypeCode" : 2 } ],
    "type" : "text",
    "attributeName" : "listName",
    "attributeOf" : "BT-610-Procedure-Buyer",
    "presetValue" : "entity-activity",
    "businessEntityId" : "buyer",
    "legalType" : "CODE",
    "repeatable" : {
      "value" : false,
      "severity" : "ERROR"
    }
  } ...

Both have the same parent node id "ND-ContractingParty":

{
    "id" : "ND-ContractingParty",
    "parentId" : "ND-Root",
    "xpathAbsolute" : "/*/cac:ContractingParty",
    "xpathRelative" : "cac:ContractingParty",
    "xsdSequenceOrder" : [ { "cac:ContractingParty" : 29 } ],
    "repeatable" : true, <------------------------------------------ REPEATABLE
    "identifierFieldId" : "OPT-300-Procedure-Buyer",
    "captionFieldId" : "BT-500-Organization-Company",
    "businessEntityId" : "buyer"
  }

The node is "repeatable" : true, which means it must be repeated if the id differs (see identifierFieldId)
The id is somewhere in "/*/cac:ContractingParty/cac:Party/cac:PartyIdentification/cbc:ID"

If the id differs:

<cac:ContractingParty>
  ... id ORG-0001
  <cac:ContractingActivity>
     <cbc:ActivityTypeCode listName="abc">def</cbc:ActivityTypeCode>
  </cac:ContractingActivity>
</cac:ContractingParty>

<cac:ContractingParty>
  ... id ORG-0002
  <cac:ContractingActivity>
     <cbc:ActivityTypeCode listName="bla">bli</cbc:ActivityTypeCode>
  </cac:ContractingActivity>
</cac:ContractingParty>

If the id is the same you get something like you can find in the SDK examples:

	<cac:ContractingParty>
		<!--BT-508-->
		<cbc:BuyerProfileURI>https://profile.example.com</cbc:BuyerProfileURI>
		<cac:ContractingPartyType>
			<!--BT-11-->
			<cbc:PartyTypeCode listName="buyer-legal-type">body-pl</cbc:PartyTypeCode>
		</cac:ContractingPartyType>
		<cac:ContractingActivity>
			<!--BT-10-->
			<cbc:ActivityTypeCode listName="authority-activity">econ-aff</cbc:ActivityTypeCode>
		</cac:ContractingActivity>
		<cac:Party>
			<cac:PartyIdentification>
				<!--OPT-300-->
				<cbc:ID schemeName="organization">ORG-0001</cbc:ID>
			</cac:PartyIdentification>
			<cac:ServiceProviderParty>
				<cbc:ServiceTypeCode listName="organisation-role">ted-esen</cbc:ServiceTypeCode>
				<cac:Party>
					<cac:PartyIdentification>
						<cbc:ID schemeName="organization">ORG-0007</cbc:ID>
					</cac:PartyIdentification>
				</cac:Party>
			</cac:ServiceProviderParty>
		</cac:Party>
	</cac:ContractingParty>

[rouschr]

Also ideally create a new discussion next time.

[rouschr]

You can also have a look at:
https://docs.ted.europa.eu/eforms/latest/guide/xml-generation.html

[rouschr]

Shouldn't the 2 fields have xpaths like : BT-10-Procedure-Buyer:

"xpathAbsolute": "/*/cac:ContractingParty/cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='entity-activity'])]/cbc:ActivityTypeCode[@listName='authority-activity']",
"xpathRelative": "cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='entity-activity'])]/cbc:ActivityTypeCode[@listName='authority-activity']"

BT-610-Procedure-Buyer:

"xpathAbsolute": "/*/cac:ContractingParty/cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='authority-activity'])]/cbc:ActivityTypeCode[@listName='entity-activity']",
"xpathRelative": "cac:ContractingActivity[not(cbc:ActivityTypeCode[@listName='authority-activity'])]/cbc:ActivityTypeCode[@listName='entity-activity']"

I see that similar solutions have been applied in other xpaths too.

Here we have specific fields for each case (BT-10-Procedure-Buyer and BT-610-Procedure-Buyer).
When you generate the XML you know which field you are dealing with.
The use of "not" would make it a bit simpler to parse but is not required.
So the fields.json gives you all the needed information to build the XML from nodes, fields and relative xpath (avoid relying on absolute xpath). You are always is a narrow context (a node or a field) and you always have a parent (ok except the root...).
I recommend you read the guide: https://docs.ted.europa.eu/eforms/latest/guide/xml-generation.html