There are many supplier types. Some suppliers ready to get your data format, another — don't; some use SOAP protocol, another — REST; some uses EAN for goods identification — another uses offer ID; some ready to give you order status, another — don't, some uses vocabulary, witch elements you need to map with your’ s master data, another — don't. Simply put — suppliers are extremely different.
However, we need to automate the data exchange. Now is the 21st century. In perfect world we need to found the B2B conference for suppliers and them customers (in auto part market in my case) and develop one data exchange format and be happy. Unfortunately, we live in imperfect world.
All suppliers have the same source idea: they ready to let us know about his assortment and ready to get our order. But all suppliers have different business-process and it has consequences for the API design and different architects and developers, who made final design, make the APIs extremely different.
How we can develop exchange with 50 suppliers? Develop 50 different modules? Let’s imagine: in best case scenario, we need 1 man-week for develop 1 exchange module and men-hour cost $35 for us. Did you multiply it? And it will be huge cost for support and update.
Do we have any another way? My answer is developing one exchange!
My method is come up with 51st supplier and develops exchange with him. It should be ideal supplier for our business, our processes. This supplier should have common features of all suppliers, but he should not exactly similar.
We need to take a look not from supplier, but from us, our business and this help us to simplify task. We have one business and one process and one design of system. If something doesn't match our business — we can ignore. Such approach bring us lose some logic, and we forced to decline some functionality, but this is price, which we pay for simplicity.
I'm classifying all simplifications on three type:
When i thought about design of my ideal 51st supplier, i read API documentation of all suppliers, which was available at that moment and qualified all the tricks by four categories
Actually, generalization is a main method of development of any functionality. Supplier exchange is not an exception.
For instance, one of our supplier have order status model: «received», «in progress», «sent», «cancelled», and another supplier have «in progress (manager)», «in progress (warehouse)», «in progress (logistic)», «closed (successful)», «closed (unsuccessful)». Another supplier has 27 statuses. I'm not joking, I've counted it. We can't process such different status models. We need to generalize statuses to minimal convenient set «new», «sent to supplier», «received», «trouble».
After it we just need to map each status of each supplier to our status.
We can't generalize everything. Sometime, we need to segregate functional.
For instance, if one supplier receive order as one document and another supplier receive order by e-shop basket style (add good — add good — make order) — we can't develop this functional by come general way.
In this case our exchange with 51st supplier should be segregated. In another words, our exchange with 51st supplier should be able to send order by two different ways optionally. I have a setting «order sending method» and two options: «whole order» and «line-by-line». By the way, i have a setting «status checking method» as well, and it has three options: «ask by order, receive by order», «ask by order, receive by line» and «ask by line, receive by line».
However, functional segregation is dangerous way: it leads as to complexity growing. And I strongly recommend to don't do tree-style segregation. Try to keep segregation options flat. If you make functional segregation option, which depends on another option — it leads to settings hell. Did you see the Boing control panel?
The cutting method is intuitive and we can use it in case if supplier functional is too rich for us.
For instance, supplier can give as possibility to edit order before some step in its business process.
But, we aren't interested in this functionality, all other suppliers don't give us such possibility and our business process doesn't involve editing step — we just need to state, that this functional is outboard.
Let's repeat: we need to look from our process and we need to build automatization border based in our business. All what out of border is needed to be excluded don't write the code for nothing.
The Default method is opposite by the cutting method. In this case, automatization border can include some functionality, which supplier's API doesn't support. For instance, some suppliers can don't support order statuses.
In this case, you don't need to take into consideration this fact for all next algorithms and make if-else statement based on some option like «supplier returns status». The better way is make default-value for status and to think that supplier status is finale-well status.
In this way you can to analyze your needs and your capabilities and make the automatization border. This is the end of designing of 51st supplier API. Technically, of course we will have 50 suppliers in the system, but the exchange code will the same and it will build XML for every supplier in the 51st supplier format.
I got focus on the order exchange in this article, because it's most variable and complex part of exchange (for me at least), but all described you can apply to order, master-data and whatever you want.
The next step — you need to think about exchange format for 51st supplier
In this way we can build file, which consist all the necessary data, but in not valid format.
It's time to move from theory to practice, from virtual 51st supplier to real end-supplier. I think, you all guess, that XML for 51st supplier we will convert to get XML valid for real supplier. And we need to use two stages concertation:
First of all we need to create data frame. XSLT language is quite convenient for this goal. It's not difficult language, you can learn it for couple days, but 99% of my conversations use only two instructions: value-of and for-each.
The next stage, after structure creation, we maybe need to convert it to another format (not XML, but JSON for instance). In this task, most difficult concertation is XML <-> JSON. I have many format converters, for instance XML to HTTP request, i.e. to string looks like?
item=12345&qty=4&price=19.50, but it's easy. In this step we should achieve message text, which valid for real end-supplier.
And finally, we need to send message. This functional i call Universal caller. It can hand over result of format converter and call REST and SOAP services. Caller should be able to handle different scenarios: as I’ve mentioned REST and SOAP, GET and POST, basic auth and OAUTH2.
In theory, you can isolate functionality of XSLT transformation, Format converter and Universal caller to separate instance. I've did this way. I call it Interface converter.
Code of Interface converter less than 300 lines, main settings (XSLT text, convert settings and calling settings) exist as data.
Process of order sending looks as follows:
All this actions happen on the fly, ERP thinks that it makes exchange with all suppliers in one format and it doesn't sense something is wrong. Of course, this approach is convenient only in case if your orders it not big. If you plan to move 1Gb price lists — you need to think about asynchronous plugin.
Even each exchange has a values, which necessary for nothing. Sometimes, it's because supplier has more functional than we need, for instance «percent for price increasement without additional coordination». Sometimes, it's just hard values: «you need to feel 4000 value for this field». «Why 4000? 4000 of what?». «Don't ask, just 4000». There is no necessary to hold this hard values in ERP, we have Enrichment functional the Interface converter. Actually, it's just static additional XML, which enrich each incoming XML before it is sent to XSLT converter.
Additional XML is separate for each supplier. Enrichment is done by easy gluing with common tag:
The mechanic of Interface converter lets us to decrease time-to-go for new supplier. Finally, we've got mechanic, which allow us to plug new supplier for few easy steps:
All this steps we are able to do for 3 hours. Honestly, i did few cases with «special» settings, but 95% supplier settings get less than 3 hours.
However, we need to automate the data exchange. Now is the 21st century. In perfect world we need to found the B2B conference for suppliers and them customers (in auto part market in my case) and develop one data exchange format and be happy. Unfortunately, we live in imperfect world.
All suppliers have the same source idea: they ready to let us know about his assortment and ready to get our order. But all suppliers have different business-process and it has consequences for the API design and different architects and developers, who made final design, make the APIs extremely different.
How we can develop exchange with 50 suppliers? Develop 50 different modules? Let’s imagine: in best case scenario, we need 1 man-week for develop 1 exchange module and men-hour cost $35 for us. Did you multiply it? And it will be huge cost for support and update.
Do we have any another way? My answer is developing one exchange!
My method is come up with 51st supplier and develops exchange with him. It should be ideal supplier for our business, our processes. This supplier should have common features of all suppliers, but he should not exactly similar.
We need to take a look not from supplier, but from us, our business and this help us to simplify task. We have one business and one process and one design of system. If something doesn't match our business — we can ignore. Such approach bring us lose some logic, and we forced to decline some functionality, but this is price, which we pay for simplicity.
I'm classifying all simplifications on three type:
- we need some functionality, but any suppliers can't give it us
- supplier needs some functionality (supplier doesn't let us to push the order), but it's not necessary for us
- two suppliers can implement some functional by extremely difference ways
When i thought about design of my ideal 51st supplier, i read API documentation of all suppliers, which was available at that moment and qualified all the tricks by four categories
Method 1: Generalization
Actually, generalization is a main method of development of any functionality. Supplier exchange is not an exception.
For instance, one of our supplier have order status model: «received», «in progress», «sent», «cancelled», and another supplier have «in progress (manager)», «in progress (warehouse)», «in progress (logistic)», «closed (successful)», «closed (unsuccessful)». Another supplier has 27 statuses. I'm not joking, I've counted it. We can't process such different status models. We need to generalize statuses to minimal convenient set «new», «sent to supplier», «received», «trouble».
After it we just need to map each status of each supplier to our status.
Method 2: Segregation
We can't generalize everything. Sometime, we need to segregate functional.
For instance, if one supplier receive order as one document and another supplier receive order by e-shop basket style (add good — add good — make order) — we can't develop this functional by come general way.
In this case our exchange with 51st supplier should be segregated. In another words, our exchange with 51st supplier should be able to send order by two different ways optionally. I have a setting «order sending method» and two options: «whole order» and «line-by-line». By the way, i have a setting «status checking method» as well, and it has three options: «ask by order, receive by order», «ask by order, receive by line» and «ask by line, receive by line».
However, functional segregation is dangerous way: it leads as to complexity growing. And I strongly recommend to don't do tree-style segregation. Try to keep segregation options flat. If you make functional segregation option, which depends on another option — it leads to settings hell. Did you see the Boing control panel?
Method 3: Cutting
The cutting method is intuitive and we can use it in case if supplier functional is too rich for us.
For instance, supplier can give as possibility to edit order before some step in its business process.
But, we aren't interested in this functionality, all other suppliers don't give us such possibility and our business process doesn't involve editing step — we just need to state, that this functional is outboard.
Let's repeat: we need to look from our process and we need to build automatization border based in our business. All what out of border is needed to be excluded don't write the code for nothing.
Method 4: Default
The Default method is opposite by the cutting method. In this case, automatization border can include some functionality, which supplier's API doesn't support. For instance, some suppliers can don't support order statuses.
In this case, you don't need to take into consideration this fact for all next algorithms and make if-else statement based on some option like «supplier returns status». The better way is make default-value for status and to think that supplier status is finale-well status.
Analisis result
In this way you can to analyze your needs and your capabilities and make the automatization border. This is the end of designing of 51st supplier API. Technically, of course we will have 50 suppliers in the system, but the exchange code will the same and it will build XML for every supplier in the 51st supplier format.
I got focus on the order exchange in this article, because it's most variable and complex part of exchange (for me at least), but all described you can apply to order, master-data and whatever you want.
And how exactly does it work?
The next step — you need to think about exchange format for 51st supplier
- first of all we need to use XML (I’ll explain it later)
- the next item — format should contain all data, which we have. In other words, we don't need to think: should we send the VAT rate to this concert supplier, we should to add as much data as we can
- and the last — all values, which is necessary to convert — should be converted. In other words, when we build XML for 51st supplier — all identifications should be converted to end-supplier values. It won't be difficult, because we have all necessary data in source system.
In this way we can build file, which consist all the necessary data, but in not valid format.
It's time to move from theory to practice, from virtual 51st supplier to real end-supplier. I think, you all guess, that XML for 51st supplier we will convert to get XML valid for real supplier. And we need to use two stages concertation:
- XSLT
- Format converter
First of all we need to create data frame. XSLT language is quite convenient for this goal. It's not difficult language, you can learn it for couple days, but 99% of my conversations use only two instructions: value-of and for-each.
The next stage, after structure creation, we maybe need to convert it to another format (not XML, but JSON for instance). In this task, most difficult concertation is XML <-> JSON. I have many format converters, for instance XML to HTTP request, i.e. to string looks like?
item=12345&qty=4&price=19.50, but it's easy. In this step we should achieve message text, which valid for real end-supplier.
And finally, we need to send message. This functional i call Universal caller. It can hand over result of format converter and call REST and SOAP services. Caller should be able to handle different scenarios: as I’ve mentioned REST and SOAP, GET and POST, basic auth and OAUTH2.
In theory, you can isolate functionality of XSLT transformation, Format converter and Universal caller to separate instance. I've did this way. I call it Interface converter.
Code of Interface converter less than 300 lines, main settings (XSLT text, convert settings and calling settings) exist as data.
Process of order sending looks as follows:
- ERP creates an order and sends it to Interface converter if 51st supplier format
- Interface converter receives XML and enrich it
- Interface converter makes XSLT transformation and format convert
- Interface converter sends message to end-supplier
- Supplier receives, process and give response for the order
- Interface converter receives the supplier's response
- Interface converter makes format convert and XSLT transformation (in the revert order)
- Interface converter send the answer to ERP in 51st supplier format
All this actions happen on the fly, ERP thinks that it makes exchange with all suppliers in one format and it doesn't sense something is wrong. Of course, this approach is convenient only in case if your orders it not big. If you plan to move 1Gb price lists — you need to think about asynchronous plugin.
Enrichment feature
Even each exchange has a values, which necessary for nothing. Sometimes, it's because supplier has more functional than we need, for instance «percent for price increasement without additional coordination». Sometimes, it's just hard values: «you need to feel 4000 value for this field». «Why 4000? 4000 of what?». «Don't ask, just 4000». There is no necessary to hold this hard values in ERP, we have Enrichment functional the Interface converter. Actually, it's just static additional XML, which enrich each incoming XML before it is sent to XSLT converter.
Additional XML is separate for each supplier. Enrichment is done by easy gluing with common tag:
<envelope>
<ERPData>
...
</ERPData>
<AdditionalData>
...
</AdditionalData>
</envelope>Conclusion
The mechanic of Interface converter lets us to decrease time-to-go for new supplier. Finally, we've got mechanic, which allow us to plug new supplier for few easy steps:
- Read the supplier API documentation
- Create and tune new supplier on ERP
- Describe convert rules at XSLT language
- Create the format converter rules
- Create the connection parameters
All this steps we are able to do for 3 hours. Honestly, i did few cases with «special» settings, but 95% supplier settings get less than 3 hours.