A recent discussion with a team member pointed me in a direction that the word API as used today seems to be squarely targeted at what I understand as Web APIs or REST APIs. Upon continuing that discussion further I understood that there is no clear demarcation in the definitions for certain terminologies. I am compelled to write this post to clarify my understanding of this topic to myself and perhaps this may be useful to others as well. This post is necessarily very long so that all connected terms are defined in one place. I have had a few discussions around "Reuse", "Modularity" and "Composability" with my friends and team members in the past and I attempt to lay down the terminologies and the definitions from the standpoint of "Reuse", "Modularity" and "Composability". All of these are connected to decomposition.
Decomposition is an essential tenet of computational thinking. Decomposition means breaking down a large or complex problem into simpler, smaller ones. Specifically when decomposing a larger problem into smaller ones each part needs to be identified as an independent unit that can be solved. These individual units then need to be composed back together to solve the larger problem. The act of identifying and creating smaller independent units is modularizing or creating modules and making those modules act together is composing. Practical difficulties or convenience may make the modules less independent or even interdependent. When only such modules are truly independent they become reusable.
Reusability is when you write code once and use it again. This follows the DRY principle. Additionally it improves maintainability of code. Testing and bug-fixing is simplified which directly improves productivity. Code reuse also has a direct impact on time-to-market because it reduces development time and effort. Reusability can be at many levels and in this post we are specifically discussing only code reusability.
Time to elaborate on this point and along the way clarify some common terminologies associated with these concepts in the software engineering world. Some often-heard terms in the field of software engineering that are closely related and whose definitions overlap (many have been in existence perhaps for the entire history of software engineering) include:
- Functions / Procedures / Methods / Subroutines
- Libraries
- APIs
- Toolkit
- Framework
- SPI
- Namespaces
- Component
- SDK
- Platform
In software engineering, the solution to a problem is represented using a language that programmers can understand such as Python, JavaScript, Java, C, C++, C#, or any of the hundreds of the languages that have been invented over time. Decomposition helps the solution from getting out of hand (complex and large). A properly decomposed solution is easier to program and maintain. Modern software programs are essentially built as functions. Where each function solves a very specific problem. A large or complex solution is typically decomposed into manageable units called functions. Some languages call them subroutines, some prefer to call them procedures, some refer to them as functions, and yet others as methods. These functions are composed back together to form the solution. Functions typically take some input called arguments and return some output or an exception. There are consumers of functions and functions are essentially consumed by other functions. Functions may or may not be independent. Several functions may be interdependent and work together to deliver the desired functionality. It wholly depends on how the solution to the problem was decomposed.
So in that sense Functions / Procedures / Methods / Subroutines are the basic units of programming and are the direct result of decomposing a solution to the problem that one tries to address in software engineering. Additionally, it is simpler to only expose the declaration of the function which consists only of the name, input arguments, output, and possible exceptions. The actual implementation, i.e., work the function does with its input to produce the output is not of any concern to the consumer of the function. This way consumers of the functions don't have to concern themselves with the inner working providing them a higher level of abstraction.
For example, consider a function that raises an invoice. It relies on a pricing function to get the price and calculate the totals of the invoice. Invoices also include taxes. Taxes may be provided by a different function and an input for the taxes function may be state, county, city, and so on. The invoice function does not have to burden itself with the knowledge of tax calculation. The programmer's mind time is also freed from the concern of calculating taxes in the "Create Invoice" function. The "Obtain Taxes" function is practically a black box.
Consider that there is a need to order the line items of the invoice in alphabetical order. Then a sort function is necessary which takes as input perhaps a list of line items and returns as output a sorted list of line items. Depending on Space and Time complexity requirements you may choose to change the sort function at a later date. Also, the sort function is not specific to "invoicing". It may be useful anywhere. If the function is written in such as way that there is no interdependence on other parts of invoicing, the sort function may be used in a context or domain outside of invoicing. This way functions tend to become "reusable".
A collection of such functions around a common purpose are typically called libraries. Libraries can contain more than just functions. Artifacts contained within a library depend on the language of choice. For instance, C libraries may typically have structs and functions within a library while C++ may also have classes and templates contained within libraries. Libraries tend to be specific to the language and/or the operating system. Libraries have co-evolved with compiler designs. Compilers are typically used to convert programs written in human-readable form or source code to a machine-readable form or object code. Programs are written to be executed and compilers know which function to execute first, typically the main function. Libraries are however not written to be executed but to be included in other programs. But still are compiled and distributed as object code. Complex conventions involving libraries have evolved such as calling conventions, sharing between multiple programs, and distribution mechanisms.
An Application Programming Interface (API) in the traditional sense is again nothing but functions. However, they acquire a conflated meaning depending on the context in which they are created and being used. A single function could be an API or multiple functions working in concert to provide a functionality could be an API. In practice software categorized as APIs are of the latter type. Inherently APIs are also libraries by definition. An API typically consists of multiple functions and the state is shared across these functions somehow. Data structures or data entities serve as the medium of shared state.
What is the state? The values of a variable which is shared between two functions. The variable is said to hold a state when one function sets a value and another function depending on the value in that variable can decide the execution path. Now the two functions share a state. Now when two functions share a state they also become interdependent. Sometimes such dependencies are necessary and are advantageous.
To understand APIs better, consider a set of functions used to provide networking functionality. A typical high-level situation would be an applications programmer creating server listening on a specific port. A function creates a network socket that listens on a port. Another function handles incoming connection requests to that port. These two functions are dependent on each other and at runtime are also dependent on the port in which the socket is listening. This intricate pattern of dependencies and shared state is a characteristic of APIs. The internal workings of an API need not be known and may as well be hidden from the consumer of the API. Java Servlet API is a classic example of an API. An API only exposes its interface to its consumers. The interface of an API is nothing but its input, output, and any exceptions. In that sense here the interface contains one or more function declarations.
The reason why an API hides the implementation details from its consumers is that there may be a need to switch implementations without the consumer of the API ever being aware of the switch. Consider the example of the sort function described earlier. Let us say you choose to change the sorting algorithm for a better space and time complexity, you can switch the implementation without ever invoicing function realizing that the sorting algorithm was changed. Invoice only depends on the declaration between the two sort functions being the same (declaration being the input, output, and any exceptions). There are many advantages to this which may not be immediately obvious.
Toolkits are typically large APIs delivering more than one related functionality. For instance, Abstract Windowing Toolkit (AWT) in Java is a toolkit. It is purpose-built for providing Graphical User Interface (GUI) support in Java language. It consists of an API for GUI artifacts such as windows, menus, buttons, text fields, etc, a set of related APIs to capture input events for those artifacts, and APIs to draw the graphical elements of the GUI on-screen. In its totality, it is a toolkit. It is not the only GUI toolkit, one can use Swing or Eclipse SWT or other toolkits with Java. Outside of Java, QT is a popular windowing toolkit used for building cross-platform GUI applications. If a toolkit is characterized as a collection of API then why not simply call it an API rather than create a new term for it. IMHO the difference is in the way toolkits are created. Toolkits typically are a. self-contained and b. cannot be switched as the implementation of an API can be. Toolkits are purpose-built for not being switchable. As an example, you cannot switch AWT with SWT or vice-versa.
The difference between an API and a Toolkit is that APIs are purpose-built for implementations to be switched and Toolkits are purpose-built for being used as-is. But both share the characteristic of extensibility. Let us explore extensibility when discussing frameworks next.
A very simple way to define frameworks would be that they are opposite to APIs. APIs are written to be consumed by programs directly. Frameworks work the other way around. Frameworks consume the function definitions written by you. It is a way of creating a generic shell and the programmer can provide the specializations which would be used by the framework as necessary. This is generally called "the Hollywood principle". Don't call us, we will call you. You don't call the framework code, it calls your code. This provides extensibility. Consider the Java Servlet API example provided above. This API is used for writing applications. If all the work of standing up a web server needs to be written for every web application, programmers would be spending time writing system code and not solving the business problem. There are three distinct parts to a web application. A webserver is one end of the system listening to requests from a web browser. Then there is the business application which reacts to the request received from a web browser and sends an appropriate response back. And the glue binding the webserver and the business application together which is the framework here. Every business application is unique. The framework is written in a way such that as soon as the webserver receives the request it calls the framework function and in reaction to the request the framework "calls the code written by the programmer". You are extending the framework by providing functions that would be called by the framework.
For this to work successfully a contract has to be defined between the framework and the "to-be" application code which would be written in the future by another programmer. This contract is typically a function declaration or in API parlance an interface. The contract defines which function will be called, in which sequence or order it will be called and what arguments will be passed in. But only the declarations exist in the framework the other end of this contract is a black box. The actual implementations are provided by the programmer. The consumer here is the framework. The framework is consuming the function created by you but those functions should follow the declarations imposed by the framework. As you can see at a very high level, only the black box differs.
A Service Provider Interface or an SPI is one more level in the API/Framework list of artifacts. Sometimes both the sides of a framework are black boxes and only a contract exists. This kind of nuance exists when commercial software is involved which requires a standardized mechanism for application programmers to interact with. A classic example would be the JDBC API. This is an API for database access. The JDBC API defines a contract for application programmers to use when making a connection to databases and running requests against databases. While one end of this API is used by the consumer the other end of the API is also used by another consumer (service provider). Typically the database vendor in the case of the JDBC API. Database vendors have a specific contract to follow. This is defined through the SPI.
With so many consumers and service providers, one can see that we can very quickly run out of meaningful names. At the beginning of this article, I had spoken about identifying the decomposed parts. Unless names are unique it is very difficult for software programmers and compilers to differentiate between same-named artifacts. Many humans have the same name, but we tell apart from one "Joe Bloggs" from another naturally. When you use two libraries or APIs and both have a function with identical names, arguments, returns, and exceptions how does the compiler differentiate? Namespacing helps here. It is a technique of adding identification to function to differentiate between the two.
As you can see reusability is a direct benefit of decomposition. Regardless of whether it is a library of simple functions, an API Framework, or a Toolkit these are reusable. But there is one more flavor of reusability. Programmers use a variety of tools to program. The key tool being an Integrated Development Environment (IDE). Typically when creating GUI applications IDEs are an indispensable tool. IDEs provide us with a visual design tool that allows us to construct the GUI. Consider using a toolkit within an IDE. Toolkits provide artifacts such as a button or text field which you place on a canvas and customize to provide size, color labels, and so on. This is called the "design time" as opposed to when the finished application is running where it is called the "runtime". The runtime visuals are prepared by the programmer during the design time. The toolkit has to provide a mechanism for it to be recognized by the IDE and "advertise" the capabilities that the artifacts provide during design time. This type of reusable artifact responds to a design-time contract and provides a runtime contract and the due complexity is called a component. But GUI components are the only example, components can be very simple and basic that it is only one function or it can be complex and big enough to carry within it its framework.
Software Development Kit (SDK) is a collection of Frameworks, APIs, libraries, compilers, assemblers, debuggers, performance monitoring tools, profilers, and tools needed by programmers. When a language or any other software that allows it to be programmed is published it consists of these tools essential for the programmer. It may or may not include an IDE. Usually, an IDE includes one or more SDK. Examples include Java Development Kit or Java Software Development Kit more specifically Java SDK.
The platform is generally a combination of an operating system and a processor. For example, WINTEL is Windows/Intel platform. Why both are combined is because software is specific to both. When compilers compile an program the translation process will produce object code that contains a sequence of microcode executable by the targeted processor architecture and also the operating system. For instance, a simple function to compute a hmac key may consist only of processor instructions but that key may have to be saved to the disk which would include calls to disk IO functions exposed by the OS. The disk IO function itself may be having function calls to device drivers to access the physical blocks on the disk. A classic case is that software written for Intel/Mac OS platform cannot be run on the new M1/Mac OS platform without the Rosetta 2 translation layer. The same is the case on Linux too. Linux or Linux applications cannot be run straight away on Raspberry PI (ARM Processor) without recompiling it.
Finally revisiting libraries, they are a packaging and distribution mechanism and not just a collection of functions. APIs, Frameworks, Toolkits, SPI, and components are also called libraries. The word "Library" is the most generic of terms meaning both the collection of reusable software artifacts, the distribution mechanism, and the distributed artifact itself.
With this, we come to the close of this article of a long and arduous list of definitions. Necessarily this begs to define Web APIs then which would be part of another article. Also, this article tried to keep the definitions as simple as possible. The moment we touched APIs we are entering the realm of object orientation and I have deliberately tried not to touch programming paradigms. As you might have come to appreciate the level of reusability and the flexibility each technique provides are also substantively different. Hope this will address nuanced differences in these terms and provide clarity. As always pls comment and follow.
So in that sense Functions / Procedures / Methods / Subroutines are the basic units of programming and are the direct result of decomposing a solution to the problem that one tries to address in software engineering. Additionally, it is simpler to only expose the declaration of the function which consists only of the name, input arguments, output, and possible exceptions. The actual implementation, i.e., work the function does with its input to produce the output is not of any concern to the consumer of the function. This way consumers of the functions don't have to concern themselves with the inner working providing them a higher level of abstraction.
For example, consider a function that raises an invoice. It relies on a pricing function to get the price and calculate the totals of the invoice. Invoices also include taxes. Taxes may be provided by a different function and an input for the taxes function may be state, county, city, and so on. The invoice function does not have to burden itself with the knowledge of tax calculation. The programmer's mind time is also freed from the concern of calculating taxes in the "Create Invoice" function. The "Obtain Taxes" function is practically a black box.
Consider that there is a need to order the line items of the invoice in alphabetical order. Then a sort function is necessary which takes as input perhaps a list of line items and returns as output a sorted list of line items. Depending on Space and Time complexity requirements you may choose to change the sort function at a later date. Also, the sort function is not specific to "invoicing". It may be useful anywhere. If the function is written in such as way that there is no interdependence on other parts of invoicing, the sort function may be used in a context or domain outside of invoicing. This way functions tend to become "reusable".
A collection of such functions around a common purpose are typically called libraries. Libraries can contain more than just functions. Artifacts contained within a library depend on the language of choice. For instance, C libraries may typically have structs and functions within a library while C++ may also have classes and templates contained within libraries. Libraries tend to be specific to the language and/or the operating system. Libraries have co-evolved with compiler designs. Compilers are typically used to convert programs written in human-readable form or source code to a machine-readable form or object code. Programs are written to be executed and compilers know which function to execute first, typically the main function. Libraries are however not written to be executed but to be included in other programs. But still are compiled and distributed as object code. Complex conventions involving libraries have evolved such as calling conventions, sharing between multiple programs, and distribution mechanisms.
An Application Programming Interface (API) in the traditional sense is again nothing but functions. However, they acquire a conflated meaning depending on the context in which they are created and being used. A single function could be an API or multiple functions working in concert to provide a functionality could be an API. In practice software categorized as APIs are of the latter type. Inherently APIs are also libraries by definition. An API typically consists of multiple functions and the state is shared across these functions somehow. Data structures or data entities serve as the medium of shared state.
What is the state? The values of a variable which is shared between two functions. The variable is said to hold a state when one function sets a value and another function depending on the value in that variable can decide the execution path. Now the two functions share a state. Now when two functions share a state they also become interdependent. Sometimes such dependencies are necessary and are advantageous.
To understand APIs better, consider a set of functions used to provide networking functionality. A typical high-level situation would be an applications programmer creating server listening on a specific port. A function creates a network socket that listens on a port. Another function handles incoming connection requests to that port. These two functions are dependent on each other and at runtime are also dependent on the port in which the socket is listening. This intricate pattern of dependencies and shared state is a characteristic of APIs. The internal workings of an API need not be known and may as well be hidden from the consumer of the API. Java Servlet API is a classic example of an API. An API only exposes its interface to its consumers. The interface of an API is nothing but its input, output, and any exceptions. In that sense here the interface contains one or more function declarations.
The reason why an API hides the implementation details from its consumers is that there may be a need to switch implementations without the consumer of the API ever being aware of the switch. Consider the example of the sort function described earlier. Let us say you choose to change the sorting algorithm for a better space and time complexity, you can switch the implementation without ever invoicing function realizing that the sorting algorithm was changed. Invoice only depends on the declaration between the two sort functions being the same (declaration being the input, output, and any exceptions). There are many advantages to this which may not be immediately obvious.
Toolkits are typically large APIs delivering more than one related functionality. For instance, Abstract Windowing Toolkit (AWT) in Java is a toolkit. It is purpose-built for providing Graphical User Interface (GUI) support in Java language. It consists of an API for GUI artifacts such as windows, menus, buttons, text fields, etc, a set of related APIs to capture input events for those artifacts, and APIs to draw the graphical elements of the GUI on-screen. In its totality, it is a toolkit. It is not the only GUI toolkit, one can use Swing or Eclipse SWT or other toolkits with Java. Outside of Java, QT is a popular windowing toolkit used for building cross-platform GUI applications. If a toolkit is characterized as a collection of API then why not simply call it an API rather than create a new term for it. IMHO the difference is in the way toolkits are created. Toolkits typically are a. self-contained and b. cannot be switched as the implementation of an API can be. Toolkits are purpose-built for not being switchable. As an example, you cannot switch AWT with SWT or vice-versa.
The difference between an API and a Toolkit is that APIs are purpose-built for implementations to be switched and Toolkits are purpose-built for being used as-is. But both share the characteristic of extensibility. Let us explore extensibility when discussing frameworks next.
A very simple way to define frameworks would be that they are opposite to APIs. APIs are written to be consumed by programs directly. Frameworks work the other way around. Frameworks consume the function definitions written by you. It is a way of creating a generic shell and the programmer can provide the specializations which would be used by the framework as necessary. This is generally called "the Hollywood principle". Don't call us, we will call you. You don't call the framework code, it calls your code. This provides extensibility. Consider the Java Servlet API example provided above. This API is used for writing applications. If all the work of standing up a web server needs to be written for every web application, programmers would be spending time writing system code and not solving the business problem. There are three distinct parts to a web application. A webserver is one end of the system listening to requests from a web browser. Then there is the business application which reacts to the request received from a web browser and sends an appropriate response back. And the glue binding the webserver and the business application together which is the framework here. Every business application is unique. The framework is written in a way such that as soon as the webserver receives the request it calls the framework function and in reaction to the request the framework "calls the code written by the programmer". You are extending the framework by providing functions that would be called by the framework.
For this to work successfully a contract has to be defined between the framework and the "to-be" application code which would be written in the future by another programmer. This contract is typically a function declaration or in API parlance an interface. The contract defines which function will be called, in which sequence or order it will be called and what arguments will be passed in. But only the declarations exist in the framework the other end of this contract is a black box. The actual implementations are provided by the programmer. The consumer here is the framework. The framework is consuming the function created by you but those functions should follow the declarations imposed by the framework. As you can see at a very high level, only the black box differs.
A Service Provider Interface or an SPI is one more level in the API/Framework list of artifacts. Sometimes both the sides of a framework are black boxes and only a contract exists. This kind of nuance exists when commercial software is involved which requires a standardized mechanism for application programmers to interact with. A classic example would be the JDBC API. This is an API for database access. The JDBC API defines a contract for application programmers to use when making a connection to databases and running requests against databases. While one end of this API is used by the consumer the other end of the API is also used by another consumer (service provider). Typically the database vendor in the case of the JDBC API. Database vendors have a specific contract to follow. This is defined through the SPI.
With so many consumers and service providers, one can see that we can very quickly run out of meaningful names. At the beginning of this article, I had spoken about identifying the decomposed parts. Unless names are unique it is very difficult for software programmers and compilers to differentiate between same-named artifacts. Many humans have the same name, but we tell apart from one "Joe Bloggs" from another naturally. When you use two libraries or APIs and both have a function with identical names, arguments, returns, and exceptions how does the compiler differentiate? Namespacing helps here. It is a technique of adding identification to function to differentiate between the two.
As you can see reusability is a direct benefit of decomposition. Regardless of whether it is a library of simple functions, an API Framework, or a Toolkit these are reusable. But there is one more flavor of reusability. Programmers use a variety of tools to program. The key tool being an Integrated Development Environment (IDE). Typically when creating GUI applications IDEs are an indispensable tool. IDEs provide us with a visual design tool that allows us to construct the GUI. Consider using a toolkit within an IDE. Toolkits provide artifacts such as a button or text field which you place on a canvas and customize to provide size, color labels, and so on. This is called the "design time" as opposed to when the finished application is running where it is called the "runtime". The runtime visuals are prepared by the programmer during the design time. The toolkit has to provide a mechanism for it to be recognized by the IDE and "advertise" the capabilities that the artifacts provide during design time. This type of reusable artifact responds to a design-time contract and provides a runtime contract and the due complexity is called a component. But GUI components are the only example, components can be very simple and basic that it is only one function or it can be complex and big enough to carry within it its framework.
Software Development Kit (SDK) is a collection of Frameworks, APIs, libraries, compilers, assemblers, debuggers, performance monitoring tools, profilers, and tools needed by programmers. When a language or any other software that allows it to be programmed is published it consists of these tools essential for the programmer. It may or may not include an IDE. Usually, an IDE includes one or more SDK. Examples include Java Development Kit or Java Software Development Kit more specifically Java SDK.
The platform is generally a combination of an operating system and a processor. For example, WINTEL is Windows/Intel platform. Why both are combined is because software is specific to both. When compilers compile an program the translation process will produce object code that contains a sequence of microcode executable by the targeted processor architecture and also the operating system. For instance, a simple function to compute a hmac key may consist only of processor instructions but that key may have to be saved to the disk which would include calls to disk IO functions exposed by the OS. The disk IO function itself may be having function calls to device drivers to access the physical blocks on the disk. A classic case is that software written for Intel/Mac OS platform cannot be run on the new M1/Mac OS platform without the Rosetta 2 translation layer. The same is the case on Linux too. Linux or Linux applications cannot be run straight away on Raspberry PI (ARM Processor) without recompiling it.
Finally revisiting libraries, they are a packaging and distribution mechanism and not just a collection of functions. APIs, Frameworks, Toolkits, SPI, and components are also called libraries. The word "Library" is the most generic of terms meaning both the collection of reusable software artifacts, the distribution mechanism, and the distributed artifact itself.
With this, we come to the close of this article of a long and arduous list of definitions. Necessarily this begs to define Web APIs then which would be part of another article. Also, this article tried to keep the definitions as simple as possible. The moment we touched APIs we are entering the realm of object orientation and I have deliberately tried not to touch programming paradigms. As you might have come to appreciate the level of reusability and the flexibility each technique provides are also substantively different. Hope this will address nuanced differences in these terms and provide clarity. As always pls comment and follow.
No comments:
Post a Comment