From Artificial Neural Network (ANN) to Generative Pre-trained Transformer (GPT): Evolution of Large Language Model

!! This article is generated by Chat GPT 4.0 !!

Traces of the evolution of AI from Artificial Neural Networks (ANNs) to models like GPT involves covering a vast landscape of developments in machine learning and neural network architectures. Each step in this evolution brought new capabilities and understandings. Here's a revised overview with technical summaries of each key development.

1. Artificial Neural Networks (ANNs)

• Technical Summary: ANNs consist of layers of interconnected nodes (neurons) that simulate the way biological brains process information. Each connection has a weight, which is adjusted during the training process to minimize the difference between the actual output and the predicted output.

• Evolution: Early ANNs like perceptrons were limited to simple linear tasks. The advent of multi-layer networks and backpropagation algorithms in the 1980s allowed ANNs to learn from complex data, laying the foundation for modern deep learning.

2. Deep Learning and Convolutional Neural Networks (CNNs)

• Technical Summary: Deep learning involves ANNs with multiple layers (deep networks) for feature extraction and transformation. CNNs, a class of deep neural networks, are specifically designed for processing data with a grid-like topology (e.g., images). They use convolutional layers to filter inputs for useful information.

• Evolution: CNNs, exemplified by models like AlexNet, significantly advanced fields like image and video recognition, enabling systems to identify and classify content within images with high accuracy.

3. Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) Networks

• Technical Summary: RNNs process sequences of data by maintaining a 'memory' of previous inputs in their internal state. LSTMs, an advanced RNN architecture, address the vanishing gradient problem of standard RNNs, allowing them to learn long-term dependencies.

• Evolution: LSTMs improved the performance of models on sequential data, particularly in language processing tasks like translation and speech recognition.

4. Transformer Models and the Attention Mechanism

• Technical Summary: Transformers, introduced in the "Attention Is All You Need" paper, use an attention mechanism to weigh the influence of different parts of the input data. Unlike RNNs, they process data in parallel, significantly improving efficiency.

• Evolution: The Transformer model, through architectures like BERT and GPT, revolutionized NLP, allowing for more sophisticated understanding and generation of human language.

5. Generative Pre-trained Transformer (GPT) Series

• Technical Summary: GPT models are large-scale transformer-based models pre-trained on vast amounts of text data. They use unsupervised learning to generate human-like text and can be fine-tuned for specific tasks. Each successive version of GPT has increased in model size and complexity, enhancing its capability.

• Evolution: GPT models, especially GPT-3, demonstrated breakthroughs in generating coherent and contextually relevant text, answering questions, and even coding, showcasing the immense potential of transformer-based architectures in AI.

6. Beyond GPT: Emerging Trends and Future Directions

• Technical Summary: Post-GPT developments are focusing on enhancing efficiency, context-awareness, and ethical considerations in AI. This includes exploring smaller, more efficient models, multi-modal learning (combining text, images, and other data types), and addressing issues like bias and fairness in AI.

• Evolution: The future of AI promises advancements in AI ethics, explainability, and integration into various aspects of human life, marking a shift towards more responsible and versatile AI applications.

Conclusion

From simple ANNs to the sophisticated GPT models, AI has undergone remarkable transformations. Each stage of evolution has built upon the last, progressively enhancing the capabilities and applications of AI. As we move forward, the focus is shifting towards more efficient, ethical, and contextually intelligent AI systems, promising a new era of innovation and integration into everyday life.

What is Design Thinking

A hypothetical conversation is taking place in a conference room between a software engineer and a business user.

"I need to have the development team available and ready twenty-four by seven during the filing period. This is Fed mandated SLA and we would have to react within 4 to 24 hours. If any approval is needed to do immediate deployment to the production, secure the necessary management approval upfront.", said the business user.

"Do you really need the development and production support team to seat at their desk and waiting to jump in to reintegrate the financial models into the production environment? What problem you are trying to solve here? Are you looking for a way to have the changed models reintegrated into the production environment within a short period period of time to meet the stringent Fed mandates SLA?" The software engineer replied with an empathic voice. Further adding to it by proposing a potential solution to that problem, "How about we provide you a self-service capability to reintegrate the models into the production system? You can do that anytime you want it and any number of times you need it."

"That sounds interesting but I don't want anyone to change the production system anytime without a proper approval", the business user reacted in a receptive tone.

"I don't want that either", the Software Engineering Manager is now chipping into the conversation, "We can enforce four-eyes check but let's talk more about the detail before we jump into the final solution", and has steered the discussion towards finding the right solution.

Though this may be a hypothetical conversation but certainly you have seen the similar conversation where the business user approaches the software engineering or product development team with a "brilliant" IT solution of a business problem without even mentioning what business problem the user was trying to solve. However, the goals of the software engineering team should be to steer the conversation towards understanding the users' pain points, find the fundamental problem and then propose the right solution.

To me, this is the essence of Design Thinking.

Design Thinking is a not the new guy in the town even though its reincarnation sounds just like that. I don't want to spend whole lot about its historical aspect but let's put just enough history for the sake of giving a context.

Design Thinking as a concept came into existence in the late sixties when Herbert A. Simon published his book, "The science of the Artificials". This got into the mainstream through the establishment of Stanford University's Design School.

Before delving into the detail of the Design Thinking, let's first clarify, "what's Design?"

Design, though it sounds like the surface or outward appearance of a thing, however, this concept of design is furthest from that vain outwardly look and feel. IBM Design Thinking defines Design as "The Intent behind the outcome". But the most intricate definition of Design came from the man who had changed the way we perceive the computer products, Steve Jobs, who once said in his interview that the reason he doesn't like the Microsoft's product because "it doesn't have the taste", and defined Design as "...the fundamental soul of a man-made creation that ends up expressing itself in successive outer layers". And Design Thinking is the art of creation of Design.

Now, let's take the words from two other most influential persons who have helped the Design Thinking to come to its current state: Don Norman, the author of "The Design of Everyday Things", has described the Design Thinking as "...Designers resist the temptation to jump immediately to a solution for the stated problem. Instead, they first spend time determining what basic, fundamental (root) issue needs to be addressed. They don't try to search for a solution until they have determined the real problem, and even then,, instead of solving that problem, they stop to consider a wide range of potential solutions. Only then will they finally converge upon their proposal. This process is called design thinking." and Tom Brown, the founder of IDEO, has defined Design Thinking as "...a human-centered approach to innovation that draws from the designer's toolkit to integrate the needs of people, the possibilities of techno technology, and requirements for business success".

In the second part of this post on Design Thinking, I will cover the method of Design Thinking and shed some light on the IBM Design Thinking and finally on how the Agile development methodology can coexist with Design Thinking.

Micro blog: Designing computer as our brain is designed

When we learn a new skill, such as, playing violin, driving or swimming, a set of neurons is used to execute the instructions and when done repeatedly, they are kind of hardwired to perform that job. That's why when we drive or walk, we actually don't think consciously but our subconscious mind executes most of the tasks to get the job done. It's like task is hardwired in our brain neurons. How about we design our computer memory and processors' transistors to act similarly. That would make a computer very much efficient and faster in processing. It was not practical at the early age of computers due to the cost of memory and processing units. As the memory is getting cheaper and cheaper, and the microprocessors are cramming double amount of transistors in every eighteen months, the execution of a software can now easily be allocated dedicatedly to a certain set of memories and processing units and reuse that set of memories when that particular function is executed. Currently it does similar thing in the memory when a software program is loaded but not by actually forming a physical connectivity among the memory cells and processor's transistors. This would need to create kind of physical/pseudo physical connectivity among those memory chips and processors. In this way, the hardware would behave like software in physical form. There could be so much optimization to efficiently utilize the hardware.

Technology Trend Series: Machine Learning

In the simplistic term, Machine Learning (ML) is all about making machine that can learn from surrounding and apply that learning in action. The more formal definition is from Tom Mitchell: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” Machine Learning is now has reached to the point where we actually don’t even realize its presence unless explicitly called out. Google, Amazon, IBM, Apple, Microsoft, Netflix, Uber, Tesla, etc. have adopted Machine Learning successfully. Google uses ML for spam filtering, image recognition, translation, self-driving car. Amazon uses ML for Recommendation system, Drone delivery, etc. Apple’s Siri, Microsoft’s Cortana and Amazon’s Alexa and Google’s Google Home use ML to provide Virtual Private Assistance (VPA). IBM’s Watson provides assistance to doctors to diagnose patients. Uber and Tesla created Machine Learning based autonomous vehicles. Machine Learning is used in the stock market for High Frequency Trading and in the postal service for handwriting recognition.

The Machine that Learn is essentially a computer algorithm to best model the input to its output. Carefully prepared large training dataset is fed into the algorithm to find out the optimal model. Machine Learning can be Supervised Learning, Unsupervised Learning, Reinforcement learning, Artificial Neural Network, Deep Learning, and so on. Let’s take an example of how a self-driving car uses machine learning: car is driven by a human driver and through various sensors: steering wheel, accelerator, brake, camera, GPS, radar, Laser, etc. it captures the data and the behavior of the human driver and used as training data set and find the optimal model to map the observed data from all the sensors to driver’s action.Machine Learning is progressing faster and superseding its own legacy. The drivers for continued massive growth and adoption of Machine Learning are the growing surge in data volume and complexities that conventional engineering approaches are increasingly unable to handle. For example, the amount of data created in the last 2 years exceeds the data created in the entire human history. So the traditional brute force computation is no longer a viable option to process this ocean of data. Meanwhile, as predicted in the “Moore’s Law”, the exponential growth of computing power is unleashing the power of Deep Learning. So, the businesses have to be fully prepared to utilize the most out of it to stay competitive in this disruptive and emerging technology landscape.

Machine Learning is ubiquitous in commercial usage and has reached to “Adolescent”, according to Gartner’s Maturity Level. Enterprises that have not yet started adopting the technology should start training and hiring talents in Machine Learning and Data Science and building infrastructure. The precursor of successful Machine Learning adaptation is to be able to manage “Big Data” for training the machines with the right training dataset. Enterprises need to invest more in dedicated Center of Excellence (CoE) for the “Emerging and disruptive Technology” like Machine Learning and embrace the transformation into their core business. For individual computer professionals, this is the time when the focus on learning the concept of Machine Learning and Artificial Intelligence, be familiarize with development tools and technologies (Matlab, R, Apache Mahout, Python ML libraries, etc.), specially, around data and algorithms, and finally dust off the the math and statistics skills long sitting unused. Those days aren't far away when the Java and .NET programmers would be considered as legacy programmers like the way the COBOL programmers were being considered.

Applying Agile in the snow cleaning job

We have just survived through the Nor'easter "Juno", and to my biggest relief, without any power disruption. As forcasted, the snow accumulation was more than 2 feet. I had cleaned 3 feet of snow in past but this time I thought to apply one of the Agile principle for the snow cleaning job which is: Deliver increments that add value to the customer (i.e. myself) in a small iteration. 

Let's go into detail of my Agile project. I have defined "Cleaning snow of my house" as the Epic. And here are the stories: 

1. Clean the entrance stair so that we can come out of our house through main entrance
2. Clean the snow of the driveway in front of the garage so that I can take out my van, which I use to drive on the highway
3. Clean the snow of the other driveway so that I can take the sedan out so that I can drive inside the city
4. Clean the sidewalk, which I have to do as a city law and also my kids would use that to go to school
5. Clean the bus stop at the end of the sidewalk, so that people can use that stop
6. Clean the patio snow, so that we can use the patio door to go out for fresh air
7. Clean the extra two spot of my drive way so that the guests can park the car

 I had started first prioritizing the stories based on the business value, to be said in simplistic term, how fast can I take my car out of my garage and driveway and then make a way out from the house. Also, I have re-prioritized the stories while progressing through the backlog of stories. I had also dropped the story#5: cleaning the bus stop due to the fact that the cost is too high for the benefit. Also, the snow plower guy from the city always dump the snow on the stop, so there's no point of cleaning this hardest part without getting sustaining benefit.

I have tried the Agile snow cleaning just for fun but after doing that I found it's actually fun and beneficial. It was "beneficial" because I was seeing the value of my work quicker and it was "fun" because I didn't have time to be tired by thinking too much down the road as I was taking a small bite at a time

Disclaimer: I have written this short post in March, 2015 after the massive snow apocalypse but kept it in Draft status as I didn't think this as valuable but while reviewing it today, I changed my mind and decided to publish.

Book Review of “Blink: the power of thinking without thinking”, Gladwell, Malcolm (2007)

“Blink” defies the long standing common sense and conventional wisdom within our society and, particularly, within our professional workspace, by putting an intriguing and thought provoking statement at the very beginning of the book as the cover page subtitle, i.e. “The Power of Thinking without Thinking”. The journey begins from there and ends through the six chapters of the book where the author, Malcolm Gladwell, explores various aspects of human psychology while making the case for his core propositions, i.e. the ingenuity of human subconscious, through his powerful skills of storytelling.

 

Gladwell started the book with a real life story that took place in a museum where a complex expert decision making, to figure out the authenticity of a Kouros’, fell apart. The professionals at the Getty museum meticulously investigated the Kouros for months. They used all the state-of-the-art techniques and skills they could ever use to verify its authenticity and finally declared that statue is an original work of art. However, Thomas Hoving, the former director of Metropolitan Museum of Art, New York, in his first glance, found the Kouros, suspiciously “fresh” and then proved to be unauthentic, contrary to the thorough and systematic investigation of all the experts at the Getty museum. This is what the author wants us to realize the power of intuition and snap judgment that often supersedes the experts’ long systematic analytical process.

 

The author claims that the “Thin Slicing”, the ability of our subconscious processing of the brain, called as the “Internal Computer”, to find patterns in situations based on very narrow slices of experience, is a central part of what it means to be human. The “Coup d’oeil” i.e. Power of Glance as well as snap decision make huge difference in the situations that demands to be really quick. To show the power of snap decision, the author brings the story of Vic Braden, one of the world’s top tennis coaches, who was able to call out a “double fault”, which is a rare event for a tennis player in a professional game, almost without any miss. There were no explainable clues that he could come up with on how he could make that snap decision just before the player’s going for double fault. In the book, it describes as: “Snap judgments are, first of all, enormously quickly: they rely on the thinnest slices of experience but they are unconscious. They take place behind a locked door and we are not very good at dealing with the fact of that locked door as they happen in our subconscious part of the brain processing”. Now the question may come to the mind that is there any formal way to create a process or structure that is conducive to make effective snap judgment? It turns out that, yes, there can be structures created to support spontaneity to foster snap judgment. Malcom Gladwell has shown that the spontaneity of decision can be better made through “less is more” i.e. by not overwhelming with voluminous data and processing. The proves in the book are: the mind blowing victory of Van Riper in the Millennium Challenge war game and simplifying the triage process of heart attack patients in Cook county hospital. Though spontaneity may sound like a random and chaotic structure but it’s far from that. The ability to become effectively spontaneous demands years of training in the relevant field.

 

If the Thin Slicing and Snap Judgments are all sounds like the panacea, let’s be clear that, it’s not the only color Gladwell wanted to paint the entire book with. There are dark sides that he wants the readers to be aware of. He reminds the shocking abject judgment in the American political history that he called as “The Warren Harding Error”. The dark side of Thin-slicing is that the snap judgments are made without getting below the surface of the situation that creates a condition where the “Blink” is often the root of prejudice and discrimination. The author has explained the famous Implicit Association Test (IAT) and showed how it unearths the bitter reality of our unconscious attitudes that may be utterly incompatible with our stated conscious values.

Gladwell then told us the story of Kenna who is a victim of snap decision not because that snap judgment was wrong but because, the snap judgment was inappropriately used to assess an unconventional musician. In the same line, Pepsi Challenge through a blind test was a famous failure for Coca Cola Company that entirely missed that what people says about a snap judgment doesn’t necessary holds true when the real consumption happens. In those instances, the experts’ skills of meticulous evaluation is much more effective than that of the thin slicing of experience as they might not be able to grasp the totality of an experience. So, it’s also important to use the right way to ask people for their snap judgments.

Gladwell then explored the science behind a fascinating side of human psychology i.e. the Art of Mind reading. He explains the research of Ekman and Friesen who have identified close to three thousand facial expressions and encoded them into “action-unit (AU)” to identify the facial displays of human emotion. Though human mind can be read through those “AU” but in a fast moving and stressed situation, people sometime lose the ability to read those clues, called as “temporary autism”, the drives people to bad judgment and inaccurate decision making. The case study of Diallo of Bronx, shot and killed by police officers, was such an example. He suggested that the gaffe of mind reading caused by “temporary autism” can be prevented by creating a white space i.e. giving enough time and space between the observation and decision making.

At the end of reading the six chapters of the book, readers may feel an apparent contradiction where the author blames the blind testing in one chapter and then advocates for the same in the next. But this would be clearer by his suggestion in the conclusion where he strongly recommends utilizing the moment of blink carefully. He puts forth a guiding principle on when to blink and when to think: “On straightforward choices, deliberate analysis is best. When questions of analysis and personal choice start to get complicated - when we have to juggle many different variables - then our subconscious thought processes may be superior, …contrary to conventional wisdom”. One thing the author could do differently in the book was to keep out the too much detail from the stories to retain the brevity of the book, but nonetheless, this is a fascinating book that took us through the complex ally of unconscious brain to show us the power of spontaneity and snap judgment i.e. “blink”.

Reference:

Book Review: Gladwell, Malcolm (2007). “Blink: the power of thinking without thinking”.

ISBN: 978-0-316-01066-5.

Myths and truths of collocation in software development

The benefit of collocation is enormous. This doesn't just dramatically reduce the communication cost but also tremendously helpful to avoid confusion and improve team bonding. This is becoming a norm in the software development work where agility is new buzzword in the town. But how this collocation is implemented that actually is more important than just cramming people into a giant blob of space which can make more harm than benefit for the software development team.

Traditionally, the large non-software development enterprises are not designed for collocating software developers in a common space. So when they are asked by their Software development   department to create space, for instance, for their Agile software development team, the people from realty department free up a big enough conference room that can "just fit" the entire team. The expectation is, now the team would start producing the fruits of collocation. There are quire a few things missing in this expectation. Let's check them one by one:

1. Those (just freed up) conference rooms are built and organized for face to face communication in a meeting setting. A meeting is expected to run in terms of minutes, not hours. So there may not be enough space to place the computer on the table, chairs are not healthy enough for day long seating, not enough lights and ventilation for 10 to 15 people, and the list continues.

2. A team comprised of people and people have certain biological and psychological needs. People needs enough leg room, yawning space, etc. without bumping to each other. Except few people, most of the people needs off-face time. Otherwise the side effects of 24/7 cable news channels pops up.

3. Creativity needs both the conscious and subconscious minds to tango. With only the conscious mind in play, only so much can be achieved but to really come up with a good idea and solutionize that idea, the subconscious mind needs to be triggered which need a relatively quiet and private space. Specially, this is absolutely true for software programmer. They need the interaction with other fellow programmers to discuss and bring new ideas and then go back to relative isolation to concretely thought through that idea to make it to work. In lack of that, you will see people are putting on their earbuds to cancel the surrounding noise where no physical isolation is available.

4. Believe it or not, people can't work, specially people who are in creative business like software development, continuously for hours after hours without breaking away from that monotone. It works in a certain situation and setting but majority time it needs social break. So, if we force people to be in the crowd all the time, we may get lines of code that just do the work but not a breakthrough solution that will down the total cost of software development and maintenance.

I am not necessarily suggesting that people needs to be mile apart from each other but at the very least we need to recognize the need of balancing the collocation with personal space to get the best out of collocation.

Triangulation in software development requirements

I can bet that almost every software development project faces with a dilemma of questioning some of the business requirements that actually come from the very business that you are trying to systematize. This is arguably the most dangerous zone for any software development manager to get into as your suggestion would inevitably be considered as if you are questioning the business people's core skills. But you can't just accept it by its face value without critically analyzing the requirements. After all, in the current business environment, neither the business nor the technology independently drive the company's future. Now the sixty four thousand dollar question is, how would you do it in a methodical way? The answer lies in the word "Triangulation". This exists in other disciplines for long time and can be used in software development management.

Let's see a usage of Triangulation in other discipline. Triangulation is used by your Global Positioning System (GPS) device. There are at least three GPS Satellite that sends the positions of your device that help the GPS device to accurately pin point your location within a few feet range. 

Diagram 1: Framework for Triangulation of business requirement justification

You can use the same methodology to check if a business requirement is justified to be taken into the project. Take a typical business software project. Use the following triads to justify the business requirements: (1) Project Business Objective, (2) Minimum Viable Product (MVP) and (3) Total Cost of Ownership with perpetual cost of maintenance

(1) Project Business Objective is the first filter you have to let the business requirement go through. If the requirement doesn't align or contradict with the project's objective, the chance is very high that there are other motivation hidden behind the requirement that needs to be unearthed

(2) Minimum Viable Product is the Lean concept where it encourages the validated learning through early delivery of the viable software with a minimum set of features. Put that requirement to the test of MVP and see if it passes or fails

(3) Finally, there's no cost free feature in a software. Even for a feature that's working without adding a single line of code can have high maintenance cost. There are several types of cost, such as: now you have to make sure that in every upgrade you do or every new feature you add, the no-code-feature has to be tested to make sure that it functions as is; if you find a better technology/algorithm to implement in that software which would stop that no-code-feature to function, then you either have to stick to the inferior technology/algorithm or now add code to make that no-code-feature to work

Finally, never forget to identify the true need of the business (sometime referred as "problem statement") behind that requirement. You would often find that the true need may be actually vastly different than the very requirement is trying to solve and the requirement may be a solution that the business person came up with to solve that true need. Knowing the true business need would allow you to come up with a true solution of that problem which may be much simpler to implement. Undoubtedly, the simplest solution is the best solution and you would need a complex thought process to come up with that simplest solution.

A super brief note on Blockchain and Silver Bullet

The Blockchain is everywhere and the promise of the technology has inspired a big number of technologist and business executives to jump on this bandwagon. Yes, the promise of the the Blockchain technology is huge but it may be oversold specially around it's non-repudiation and decentralize nature of security. The defense is made that it's impossible, or not practically feasible, to break the chain due to its nature of linking the prior blocks using the hash and so forth. Similarly about the non-repudiation characteristic of the technology.  

I no way claim that I know how to break this or it can be broken today but I know for sure that there's nothing called absolute secured technology in the computing field. It's all about relative security where we call a system secured if its security can't be broken in relatively short period of time that makes it vulnerable. We should definitely invest big in block chain, not because of considering it as the panacea but because it is superior in the area of non-repudiation and digital identity, etc. features while keeping in mind that some time, some smart programmer will break this using a superior algorithm or a unexpected breakthrough in computing power.

How to reap the best out of Agile

As we have debunked the myth of the Agile methodology, let's look into how we can get the best out of an Agile team. There are volumes of books, training and certifications that are available on building and running Agile teams but here is a shortlist of  best practices (or you may call them as principles, if you will) to build a successful Agile team that produces the most value to its cost.

[A quick disclaimer: for simplicity, this article is focused primarily on SCRUM Agile development methodology as the term "Agile" is more familiar than "SCRUM". So, one can use the word "SCRUM" and "Agile" interchangeably while reading this article]

1. Hire Team members who believe in Agile

Eric Schmidt, the former CEO of Google, has said in his book, "How Google Works", that Google doesn't transform a person into problem-solving ninja but the company attracts those ninjas and they build the great products at Google. So, if you want to get your Agile  team to excel, hire the "smart creative" people who believe in Agile. Agile is a cultural mind set, not just a mere methodology that can be memorized. You can't turn or train people to become Agile unless they have that "Agile" gene deep into them. But you can tune them in Agile, if that is what they believe in. 

Now, let's talk in a practical term. Neither all companies are Google nor you would have freedom to pick your team at all the time. So, at the bare minimum, when you don't have that liberty to build from the scratch, before building a team for Agile, train them in Agile. Really and seriously! Train them in Agile!

2. Get business buy-in...this is the master key of Agile success

In my "Myth and Truth about Agile..." post, I have explained the importance of having the business people fully engaged in the Agile process. Without the full participation of business, there's almost no chance of running the show in Agile. There are occasions where the Software Development team claims that they are following Agile without the business participating in it which is kind of oxymoron and hard to believe.

So if you can not secure the buy-in of business into your Agile process, it's better to use other non-Agile methodologies, such as Rational Unified Process (RUP) or any other iterative method, except Waterfall. Believe it or not, those non-Agile methods work just fine when practiced in a consistent manner.

3. Avoid the quick sand of Architectural and Design soundness

Let's not use the Agile as an excuse for a poor Architecture or a closed design. Yes, the Agile doesn't let the team to spend a lot of time to focus on a robust architecture and comprehensive software design rather emphasize on working software. It asks for minimal upfront design and to continue to design over the period of product development life cycle. The crux of the game is "refactoring". The refactoring has to be done both in code and design level. This means continuous rework and occasional throw away code. If you are not ready to accept that notion of refactoring, may be Agile isn't fit for your project. 

There's a catch in that minimal upfront design through, which we forget often. The notion of “minimal upfront design” in Agile requires that the developers are skilled to design, and architect and the Designer or Architect codes. Your organization needs to be ready to invest on people who poses that level of skill set. 

4. Still you do Release Planning

Even though Project Management principles are in direct conflict with Agile principles but there's fair amount of planning involved in Agile projects. What you have in traditional project management as Work Breakdown Structure, in Agile you would have Product Backlog with User Stories. Through Affinity Sizing the product backlog would give a sense of project duration which would be broken down in to Releases based on business priority, value and dependencies to get Agile Project Scheduling.

Some Agile practitioners feel very uncomfortable with this planning aspect of Agile that's based on affinity sizing. Affinity sizing is done at high level of understanding with minimal detail. This lack of comfort is not the fault of those people but the fault lies in for not utilizing the affinity sizing with right intention. The affinity sizing can no way be used as the basis for commitment but the user stories have to go through multiple passes of refinement and re-sizing over the period of time to come to the level of confidence to make project commitments. The Cone of uncertainty in software estimation has to be kept in consideration when making commitments because "you can't beat the Cone of uncertainty but you just can be more lucky"

5. Do the right Sprint planning

Task, Task and Task....this is crucial that team members create the tasks for each user stories!

One of the objectives of the Sprint planning is to create tasks for the user stories. The purpose of keeping the Sprint shorter is to ensure that the developers can plan in detail and the manifestation of a successful planning is "lot of tasks". Human brains are not good at keeping every detail for next few months and that's why the Sprints are recommended to keep between two to four weeks. This is a very crucial tool for a successful Sprint.

Yes, the first objective is to create the Sprint Backlog (scope) for the upcoming iteration but the more important objective is to go into the next level to each of the stories and break them into tasks (tasks should be no more than that a day, and smaller tasks than a day is preferable). Effectively 80% to 90% tasks should be created in this planning meeting and put on the story board for anyone to pick up. There will be some unidentified tasks for sure, but that volume shouldn’t be more than 10% to 20%, and those would be identified during the sprint period.

At the end of the Sprint Planning, the story board should be filled with newly created tasks with the status of “Not Started”. Otherwise, it’s an indication that the stories are not groomed enough or the team doesn’t know what to do with that story. Either of them are not a good sign for a productive Agile team.

6. Delivery early and quickly

Agile is all about working software. So it is pretty obvious to have continuous delivery of working software. I would like to emphasize on it in light of Architectural components that usually takes more time to build at the beginning of a project. But it's better to keep that working software concepts even for the architectural components. 

For example, if there's framework to be built for workflow, messaging, enterprise integration, etc., it's better to plan to deliver the smallest piece of that working architectural components with minimal functional user stories (much like a Prototype but a working piece of software that would be the basis of further development). This is also aligned to the Lean concept of "fail fast and early" so that the recovery cost is low and manageable. Apart from early proof of concept, this task orientation also helps to rally the team around the delivery and gets the team in performing stage quickly. 

7. Track and remove impediments effectively

The Sprints are short. So it's the responsibility of the organization to put a framework to track impediments and remove them as quickly as possible. The agility can't be achieved if the team is playing rat and mouse game with the organizational process instead of developing working software. It sounds easy and every organization will pledge to this principle immediately as soon as you ask for it but it's actually much bigger than just giving a pledge. The entire organization needs to be oriented to Agile culture to server an Agile team effectively.

8. Use visuals everywhere and anywhere

Human brains are not naturally built for reading lots of texts and interpreting the progress status of the team from text is not intuitive. Also, as Agile is like a fast moving car running with pedal to the metal, the status of the team and product should be readable with a glimpse of an eye. So, keeping the status of Release and Sprint, in the form of burn down graph, in-front of eyes (or accessible with one click) is key. Every member of the team should be on the same page on where the team stands right at this moment. 

Kanban Dashboard is another nice tool to visually track the progress of an Agile team. Just by a glimpse at the dashboard, one can paint the right picture of the team's current state.

9. Realize the self organized team

It's easier said than done. Some time we don't even understand the breadth of the self organization aspect of team management. Managing a self organized team needs a whole new set of skill set and attitude that aren't easy for a traditionally trained manager. It would take another article to explain the managing of self organized team but in a quick nutshell: it's like you tell the team to get things done by not just doing that the "telling" thing.

Moreover, it's not that having self organized team is good to have but it's actually absolutely necessary to have the team self-organized due to the fact of business involvement and transparency in Agile. It's like you break the wall of the kitchen in your restaurant and allow the diners to come and talk to the chef and ordering their food while standing next to the cook in the kitchen. There's no place of hide and seek in an Agile team.

Myth and Truth about Agile software development methodology

The term "Agile" has now become close to a fad than true appreciation. You will see people brag about how "agile" they are in their software development organization. Not only that, I have also seen people use the term "Agile" to activity such as, project planning where no software development is involved at all. But who would counter them? You would be lectured by seasoned consultants by deconstructing word by word and how their method of creating a plan is a true agile process. On the other hand, people who claims to love Agile and live on Agile, will tell you how Agile development is much more superior than "waterfall method". By the way, who will tell them that between the stone age and industrial age, there was iron age and don't compare the industrial age with stone age by skipping the iron age civilization.

With all those, I felt obligated to explain the "Truth" of Agile software development methodology. Let's begin with a simple fact

Agile is for Software Development

Believe it or not, Agile is, and came for the software development. People may tell you otherwise by showing the linguistic deconstruction of noun vs. adjective and prove you that Agile can be used for your apple picking job as well. For the record: I had once done a "Agile snow cleaning" in a winter - just to have fun as I was tired of snow cleaning due to repeated blizzard that threw couple of feet of snow. It was just for a pure fun, no academic purpose was involved in that. But if anyone claims that Agile can be implemented anywhere, then I would argue that you can develop software using "Taekwondo" principles as well. Spinning can be endless.

Agile is not the successor of Waterfall methodology

As I had touched upon at the beginning, prior to Agile, Iterative methods were very successful and effective. Actually, if you ask me what method I would prefer: Agile Scrum or Iterative Rational Unified Process? I would tell you, pick any but be consistent

Agile is not a silver bullet

Business sometimes think that Agile is a silver bullet and solve every problem the business is facing today. Agile is rather a software development methodology that comes into existence to reduce the risk of software development and maximize the business value of software but it is not a panacea. Agile is not free of cost though. The cost includes: risk of loosing big picture of the solution, quicksand for architectural soundness, overemphasizing on working software over long term stability, fading documentation, etc.. Also, Agile demands certain creed without that it is hard, if not impossible, to be successful. Though not all the risks are due to the Agile practice but sometime due to the fact that people use Agile as escape goat. 

Agile needs business people to be involved...fully involved

When business wants the IT to be Agile, they often don't realize that the Agile is not methodology that IT will practice on their own and business will continue their existing operational model. Agile needs business to change or adapt to the Agile development model. Business flocks into Agile to get the fruit of transparency on how it's being cooked in the kitchen but to reap that fruit of Agile, business needs to play their part. It takes two to tango!

SCRUM is not synonymous to Agile

SCRUM is widely popular in the industry over other Agile implementation such as Test Driven Development (TDD), Xtreme Programming (XP), etc. So people who jumped late on the Agile bandwagon sometime confused SCRUM as the Agile process. SCRUM is just one implementation of Agile that has given a framework to manage Agile software development project very effectively. As SCRUM has taken Agile close to the business people (and also other Agile implementation are too technical for non-technical business people to grasp), they have taken the familiarization with Agile through that framework. 

Agile is a philosophy and demands cultural change

The Agile Manifesto talks about the philosophy behind the Agile Software Development methodology. As this doesn't dictates one on how to achieve those principles but this can be with a level of certainty that without changing the culture in the software development, it is impossible to achieve the principles. I have seen instances where business sprinkles the principles of Agile software development on engineering team without emphasizing the underlying cultural change and blames Agile for their failure.

Digital Signature and how it is used on the Internet

Digital Signature is, as it implies, the digital equivalent of handwritten signature that carries a distinctive digital pattern to identify a person or system. The public-key cryptography is used to sign a document that would provide a non repudiation way to prove that a document is sent by the owner of the document and not tempered with in the middle. Let's see this in action -

• The process starts by generating a key-pair: public key and private key. The public key is distributed publicly to the world and private key is kept secret to the owner of the key pair. The distribution of public key is done in various way, such as: key signing party, publishing on a well known website etc.

• Now, using an one-way cryptographic hash function (e.g. MD5, SHA-1, SHA-2, etc.), the message digest or hash code is generated from the original document. Hash functions take an arbitrarily long piece of plaintext and compute from it a fixed length string

• The message digest is then encrypted using the private key and the encrypted hash code is appended to the document. This is the digital signature

• The receiver would first generate the hash code of the document using the same hash function of the sender

• The receiver decrypts the added hash code using the sender’s public key

• The receiver compares these two hash codes and if they matches, then it’s proved without doubt that the document is sent by the owner and moreover, the document wasn’t tempered in the delivery. If the two hash codes do not match, it’s an indication that either the document is not sent by the owner or it’s tempered in the communication

How your web browser gets the IP address of a website?

Domain Name System (DNS) in the Internet that works tirelessly to give back the answer to your browser when it needs an IP address of a website. DNS is a hierarchical architecture that allows the resolution of the human readable names of a computer in the internet into a machine usable IP address. 

The process of resolving the website address, for instance, Youtube server, typed on the browser is as follows:

• As this is the first time accessing the www.youtube.com, the browser creates a DNS query to the local DNS server that’s configured in the operating system. This would be the DNS server in your ISP network (e.g. Comcast) which is called as caching DNS

• If the Comcast DNS never communicated with the Youtube server (which would be very unlikely), this wouldn’t find any entry in its cache. This DNS server makes a request to the root name server. The root name server is configured manually in the Comcast DNS server

• The root name server is the authoritative server that looks up the right most portion of the domain name and returns the name server(s) for the Top Level Domain (TLD) of the dot com (".com") domain

• As the Comcast DNS gets the "com" TLD name server, it makes another request to the "com" TLD DNS server which in turn returns the name of the Youtube's ISP (I presume it's Google) DNS server’s IP address.

• Now the Comcast's DNS server makes the last request to the Google's DNS server to resolve the IP address for the domain www.youtube.com. This ends the recursive calls made by this caching server and creates a cached entry of this resolved IP address with a Time To Live (TTL) value. The TTL tells when this cache entry would expire. It respond back to your computer (e.g. laptop) which made the initial request 

• Your laptop now knows the IP address to make the request to the www.youtube.com server to retrieve the web page on the browser. It would also cache the resolved entry in the Operating System level to make the subsequent request to the same server much faster by avoiding all the above calls until the cache entry expires

It's good to know the concept of DNS caching in a little detail.  DNS caching is the process through which the local DNS server (known as caching DNS) stores the already resolved IP address for a certain period of time. As mentioned in the above answer (5a), the computer operating system and local DNS servers first look into the stored cache for the IP address of a domain name. If the cache doesn’t have the record available, it reaches out to the authoritative DNS servers (root DNS, TLD DNS, destination  network’s DNS, etc.) to resolve for the IP and then stores that into the the caching DNS server as well as in the local computer’s OS. 

The caching of each record has an expiration time tagged along with that. This is called Time-To-Live (TTL) which is set (in terms of seconds) by the authoritative DNS. The cache is deleted when the TTL time is elapsed. At that time, if that domain name is requested for the IP address, it has to go through the same DNS resolution process. The DNS caching is used to improve the performance of the DNS resolution. This not only helps improving the performance of the network usage but also keeps the Internet free of DNS query traffics   

Proxy server and its benefit in a computer network

Proxy server is a middle-ware box that sits between the requesting user’s machine and the destination server to provide control to the access. Here’s how a proxy server technically operates:

• The Proxy server is configured in the network through which all the traffic would pass through. This is not a transparent middleware box like NAT box, so the client (web browser) accessing the webserver has to configure the proxy server address

• When a request comes from the web browser to the proxy server, the Proxy server first checks its cache for that requested information. If the information is available at the cache i.e. requested by some other users earlier, it returns the information back to the web browser without going to the web server. This saves a valuable network bandwidth and time

• If the requested information isn’t available in the cache, it opens up a new request to the destination web server (acting like NAT) and caches the responded information into its cache

• Then it returns the information back to the original requester

You can implement a proxy server to reap the below benefits:

• Proxy server improves the performance of the network as the cached information are reused to server multiple users. Thus saving network bandwidth and time

• Proxy server increases the security in the network as you would have more control on what information it would be allowed for users to access by restricting malicious sites. It also keeps log of all the traffic so that can be used to investigate security issues and vulnerabilities as well

• You can set your own policy in a convenient manner through a proxy server, like restricting access to illegal sites, unproductive sites etc

NFS vs FTP in a nutshell: The Networking perspective

File Transfer Protocol (FTP) is a client server model protocol to transfer files in a network, implemented on top of TCP. FTP works much like HTTP where FTP client makes specific requests to the FTP server thus not transparent to the client. Though the implementation of FTP is simpler than other protocol (like NFS), but it works a bit different way. The FTP client and Server treats control and data differently. At the connection, the FTP client connects to the standard port 21 and then agree upon to a different port to connect for the data to be transferred. The data connection can be initiated from either side depending on the implementation and network configuration. The primary use of FTP is for repository of the shared files to centrally manage them. Because of the simplicity, it allows easy access to the files across the network. A side note: DO NOT use FTP, use SFTP. FTP sends your password in plain text.

Network File System (NFS) is also a file sharing protocol which is implemented on top of RPC which typically runs on UDP. NFS is different from FTP in a sense that it does provide transparent access to the remote files resides in NFS server i.e. the NFS client access files (local and remote) in the same manner from the end user perspective. NFS server is a broadcast server that publishes itself within the network so any client connected to that network gets aware of it without explicitly looking for it. Another important distinction is that NFS doesn’t maintain the state of the connection so it is much more resilient to the network connection instability. NFS is used to centrally managed shared files (similar to FTP) without being duplicated across the network. As the NFS is transparent access, it is used to create a home directory for users in the network where that directory can be mounted to any machine, which would provide user the feeling that the home directory is connected to the local machine.

What makes a network IPv6 capable

To declare a network is capable of IPv6 connectivity, there are certain infrastructural components that have to be in place before hand.

Operating System: The operating system (or the client) has to be IPv6 enabled client. This could be in the form of dual stack client or IPv6 only client so that it would have IPv6 address assigned to it.

Dynamic Host Configuration Protocol (DHCP6) server: Now a days, almost every network is managed by DHCP to assign IP address, though this is not a mandatory device. But to declare that the network is IPv6 enabled, there has to be a DHCP6 server if the network needs to be managed in a stateful manner to autoconfigure the network IP addresses. The alternate is stateless autoconfiguration that doesn’t need DHCP6 server (which comes with some security risks). In that case no DHCP6 server is required.

Router: the router has to be able to recognize and process IPv6 packets. Other option is tunneling which doesn’t say that the network is IPv6 enable but just a work around

Domain Name System: the DNS has to be capable to resolve IPv6 addresses so that the source host can reach out the destination IPv6 hosts

There are few other types of devices that are sometime placed in the network like NAT, Proxy server, Firewall. If these devices are present in the network (and probably are), then all these devices should also be IPv6 aware so that the end to end connectivity can take place. 

There’s a hard way to make the IPv4 network works for IPv6 communication, which is through tunneling. In that way the IPv6 packets are transmitted by encapsulating into IPv4 packets. But this is a complicated way to achieve IPv6 connectivity with high cost of configuration and performance

Flow control & Congestion control: The two most important features of TCP that keeps the Internet alive

Flow control is the mechanism where the sender and receiver sync up the data rate between them to not to overwhelm the receiver, in the case where the receiver has less capacity than the sender.

Congestion control, on the other hand, is the sender trying to figure out what the network is able to handle. This the mechanism at the sender’s end to determine through the data loss on the transmission link and adjust the throttle accordingly to be most efficient.

Both flow and congestion controls are necessary to effectively transmit data from sender to receiver. Without flow control, the sender would overwhelm the receiver’s buffer and the data would be discarded and also sender would be forced to continuously re-transmit the unacknowledged data. This would tremendously impact the performance of the TCP protocol, throughput and the performance of the network link. Similarly congestion control helps the sender to determine if the data being sent over the network are capable of delivery to the receiver or not. There could be situation where both the sender and receiver are perfectly fine to accept a higher data rate but if the link in between isn’t capable enough, then a lot of bandwidth would be wasted just to re-transmit the lost data on the link. This would effectively make the data transmission slower than the true capacity of the link. Though both are necessary to achieve the optimal performance but they are essentially two different things:

• Flow control is between sender and receiver, whereas congestion control is between the sender and the network
• Flow control is dictated mostly by the receiver through negotiation, whereas the sender dictates the congestion control
• Flow control is to sync up the data transmission between sender and receiver, whereas congestion control is to sync up the data transmission between the sender and the network link
• Flow control is end to end but congestion control is not end to end but resides at the sender’s end alone

Implementation of flow control: TCP uses sliding window model to implement flow control. This is achieved through the use of advertised window size from the receiver. The receiver communicates the buffer size during the connection establishment and can change it anytime during the life cycle of the connection. The receiver and sender negotiates the buffer size where the SWS (Sender Window Size) is set to RWS (Receiver Window Size) that ensures that the Sender isn't sending more data than the Receiver can receive before acknowledging them.

Implementation of congestion control: TCP probes the network by starting with small amount of data to come up with optimal transmission rate in the sliding window model. TCP uses a new variable in the sliding window called congestion window, to control the streaming rate of bytes. In conjunction with sliding window’s advertised window size, this congestion window helps to determine the maximum size of the allowed window, which is the minimum of those two windows. Unlike advertised window, the congestion window is determined by the sender, determined by the network link’s ability on the data transmission. The loss of data is used as indication of congestion on the link and set the congestion window accordingly. TCP considers that the network is otherwise reliable (wireless is handled differently through).

There are various techniques that are used to implement congestion control: Slow start, Additive Increase/Multiplicative Decrease (AIMD), Fast re-transmit, Fast recovery etc. In AIMD, TCP starts streaming bytes at a minimum rate and increase the rate in an additive fashion. Another implementation is to use slow start with a small amount and then increase the rate exponentially up to the congestion threshold level. After that it goes back to additive increase until congestion is sensed, which triggers the TCP to sharply decrease the rate and also reset the congestion threshold to a lower number (depending on the implementation). This continues throughout the life cycle of the connection and sync up with the network link’s ability to handle the transmission