Tag: software development

The Importance of Communication in Software Development Teams

Posted on by eugene evdokimov

Programming is often thought of purely as a problem-solving activity. This may be true for the lone coder on their homegrown software in their garage, but in the multi-person environment of an Agile team, such problem solving must be collaborative. In this article, we’ll look at the role of communication in software development, particularly in an Agile framework.

Covid-19 has forced an unprecedented shift to remote working so we’ll finish up with a discussion of how Agile can be implemented in a remote setting.

Communication and Agile Development

Agile is increasingly becoming the gold standard for software development teams. With its emphasis on close-knit, self-organizing teams that work informally and respond quickly to change, Agile reserves a critical role for communication in software development.

This is because, as stated in the Agile Manifesto, people and their interactions are the driving force behind successful software development. Teams are self-organizing and need to quickly respond to change. This means that good communication is a cornerstone of successful software development. 

Daily Standups and Scrum Meetings

According to Principles Behind the Agile Manifesto, face-to-face interaction is the most effective form of communication in software development. The main way of facilitating this is via daily standups or Scrum meetings. These involve the entire team meeting at the beginning of each day in a sprint. 

Engineers take turns to explain what they have been doing and any problems they have encountered.

Whiteboarding

Whiteboards play a critical role in facilitating collaborative communication in software development. That’s because whiteboards are a low-tech, flexible, and intuitive way to facilitate communication in software development teams.

They allow developers to write down ideas and sketch diagrams as soon as inspiration strikes. The whiteboard’s physical prominence makes it a public platform for team communication; anything written on there can be seen by every team member and inspected at leisure.

Since Covid-19, many teams have switched to remote working. Software such as AWW board and Stormboard provide virtual whiteboarding solutions that allow the benefits of whiteboarding to be used anywhere in the world.

Pair Programming

Pair programming is the ultimate form of teamwork and communication in software development. It typically involves two programmers sitting side by side at one computer. One programmer, called the driver, writes the code. The other programmer, the navigator, watches what the driver is doing, spotting errors and suggesting improvements.

At first blush, pair programming seems uneconomical. Why would you get two (expensive) programmers to do the work of one? While this is the knee-jerk reaction of many managers, Alistair Cockburn and Laurie Williams tell a different story in their research, “The Costs and Benefits of Pair Programming”.  They cite several examples, including a study on a class of software engineering students to show that pair programming is economical, results in fewer coding errors, and higher team satisfaction.

Pair Programming Remotely

Surprisingly, pair programming can be done remotely.  XPairtise is an Eclipse plugin created by two computer scientists specifically for implementing pair programming remotely.  This tool allows distributed programmers to initiate pair programming sessions.

Its shared editor allows the navigator to see what the driver is typing. The navigator can’t edit the code, but they can assist the driver by highlighting fragments with the remote cursor and making suggestions using the embedded chat feature.

XPairtise is also integrated with Skype, enabling “over the shoulder” interaction by voice. Moreover, even when a programmer uses XPairtise in single-user mode, their session is available for their team to comment on, meaning the experience of each engineer always benefits the group.

Observability

Communication in software development isn’t just between human beings. It’s vitally important for engineers to know what’s going on inside the systems and platforms they use. Observability is what turns a system from a black box into a problem-solving tool.

The three pillars of observability are logging, tracing, and metrics. If you want to learn more about the role tracing plays, we’ve covered it in a previous post. We’ve also written about converting metrics into insights. Coralogix enhances traditional log and metric management with the power of machine learning.

Moving Out of the Basement

One of the principles of the Agile manifesto is “business people and developers must work together daily throughout the project.”

The importance of team communication in software development goes beyond the team itself. It extends to external teams and clients as well. For this to work, the development team needs to make sure their goals are aligned with the end-user by being in constant contact with interested stakeholders.

A Tale of Two Companies

Consider Oath, a global software company with teams in many different countries, including the US.  Their software projects must have good communication baked in from the start. As software project manager Cristina Otero comments, “it’s really important to communicate goals and expectations to everyone to make sure every piece of the machine is in sync.”

Google is another company that has tackled the problem of inter-stakeholder communication in software development head-on.  They were faced with the problem of making sure their site reliability engineering teams collaborated successfully with product development and service and infrastructure teams.

In their essay on the topic, Google compares their SRE team to a human API. Just as an API mediates between different components of an application, the SRE team needs mediation to facilitate information transfer between different project teams.

Google’s main method of doing this is via production meetings. These are weekly meetings where the SRE team “articulates to itself—and to its invitees—the state of the service(s) in their charge, so as to increase general awareness among everyone who cares, and to improve the operation of the service(s).”

Google’s production meetings are never longer than an hour, enough time to get ideas across without impairing engineer productivity with pointless meetings. While normally face-to-face, two SRE teams can meet by video, something particularly relevant during the Covid crisis.

Doing Agile Remotely

When Covid-19 arrived on the scene last year, companies were forced to shift to remote working almost overnight. Agile software teams were no exception. They’ve been faced with the challenge of taking the organic framework of Agile with its emphasis on problem solving through close-knit interpersonal interaction and making it work in a remote setting. 

In this section, we’ll look at remote communication strategies that facilitate the frictionless communication that software development teams rely on.

Communication through public channels

Agile software development depends on public communication channels to facilitate the diffusion of ideas throughout the entire team. 

In the office, everyone is aware of spoken conversations, as well as being free to examine the whiteboard. At home, everyone is at their own computers in their own rooms. Outside of using a tool such as Slack or Zoom, developers are not automatically aware of what the rest of their team is doing.

Text or Video?

In this situation, Agile’s classic emphasis on face-to-face communication has to be revised.  Kamil Lenonek argues against using video calling in favor of publicly accessible chat forums like Slack.

While video calling may seem a natural substitute for face-to-face interaction, it suffers from being private. Because everyone’s in their own house, there is no chance for the team to overhear conversations and any insights generated can’t diffuse to the rest of the team.

Instead, team members should use communication spaces where the whole team is present, such as Slack channels, even for things that they might normally do on a one-to-one basis. 

Even things that seem routine, such as setting up a piece of software, can be helpful to the entire team. Leonik gives an example of a Facebook message where somebody asked for help setting up Ghost. Leonink himself was looking for a ghost tutorial and this message seemed the perfect cue for one. But the messenger asked for help in private.

The downside of private communication is that any knowledge uncovered can’t be shared.  While many developers are scared to make all their communications public, perhaps because they don’t want to look stupid in front of their colleagues, Leonik urges them to do so.

By making insights publicly accessible, the entire team can benefit. Agile’s frictionless communication, which came perilously close to being snuffed out, can once again resume. 

Keeping it Together Remotely

The keystone of Agile is the ability for groups to solve problems that no one could tackle on their own. Any remote implementation of Agile must recognize that empathy and group morale are just as important as computer skills and Wi-Fi connections.

It’s no easy task.  According to a McKinsey report, remote work results in reduced team cohesion. Their survey found that 80% had impaired work relationships due to reduced communication and 84% reported workplace challenges dragging on for days or more.

There are lots of ways these issues can be addressed. For example, a US bank implemented virtual happy hours where team members can have a drink over Zoom and talk about whatever’s on their minds.

With hard work and dedication, a fully remote software team can communicate effectively. After all, just because a team is distributed, doesn’t mean it’s not close-knit.

Wrapping Up

With its emphasis on face-to-face interaction and collaborative problem solving, communication is fundamental to Agile software development.  We’ve seen how communication is facilitated through Scrum meetings, pair programming, and whiteboarding.

Covid-19 has put remote working in the spotlight. We’ve seen how important public communication is in this context. We’ve also seen the critical need for empathy which remote working has made more important than ever. By embracing the communication principles suggested by Agile, you and your team will continue to excel. 

What is GitOps, Where Did It Come From, and Why Should You Care?

Posted on by eugene evdokimov

“What is GitOps?” – a question which has seen increasing popularity on Google searches and blog posts in the last three years. If you want to know why then read on.

Quite simply, the coining of GitOps is credited to one individual, and pretty smart guy, Alexis Richardson. He’s so smart that he’s built a multi-award-winning consultancy, Weaveworks, and a bespoke product, Flux, around the GitOps concept.

Through the course of this post, we’re going to explore what GitOps is, its relevance today, why you should be aware of it (and maybe even implement it), and what GitOps and Observability mean together. 

What is GitOps?

The concept is quite simple. A source control repository holds everything required for a deployment including code, descriptions, instructions, and more. Every time you change what’s in the repository, an engineer can pull the change to alter or update the deployment. This principle can also be applied for new deployments. 

Of course, it’s a little more complex than that. GitOps relies on a single source of truth, automation, and software agents, to identify deltas between what is deployed and what is in source control. Then, through more automation, reconcilers, and a tool like Helm, clusters are updated or changes are rolled back, depending on a host of factors.

What’s crucial to understand is that GitOps and containerization, specifically with Kubernetes for orchestration and management, are symbiotic. GitOps has huge relevance for other cloud-native technologies but is revolutionary for Kubernetes.

The popularity of GitOps is closely linked to some of the key emerging trends in technology today: rapid deployment, containerization with Kubernetes, and DevOps.

Is GitOps Better Than DevOps?

GitOps and DevOps aren’t necessarily mutually exclusive. GitOps is a mechanism for developers to be far more immersed in the operations workflow, therefore making DevOps work more effectively. 

On top of that, because it relies on a central repository from which everything can be monitored and logged, it brings security and compliance to the heart of development and operations. Therefore, GitOps is also an enabler of good DevSecOps practices. 

The Four Principles of GitOps

Like any methodology, GitOps has some basic principles which define best practice. These four principles, defined by Alexis Richardson, capture the essence of ‘GitOps best practices’:

1. The Entire System Can Be Described Declaratively 

This is a simple principle. It means that your whole system can be described or treated through declarative commands. Therefore, applications, infrastructure, and containers are not only defined in code, but are also declared with code as well. All of this is version controlled within your central repository.

2. The Canonical Desired System State Versioned in Git

Building on the first principle, as your system is wholly defined within a single source of truth, like Git, you have one place where all changes and declarations are stored. ‘Git Revert’ makes rollbacks, upgrades, and new deployments seamless. You can make your entire workflow more secure by using an SSH key to certify changes that enforce your organization’s security requirements. 

3. Approved Changes Can Be Automatically Applied

CI/CD with Kubernetes can be difficult. This is largely attributed to the complexity of Kubectl, and the need to have your cluster credentials allocated to each system alteration. With the principles of GitOps above, the definition of your system is kept in a closed environment. That closed environment can be permissioned so pulled changes are automatically applied to the system. 

4. Software Agents for Correctness and Convergence 

Once the three above principles are applied, this final principle ensures that you’re aware of the delta (or divergence) between what is in your source control repository and what is deployed. When a change is declared, as described in the first principle, these software agents will automatically pick up on any changes in your repository and reconcile your cluster to match. 

When used with your repository, software agents can perform a vast array of functions. They ensure that there are automated fixes in place for outages, act as a QA process for your operations workflow, protect against human error or manual intervention, and can even take self-healing to another level.

GitOps and Scalability

So far, we have examined the foundational aspects of GitOps. These in themselves go some of the way in showing the inherent benefits GitOps has to offer. It brings additional advantages to organizations seeking simple scalability, in more than one way.

Traditionally, monolithic applications with multiple staging environments can take a while to spin up. If this causes delays, scaling horizontally and providing elasticity for critical services becomes very difficult. Deployment pipelines can also be single-use, wasting DevOps engineers’ time. 

Migrating to a Kubernetes-based architecture is a great step in the right direction, but this poses two problems. First, you have to build all-new deployment pipelines, which is time consuming. Second, you have to get the bulk of your DevOps engineers up to speed on Kubernetes administration. 

Scaling Deployment 

In a traditional environment, adding a new application means a new deployment pipeline, creating new repositories, and being responsible for brand new workflows. 

What GitOps brings to the table is simplicity in scalability. All you need to do is write some declarative code in your central repository, and let your software agents take care of the rest.

GitOps gives engineers, and developers in particular, the ability to self-serve when it comes to CI/CD. This means that engineers can employ both automation and scalability in their continuous delivery in much less time and without waiting for operational resources (human or technical).

Scaling Your Team

Because the principles of GitOps rely on a central repository and a pull-to-deploy mentality, it empowers your developers to act as DevOps engineers. 

Whereas you may have previously needed a whole host of Kubernetes administrators, that isn’t the case with GitOps. Version controlled configs for Kubernetes mean it’s possible for a skilled developer to roll out changes instantaneously. 

They can even roll them back again with a simple Git command. Thus, GitOps helps to create true “t-shaped” engineers, which is particularly valuable as your team or organization grows.

GitOps and Observability 

If you’ve read and understood the principles of GitOps above, then it should come as no surprise that observability is a huge consideration when it comes to adopting GitOps. But how does having a truly observable system relate to your GitOps adoption?

GitOps requires you to detail the state of the system that you want, and therefore your system has to be designed in a way that allows it to be truly understood. For example, when changes to a Kubernetes cluster are committed to your central repository, it’s imperative that that cluster remains understood at all times. This ensures that the divergence between the observed and desired cluster state can be measured and acted upon.

This really requires a purpose-built, cloud-native, and fully-wrapped SaaS observability platform. Coralogix’s Kubernetes Operator provides a simple interface between the end-user and a cluster, automatically acknowledging and monitoring the creation and cessation of resources via your code repository. With its ability to map, manage, and monitor a wide variety of individual services, Coralogix is the natural observability partner for an organization anywhere on its GitOps journey.

Summary

Hopefully, you can now answer the question posited at the start of this article. GitOps requires your system to be code-defined, stored in a central repository, and to be cloud-native to allow its pull-to-deploy functionality.

In return, you get a highly scalable, highly elastic, system that empowers your engineers to spend more time developing releases and less time deploying them. With built-in considerations for security and compliance, there’s no doubt that it’s worth adopting. 

However, be mindful that with such free-flowing spinning up and down of resources, an observability tool like Coralogix is a must-have companion to your GitOps endeavor.

Basic Principles of Kanban Software Development

Posted on by eugene evdokimov

Kanban software development is a popular, highly visual framework that falls under the Agile methodology. It requires real-time communication of availability and capacity, allowing full transparency of all work in a team. Tasks are represented visually on a Kanban board by cards. This allows all team members to see the status of every project at any time, including not started, blocked and completed tasks.  

The Kanban methodology helps development teams improve workflow in real-time and complete more work. 

This article will explore the principles of Kanban software development.

What is Kanban?

The name Kanban comes from two Japanese words, ‘Kan’ meaning sign and ‘Ban’ meaning board. It was developed in the late 1940s by Taiichi Ohno, a Japanese engineer, on the shop floor of Toyota. In order to reduce operating costs, they began experimenting with a system that processed small amounts of raw inventory quickly. By matching inventory with demand, the Kanban system helped Toyota achieve higher quality and throughput. 

Today, Kanban software development is used by teams all over the world to complete more work in less time, with a focus on customer value.

Principles of Kanban Software Development 

The methodology is guided by four key principles.

1. Visualize Workflow

Visualizing workflow in Kanban means not only visualizing the process, but visualizing each piece of work (represented by a Kanban card) as it moves through that process. Development Teams find that making the work visible, along with blockers, bottlenecks and queues leads to increased communication, collaboration and problem solving.

Visualizing all work also helps establish accountability and transparency across the team. A blocked work item can be identified with the person or team allocated to fix it. This is invaluable, especially for development teams, which require coordinated efforts to complete their work efficiently.

To begin improving workflows, it is important to visually map the current process. Only then will opportunities for improvement become obvious. For example, the delays of a development team receiving business approval for technical designs are easier to identify and alleviate. Visualization continues once Kanban is implemented and serves as a way to communicate the state of projects and work items.

2. Limit Work in Progress

kanban work in progress

Kanban software development teams can easily get overwhelmed with their long list of work items; such as coding project work, fixing production issues, reducing code debt and maintenance work. So teams often struggle to prioritize work in an organized way.

Therefore, an important concept in the Kanban process is limiting work in process by setting a WIP Limit. By limiting how much unfinished work is in progress, can reduce the time it takes an item to travel through the system. This also avoids problems caused by task switching and reduces the need to constantly re-prioritize items.

Setting a WIP Limit for both individuals and teams can help development teams move faster, reduce errors and collaborate more effectively.

3. Focus on Flow

When the first two principles of Kanban software development are in place, work flows freely. However, you need to focus your attention on interruptions in the flow.

These represent opportunities for additional visualization and process improvements. To assist flow, pull work through the pipeline. Don’t start any new work until a work item is completed. 

Blockers

Sometimes a work item cannot continue to flow through the pipeline due to an unexpected reason, so it becomes blocked. For example, the new project code which is ready for UAT, cannot be deployed to the test environment because of a system failure. Therefore, UAT cannot start. This work item would be marked as ‘Blocked’ on the Board until the issue is resolved. The number of days blocked would be recorded for reporting later. It is a good practice for blockers to be analysed in regular refinement sessions to identify any process improvements.

4. Continuous Improvement

Kanban requires constant monitoring and analysis to look for the next best way to improve development flow and remove waste. Conditions, resources and customer demands change over time, so it is always important to continually assess flow and WIP Limits and look for blockers and bottlenecks that can be removed.

Making small and meaningful changes will help the team perform more efficiently and effectively. 

Kanban Board and Cards

Kanban software development centers on the Kanban board, either a physical one in the office located in a prominent position for the whole team to view and access or as in recent times, a virtual one using software. The main purpose of the board is to create a shared understanding of value flow. However, the board does more than visualize workflow steps, it also highlights where bottlenecks are formed in the process.

To visualize your work, identify distinct steps or stages your work goes through as it moves from ‘To Do’ to ‘Done’. These will become columns on your board. Some boards may just have ‘To Do’, ‘In Progress’ and ‘Done’ columns. For a Kanban software development team, some examples of the columns created on the board include: ‘Design’, ‘Code’, ‘Unit Test’, ‘Code Review’, ‘UAT’ & ‘Prod’.

Each project or initiative should be allocated a horizontal swimlane. 

Pro-tip: If using a physical board, it is beneficial to use a magnetic one so magnets can be used as well as post it notes, which tend to fall off after a while. 

Kanban Cards

Every work item is visualized as a separate card placed on the Kanban board. Cards are moved from left to right to show progress and to help coordinate teams performing the work. The main purpose of the card is to give team members something to track as it moves throughout the workflow. This makes invisible work visible. 

Each card can contain critical information about the work item it represents. If a physical board is being used, then there is a limit of how much information can be displayed on each card. One way around this is for a team to also use issue tracking software like Jira and to raise a digital ticket too for each work item. This method allows for the meaty details to be contained in the Jira ticket and for the bare essentials details to be displayed on the card like Jira id, work item name and an estimate of effort. The advantage of tracking software is the history of a work item is stored for future reference and all team members can access and update the tickets.  

Another advantage is it allows Agile methodology to be performed so epics can be created to group work items, for example, in Jira, an epic is created for each project. Each epic can then be assigned a swimlane on the physical board. 

Avatars and Markers

There are different flavors of Kanban Boards and Cards. One flexible way is to assign Avatars for each team member often chosen by each individual and maybe as part of a theme. The avatars can be attached to magnets and are separate from the Kanban Cards. Each member only has one avatar as they can only work on one piece of work at a time. Their avatar is positioned on top of the card they are currently working on.

Magnetic Markers can also be used to highlight work on a card since the last stand up meeting i.e. a small green marker for any work completed and red markers for any blockers. These quickly allow the presenter at the stand up meetings to identify the progress or lack of progress and any blockers of work since the last stand up meeting for discussion. Once discussed, the green markers are removed from the cards, ready for today’s work to be tracked. 

Meetings

Once a board has been created, it becomes the centerpiece for at least two regular meetings:

  • Daily stand ups: Involves all team members, to discuss progress since the last stand up. These are usually scheduled in the mornings.
  • Refinement sessions: Involves all tech leads to discuss WIP Limits, board policies, blockers, bottlenecks and ‘stale’ work that has had no recent movement. These should be weekly or bi-weekly.

These meetings aim to focus on optimizing the flow that can help development teams stay productive and efficient by moving existing cards off the board. 

Summary

Kanban software development focuses on visualizing the entire project on boards in order to increase project transparency and collaboration between team members. The aim is to control and manage the flow of work, represented by Kanban cards so that the number of work entering the process matches those being completed.

Follow the four principles of the Kanban process for your team to consistently deliver high quality software.

More Changes Mean More Challenges for Troubleshooting

Posted on by eugene evdokimov

The widespread adoption of Agile methodologies in recent years has allowed organizations to significantly increase their ability to push out more high quality software. The new fast-paced CI/CD solutions pipeline and lightweight microservices architecture enable us to introduce new updates at a rate and scale that would have seemed unimaginable just a few years ago.

Previous development practices revolved heavily around centralized applications and infrequent updates that were shipped maybe once a quarter or even once a year. 

But with the benefits of new technologies comes a new set of challenges that need to be addressed as well. 

While the tools for creating and deploying software have improved significantly, we are still in the process of establishing a more efficient approach to troubleshooting. 

This post takes a look at the tools that are currently available for helping teams keep their software functioning as intended, the challenges that they face with advances in the field, and a new way forward that will speed up the troubleshooting process for everyone involved.

Challenges Facing Troubleshooters 

On the face of it, the ability to make more software is a positive mark in the ledger. But with new abilities come new issues to solve. 

Let’s take a look at three of the most common challenges facing organizations right now.

The Move from the Monolith to Microservices 

Our first challenge is that there are now more moving parts where issues may arise that can complicate efforts to fix them quickly and efficiently. 

Since making the move to microservices and Kubernetes, we are now dealing with a much more distributed set of environments. We have replaced the “monolith” of the single core app with many smaller, more agile but dispersed apps. 

Their high level of distribution makes it more difficult to quickly track down where a change occurred and understand what’s impacting what. As the elements of our apps become more siloed, we lose overall visibility over the system structure.

Teams are Becoming More Siloed

The next challenge is that with more widely distributed teams taking part in the software development process, there are much fewer people who are significantly familiar with the environment when it comes to addressing problems impacting the products. 

Each team is proficient in their own domain and segment of the product, but are essentially siloed off from the other groups. This leads to a lack of knowledge sharing across teams that can be highly detrimental when it comes time for someone to get called in to fix an issue stemming from another team’s part of the product. 

More Changes, More Often

Then finally for our current review, is the fact that changes are happening far more often than before. 

Patches no longer have to wait until it’s Tuesday to come out. New features on the app or adjustments to infrastructure like load balancing can impact the product and require a fix. With all this going on, it is easy for others to not be aware of these changes.

The lack of communication caused by siloes combined with the increase in the number of changes can create confusion when issues arise. Sort of like finding a specific needle in a pile of needles. It’s enough to make you miss the haystack metaphor. 

In Search of Context

In examining these challenges, we can quickly understand that they all center around the reality that those tasked with troubleshooting issues lack the needed context for fixing them. 

Developers, who are increasingly being called upon to troubleshoot, lack the knowledge of what has changed, by who, what it impacts, or even where to start looking in a potentially unfamiliar environment.

Having made the move to the cloud, developers have at their disposal a wide range of monitoring and observability tools covering needs related to tracing, logs, databases, alerting, security, and topology, providing them with greater visibility into their software throughout the Software Development Lifecycle.

When they get the call that something needs their attention, they often have to begin investigating essentially from scratch. This means jumping into their many tools and dashboards, pouring over logs to try and figure out the source of the problem. While these monitoring and observability tools can provide valuable insights, it can still be difficult to identify which changes impacted which other components.

Some organizations attempt to use communication tools like Slack to track changes. While we love the scroll as much as anyone else, it is far from a comprehensive solution and still lacks the connection between source and impact. Chances are that they will need to call in for additional help in tracking down the root of the problem from someone else on their team or a different team altogether.

In both cases, the person on-call still needs to spend significant and valuable time on connecting the dots between changes and issues. Time that might be better spent on actually fixing the problem and getting the product back online.  

Filling in the Missing Part of the DevOps Toolchain

The tooling available to help identify issues are getting much better at providing visibility, including to provide on-call responders with the context that will help them get to the source of the problem faster. 

Moving forward, we want to see developers gain a better, holistic understanding of their Kubernetes environments. So even if their microservices are highly distributed, their visibility over them should be more unified. 

Fixing problems faster means getting the relevant information into the hands of whoever is called up to address the issue. It should not matter if the incident is in a product that they themselves built or if they are opening it up for the first time. 

Reducing the MTTR relies on providing them with the necessary context from the tools that they are already using to monitor their environment. Hopefully by better utilizing our existing resources, we can cut down on the time and number of people required to fix issues when they pop up — allowing us to get back to actually building better products.